News:

• We did some work on Early-Stopping without a Validation Set. Code will be released in the near future.
• An extended version of our NIPS '15 conference paper on probabilistic line searches is now on arXiv. It also contains a detailed pseudocode.
• I gave a talk at Amazon Research -- Cambridge -- 03/2017.
• I co-organized the workshop `Optimizing the Optimizers' @NIPS 2016
• I gave a talk at the Workshop on Uncertainty Quantification @GPSS (Gaussian Process Summer School) – Sheffield, UK – 09/2016
• I did a summer internship at the Amazon Development Center in Berlin, Germany – 07/2016 - 10/2016
• Our work on probabilistic line-searches is selected for a full oral at NIPS 2015.
• I gave a talk at Microsoft Research – Cambridge – 09/2015
• I gave a talk at the Workshop on Probabilistic Numerics @DALI 2015

Research Overview:

Stochastic optimization methods have become an increasingly important tool in minimizing nonlinear high dimensional objectives where only noisy function and gradient evaluations are available.

Probabilistic numerics provides a framework to address these challenges by explicitly modeling noise and uncertainty.

Optimization can be cast as an inference problem where unknown quantities (e.g. the gradient or the Hessian at every location) is inferred through previously collected noisy gradient evaluations.

Often the noise variances on gradients and values is known or can be estimated with low overhead (e.g for empirical risk minimization) such that the optimizer has a quantitative understanding of how uncertain its inputs are.

My work includes i) robust estimation of first- and second order search directions for stochastic optimization. The work is closely related to previous work by Hennig and Kiefel [ ] who showed that quasi-Newton methods, such as the BFGS rule, arise as the mean of a Gaussian distribution over the elements of the Hessian matrix of an optimization objective. More distantly related work includes e.g Hennig [ ] on the solution of linear solvers.

Further areas of research are: ii) automated step size adaptation in stochatic settings, where we extended the classic line search paradigm of deterministic optimization to a fully probabilistic one [ ]. iii) overfitting prevention by early-stopping without the help of a validation set [ ] based on a lightweight statistical test, which compares gradient magintudes to their noise.

The goal of my work is a smart machine (optimizer) that needs little to no human expert knowledge to perform its task, even communicating important information about the optimization progress to us.

For more information see The Independent Max Planck Research Group on Probabilistic Numerics

About me: I am a physicist by training and I love models, math and abstract thinking. Simulations and testing own theories on data (by writing a computer program) is an essential part of science for me.