Advisor(s):
Philipp Hennig
News:
Research Overview:
Stochastic optimization methods have become an increasingly important tool in minimizing nonlinear high dimensional objectives where only noisy function and gradient evaluations are available.
Probabilistic numerics provides a framework to address these challenges by explicitly modeling noise and uncertainty.
Optimization can be cast as an inference problem where unknown quantities (e.g. the gradient or the Hessian at every location) is inferred through previously collected noisy gradient evaluations.
Often the noise variances on gradients and values is known or can be estimated with low overhead (e.g for empirical risk minimization) such that the optimizer has a quantitative understanding of how uncertain its inputs are.
My work includes i) robust estimation of first- and second order search directions for stochastic optimization. The work is closely related to previous work by Hennig and Kiefel [ ] who showed that quasi-Newton methods, such as the BFGS rule, arise as the mean of a Gaussian distribution over the elements of the Hessian matrix of an optimization objective. More distantly related work includes e.g Hennig [ ] on the solution of linear solvers.
Further areas of research are: ii) automated step size adaptation in stochatic settings, where we extended the classic line search paradigm of deterministic optimization to a fully probabilistic one [ ]. iii) overfitting prevention by early-stopping without the help of a validation set [ ] based on a lightweight statistical test, which compares gradient magintudes to their noise.
The goal of my work is a smart machine (optimizer) that needs little to no human expert knowledge to perform its task, even communicating important information about the optimization progress to us.
For more information see The Independent Max Planck Research Group on Probabilistic Numerics
About me: I am a physicist by training and I love models, math and abstract thinking. Simulations and testing own theories on data (by writing a computer program) is an essential part of science for me.
Mahsereci, M., Balles, L., Lassner, C., Hennig, P.
arXiv preprint arXiv:1703.09580, 2017 (article)