I am a first year Ph.D. student in the Probabilistic Numerics Group, where I am working on optimization methods for machine learning. Optimization algorithms are the workhorse of contemporary machine learning. This is where the numbers are crunched and information is actually extracted from the data. Intriguingly, numerical optimizers themselves are compact little "agents": they make decisions (where to evaluate next, how many and which data points to use) based on observations (function values and gradients, typically corrupted by noise). My goal is to design smarter optimizers!
Recently, I have been working on a method to adaptively choose "good" batch sizes when performing stochastic gradient descent on empirical risk minimization problems, e.g., the training of neural networks [ ].
Prior to joining the Max Planck Institute as a Ph.D. student, I studied Mathematics (B.Sc.) and Scientific Computing (M.Sc.) at Heidelberg University and spent some time as a visiting student at Tsinghua University in Beijing.
Optimization problems arising in intelligent systems are similar to those studied in other fields (such as operations research, control, and computational physics). But they also have a few prominent features that are not addressed particularly well by classic optimization methods.
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing estimate of the generalization performance. In this paper we propose a novel early stopping criterion which is based on fast-to-compute, local statistics of the computed gradients and entirely removes the need for a held-out validation set. Our experiments show that this is a viable approach in the setting of least-squares and logistic regression as well as neural networks.
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the stochastic optimization algorithm, though, since it determines the variance of the gradient estimates. This variance also changes over the optimization process; when using a constant batch size, stability and convergence is thus often enforced by means of a (manually tuned) decreasing learning rate schedule. We propose a practical method for dynamic batch size adaptation. It estimates the variance of the stochastic gradients and adapts the batch size to decrease the variance proportionally to the value of the objective function, removing the need for the aforementioned learning rate decrease. In contrast to recent related work, our algorithm couples the batch size to the learning rate, directly reflecting the known relationship between the two. On three image classification benchmarks, our batch size adaptation yields faster optimization convergence, while simultaneously simplifying learning rate tuning. A TensorFlow implementation is available.
Our goal is to understand the principles of Perception, Action and Learning in autonomous systems that successfully interact with complex environments and to use this understanding to design future systems