We prepare in-depth reviews of proven and emerging techniques, illustrate and benchmark them with best-practices playbooks and code, and communicate them with visually rich showcases and trainings. We also maintain a blog with introductory material and paper reviews, and hold a weekly seminar where we cover both the foundations and recent advances in AI.
Our areas of interest
What we are reading:
Denoising Diffusion and Score Based Generative Models
Progressive Distillation for Fast Sampling of Diffusion Models
The paper describes how the generation process of diffusion models can be sped up by using a slow teacher model for training a faster student - the latter performing fewer sampling steps. This distillation process can be iterated by using the previous student as teacher and the final student was shown to generate high-quality samples in the fraction of the original model’s time.
Comparing Distributions by Measuring Differences that Affect Decision Making
A new family of divergences for distributions is introduced, generalizing KL and others, together with an efficient estimator for it. A beautiful aspect of this family is that the divergence is based on a loss and thus can not only reflect how much distributions differ but also how much their difference affects a selected decision making scenario.
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion
In this rather technical paper, a new dynamics is proposed for the sampling of diffusion models. Instead of sampling the de-noised image directly, an auxiliary velocity variable is sampled and deterministic dynamics are used to reconstruct the de-noised data. The approach outperforms previous state of the art in terms of sample quality.
Neural Collapse in deep classifiers during Terminal Phase of Training
Neural collapse refers to the observation that the last two layers of neural networks that were trained for a long time take a very simple, restricted and universal form - namely an equiangular tight frame. Papers referenced in this pill highlight the phenomenon and provide theoretical hints for why it happens.
Come to our seminar
Data (e)valuation and model interpretation: a game theoretic approach
Attributing a “fair” (for some definition of fair) value to training samples has multiple applications. It can be used to investigate or improve data sources, but it can also help detect outliers and to investigate how certain features and the values thereof influence global model performance. It is also possible to improve model performance: removing points of low value can decrease model error.
Introduction to simulation-based inference
Scientists and engineers employ stochastic numerical simulators to model empirically observed phenomena. In contrast to purely statistical models, simulators express scientific principles that provide powerful inductive biases, improve generalization to new data or scenarios and allow for fewer, more interpretable and domain-relevant parameters. Despite these advantages, tuning a simulator’s parameters so that its outputs match data is challenging.
Hamiltonian Monte Carlo Sampling
Metropolis, Slice or Gibbs sampling tend to be inefficient when applied in high dimensions due to their local random walk behaviour. Hamiltonian Monte Carlo (HMC) was developed around a geometrical understanding of the target distribution, borrowing concepts from physics to generate transitions and exploring the target distribution. Ultimately, moving more rapidly through the target distribution.
Bayesian optimal experiment design (2 of 2)
This is the second of a two talks providing an introduction to Bayesian optimal experimental design (BOED) and how it relates to exploration in reinforcement learning and control. The first part dealt with the theoretical foundations for BOED. This talk completes the topic by presenting an application in control, which was created during my master thesis at TUDa.
Bayesian optimal experiment design (1 of 2)