Paper pills

Paper pills are written for practitioners or researchers who want a peek into recent developments but are short on time. We publish monthly digests as well as occasional highlights of major conferences.

Pills

Random Sum-Product Networks: A Simple and Effective Approach to Probabilistic Deep Learning

Is structure learning even necessary? By generating overparametrised, random model structures for sum-product networks, a class of tractable probabilistic graphical models, Peharz et al. manage to obtain models that are competitive with far more complex generative learning approaches as well as with discriminative approaches, whilst yielding well-calibrated probabilities and being robust to missing features.

A photo of Dominik
Bayesian ML and Probabilistic Programming

Automatic Posterior Transformation for Likelihood-free Inference

A sequential neural posterior estimation method that modifies the posterior approximation using arbitrary, dynamically updated proposals. Also, it is compatible with arbitrary choices of priors, proposals, and powerful flow-based density estimators.

A picture of Maternus
Simulation-Based Inference

Jump-Start Reinforcement Learning

Reinforcement learning often presents a challenge when training an agent from scratch, as reward sparsity and exploration issues can impede learning. Leveraging offline data to warm-start the policy can significantly enhance learning, however, naive initialisation tends to yield poor results. A recent paper investigates various methods to transition from offline initialisation to online fine-tuning, providing both experimental and theoretical insights on sample complexity.

A photo of Fabio
Efficient RL

Implicit Q Learning

A core challenge when applying dynamic programming based approaches to offline reinforcement learning is the bootstrapping error. This pill presents a paper proposing an algorithm called implicit Q learning that mitigates bootstrapping issues by modifying the argmax in the Bellman equation.

A picture of Robert
Efficient RL

Deep Learning with Functional Inputs

Functional data is common in many applications but has not been included into deep-learning methods naturally. This paper presents an input layer facilitating functional data naturally by integrating it into densely connected feed-forward neural networks.

A picture of Maternus
Deep Learning on Functional Data

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

The process of pre-training large language models can incur significant expenses. As these models continue to grow in size, there is an increasing need for solvers, which exhibit improved convergence characteristics, to decrease the overall costs of the model training.

A photo of Kristof
Large Language Models

Learning-rate-free learning by D-Adaptation

Finely tuning the learning rate (lr) in the training of neural networks is crucial for their performance, and often requires a costly search procedure. One of this year’s ICML Outstanding Paper Awards proposes “D-adaptation”, which sets the optimal lr automatically. D-adaptation thus removes the need of tuning the lr, and is therefore of high practical importance.

Samuel

Reasoning Traces as Learning Signal

An important feature of large language models is their ability to provide detailed responses that resemble “thinking step by step.” A recent line of work aims at reasoning beyond in-context learning by using intermediate reasoning as learning signal. This pill presents two works from that area - Chain-of-Thought Imitation with Procedure Cloning for imitation learning and Let’s Verify Step by Step that explicitly rewards sensible reasoning steps in a reinforcement learning setting.

A picture of Robert
Large Language Models

Advantage-Induced Policy Alignment

Building on the classic results on reward weighted regression and its more recent adaptation to deep learning, a new algorithm called advantage-induced policy alignment (APA) is proposed for aligning language models (LM) to human preferences. APA is claimed to significantly outperform state-of-the-art methods such as PPO. This pill also contains a short overview of the family of algorithms from which APA emerged.

A picture of Michael
Efficient RL

Direct Preference Optimization

With direct preference optimization (DPO), a language model can be aligned with human preferences without using reinforcement learning, thereby significantly simplifying the training process. DPO can be used on the same data as RLHF (RL from human feedback), and the paper claims that it results in the same or even improved performance.

A picture of Michael
Large Language Models