Calibration of modern classifiers

An introduction to the pitfalls of uncalibrated classifiers and modern techniques for (re)calibration.

Optimal decision-making based on classifiers requires that the confidence in their predictions reflect the actual error rates. When this happens, one speaks of a calibrated model. Recent work has shown that expressive neural networks are able to overfit the cross-entropy loss without losing accuracy, thus producing overconfident (i.e. miscalibrated) models.

Learning outcomes

  • Understanding quantitatively how optimal decisions depend on all probabilities and not just on the predicted class.
  • Learning to measure (mis)calibration of models.
  • Recalibration of classifiers during training: loss functions, regularisation.
  • Recalibration of classifiers, a posteriori: non-parametric, parametric, Bayesian.

Structure of the workshop

  • Introduction to the nomenclature: notions of calibration, metrics, calibration functions
  • Relation between calibration and accuracy
  • Discussion of situations when calibration is important for performance
  • Measuring miscalibration 1: histogram estimators
  • Measuring miscalibration 2: statistical tests and decision process simulation
  • Recalibration by post-processing
  • Training calibrated classifiers
  • Performance comparison of different approaches to calibration

In this series