Safety and reliability in ML systems

We review work on the operational integrity of ML models, focusing on their predictability and stability in varied and unpredictable environments. We explore mechanisms to detect irregularities and anomalies that can compromise functionality and security in applications, and look at rigorous verification methods to ensure the trustworthiness of ML systems.

This realm is of pivotal importance due to the increasing integration of ML models into critical and sensitive applications, where their malfunction or compromise can lead to significant, sometimes irreversible, consequences. Our focus on this area is driven by the imperative need to understand and address the inherent vulnerabilities and uncertainties associated with developing and deploying ML systems.

Series in this AoI

Anomaly detection

Many industrial applications of automated decision-making involve the detection of anomalous behaviour. Precisely defining what this means is a complex problem hindered by strong class imbalances, lack of useful models or changing generating distributions. Constant advances in the field enable ever more challenging applications to be tackled.

Robustness in ML

The robustness of a model is a measure of its stability with respect to perturbations of the input. We investigate and review recent developments in the field affecting the training, evaluation and deployment of ML systems.

Formal verification under uncertainty

Being able to ensure that AI systems behave as intended is essential for their adoption in many areas. This series covers formal methods for the verification of safety, fairness, and other relevant properties of (probabilistic) ML systems. We document the progress in research domains such as neuro-symbolic verification, probabilistic model-checking, and weighted model integration.