Concept Activation Vectors

In the last seminar of our XAI series, Iván Rodríguez from appliedAI talks about Concept Activation Vectors (CAVs). CAVs go beyond feature attribution and bring a quantitative approach to testing. He will discuss how this tool interprets a neural network’s internal state in terms of human-friendly concepts.

Abstract: In this XAI seminar, we’ll start by diving into the testing with Concept Activation Vectors (TCAV) method, which helps us gauge how much a model’s prediction is influenced by a user-defined concept. Additionally, we’ll explore two other papers, Concept Bottleneck Models (CBM) and Post-Hoc CBM, which present clever approaches to engage with user-defined concepts. For instance, if the model initially overlooked a bone spur on an X-ray, would it still predict severe arthritis?

References

[Kim18I]

Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV), Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, Rory Sayres.

Jun 2018

The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of …

[Koh20C]

Concept Bottleneck Models, Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, Percy Liang.

Nov 2020

We seek to learn models that we can interact with using high-level concepts: if the model did not think there was a bone spur in the x-ray, would it still predict severe arthritis? State-of-the-art models today do not typically support the manipulation of concepts like "the existence of bone spurs", as they are trained end-to-end to go directly from raw input (e.g., pixels) to output (e.g., …

[Yuk23P]

Post-hoc Concept Bottleneck Models, Mert Yuksekgonul, Maggie Wang, James Zou.

Feb 2023

Concept Bottleneck Models (CBMs) map the inputs onto a set of interpretable concepts (``the bottleneck'') and use the concepts to make predictions. A concept bottleneck enhances interpretability since it can be investigated to understand what concepts the model "sees" in an input and which of these concepts are deemed important. However, CBMs are restrictive in practice as they require dense …

References

In this series →