Abstract: Modern vision systems can recognize objects, actions, and scenes, and now even generate realistic imagery, yet we still do not know what they actually understand or how they reason. This talk asks a fundamental question: do neural networks organize visual knowledge around reusable concepts, and can we systematically uncover them?
In the first part, I introduce Video Transformer Concept Discovery (VTCD), the first framework for automatically identifying and ranking interpretable spatiotemporal concepts within video transformers through unsupervised analysis of their internal representations. VTCD reveals how models encode motion, objects, and interactions over time, and shows that many of these mechanisms emerge consistently across both supervised and self-supervised training, suggesting the presence of universal structures in video representations. We further demonstrate that these automatically discovered concepts can be leveraged for downstream tasks, such as zero-shot video object segmentation.
In the second part, I present Universal Sparse Autoencoders (USAEs), which extend this idea across models by learning a shared concept space capable of reconstructing activations from multiple pretrained networks simultaneously. USAEs uncover interpretable concepts common across architectures, datasets, and tasks by learning sparse representations in a fully unsupervised manner, aligning features ranging from low-level visual patterns to higher-level object and part representations, and enabling new forms of coordinated multi-model analysis.
Together, these works suggest that neural networks organize visual knowledge around reusable conceptual components. Discovering and aligning such concepts opens new directions for interpreting, comparing, and ultimately designing better vision systems, bringing us closer to understanding how modern AI systems perceive and reason about the world.