arxiv preprint - Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models

In this episode, we discuss Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models by Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, Mor Geva. The paper presents a novel framework named Patchscopes designed to improve understanding of the hidden representations in large language models (LLMs) by using the models themselves to articulate these representations in natural language. Patchscopes integrates and extends existing interpretability techniques, overcoming limitations like the inability to inspect early layers and enhancing expressivity. Beyond reconciling former methods, Patchscopes also enables innovative applications, including having more advanced LLMs explain the workings of simpler ones and facilitating self-correction in complex reasoning tasks.