Technology
What’s up with LLMs representing XORs of arbitrary features?
2024-01-07
Download
Right click and do "save link as"
Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.Thanks to Clément Dumas, Nikola Jurković, Nora Belrose, Arthur Conmy, and Oam Patel for feedback.
In the comments of the post on Google Deepmind's CCS challenges paper, I expressed skepticism that some of the experimental results seemed possible. When addressing my concerns, Rohin Shah made some claims along the lines of “If an LLM linearly represents features a and b, then it will also linearly represent their XOR, _aoplus b_, and this is true even in settings where there's no obvious reason the model would need to make use of the feature _aoplus b._”
For reasons that I’ll explain below, I thought this claim was absolutely bonkers, both in general and in the specific setting that the GDM paper was working in. So I ran some experiments to prove Rohin wrong.
The result: Rohin was right and [...]
---
First published:
January 3rd, 2024
Source:
https://www.lesswrong.com/posts/hjJXCn9GsskysDceS/what-s-up-with-llms-representing-xors-of-arbitrary-features
---
Narrated by TYPE III AUDIO.
view more
In the comments of the post on Google Deepmind's CCS challenges paper, I expressed skepticism that some of the experimental results seemed possible. When addressing my concerns, Rohin Shah made some claims along the lines of “If an LLM linearly represents features a and b, then it will also linearly represent their XOR, _aoplus b_, and this is true even in settings where there's no obvious reason the model would need to make use of the feature _aoplus b._”
For reasons that I’ll explain below, I thought this claim was absolutely bonkers, both in general and in the specific setting that the GDM paper was working in. So I ran some experiments to prove Rohin wrong.
The result: Rohin was right and [...]
---
First published:
January 3rd, 2024
Source:
https://www.lesswrong.com/posts/hjJXCn9GsskysDceS/what-s-up-with-llms-representing-xors-of-arbitrary-features
---
Narrated by TYPE III AUDIO.
More Episodes
The Plan - 2023 Version
2024-01-04
2024-01-04
Apologizing is a Core Rationalist Skill
2024-01-03
2024-01-03
The Dark Arts
2024-01-01
2024-01-01
Succession
2023-12-24
2023-12-24
Constellations are Younger than Continents
2023-12-20
2023-12-20
Is being sexy for your homies?
2023-12-17
2023-12-17
2023 Unofficial LessWrong Census/Survey
2023-12-13
2023-12-13
Get this podcast on your
phone, FREE
