Reading Anthropic's GRAM: An Off Switch for Dual-Use Knowledge
Anthropic, working with AE Studio, published GRAM, a method for switching dual-use knowledge inside a model on and off by category. Dual-use knowledge is knowledge that can be used for good or for harm. The original also walks through three rounds of experiments and their limits. This post follows two parts of it: the possibility of security knowledge being switched off, and the cost problem the method set out to solve.
Before we start — key terms
Pretraining data filtering: Removing data on a given topic before a model's initial large-scale training, so the model never learns that knowledge in the first place. In earlier work, Anthropic filtered information about chemical, biological, radiological and nuclear (CBRN) weapons this way.
GRAM (Gradient-Routed Auxiliary Modules): A method that adds removable groups of neurons (modules) to every layer of a Transformer, one per dual-use category. When training on text from that category, the general weights are briefly frozen and only that module learns, so the knowledge collects in the module. After training, deleting or keeping the module switches the capability off or on.
The original leads its examples of dual-use knowledge with cybersecurity.
For example, knowledge of cybersecurity can help patch critical security vulnerabilities, or it can be used to exploit them. Knowledge of virology can help a researcher create a vaccine, but it can also help a malicious actor design a deadly pathogen.
In the experiments, cybersecurity was one of the four categories to be split off, alongside virology, nuclear physics and a niche programming language. Deleting a module removed that capability about as well as never training on the data. Having studied security, I found this part a little disheartening. I can see a day coming when I ask an AI a security question and hit a wall. As of the original, GRAM has not been applied to Claude, and Anthropic says it is not sure it ever will be. The module is also designed to stay on for trusted deployments.
Opening security capabilities only to vetted people is already happening on the classifier side. In October, Anthropic expanded its Cyber Verification Program, which gives qualifying security professionals advanced cyber capabilities with reduced blocking classifiers.
Still, I found the thinking behind this technique interesting in itself. It starts from the cost of filtering.
But filtering is a blunt instrument. It produces one model with one fixed set of capabilities. Using filtering, if you want a model version that can discuss advanced virology—for deployment in a vetted biosecurity lab, say—and another version that can't, you have to train two separate models.
Frontier models are large and very expensive to train, so the original calls that cost prohibitive. GRAM gives each category its own module and trains once. With four categories, a single training run gave a model that could be configured 16 ways, each category on or off. What caught my interest was solving the one-model-per-version problem by splitting the model into compartments and training it once.
The original presents this as "a new method." Existing approaches were stuck on cost, so they built a new one. I have problems in security I want to solve, and some of them can't be solved with what is already known. I want to learn this attitude of finding a way to solve the problem somehow, and I want to do work like this.
