Reading Anthropic's Tests of Language Models Driving Robots
Anthropic's Frontier Red Team handed robots to twelve language models: classic control toys like a balancing pendulum, a simulated quadruped and humanoid, a robot arm, and a real Unitree Go2. This post follows one thread from that work: the same model gets very different results depending on how it is wired to the robot.
The team gave the models four ways in. Set motor torques directly at every step, write controller code, train a policy (a network that maps observations to actions) with reinforcement learning, or send high-level commands to a policy that was already trained.
A model's robotics score depends as much on the robot body and the control interface as on the model itself. The same model can look weak or strong depending on whether it is setting motor torques directly, writing a Python controller, supervising a pretrained policy, or training its own policy with reinforcement learning—each of which is a different way of the model completing the same task.
When models had to drive the joints themselves, they mostly failed. Under direct control of a robot arm, kitchen tasks like "put the plate on the stove" were completed 0 to 5.5% of the time. On the same tasks, letting the model supervise a pretrained VLA (a policy that turns camera images and an instruction into arm motions) raised success and progress for every model. I read this as the harness, the scaffolding that connects a model to its tools and controllers, deciding the capability more than the model does.
The cursor tool, by contrast, gives every model a large uplift on manipulation—for Mythos Preview, success on the 10-task subset goes from 6% to 32%. The compass does the same for locomotion, lifting every configuration we tested.
The cursor is a small red X on the gripper camera that the model can move and query for the object and distance at that point. The compass just tells the model which way it is facing, in degrees. Richer visual aids like depth maps and segmentation overlays did roughly nothing. The model stayed the same and one small tool changed, and success went up more than fivefold, so I take the harness side of the gap to be that large.
From a safety standpoint, this matters because pretrained policies are exactly what a deployed system would realistically provide: a model does not need to drive joints itself to act capably in the world, only access to a competent controller. Capability assessments that test the model in isolation will understate what it can do once embedded in a robotic stack.
That competent controller is not equally available for every body, though. The gait policy the team used for the quadruped is, in their words, widely available for most commercial quadruped robots. The Go2 is the same robot from the Project Fetch post. The G1 humanoid was tested only through direct control, code, and reinforcement learning, with no pretrained policy, and no model stood it up from a collapsed pose even once. The paper itself says today's frontier models cannot control humanoid robots without a pretrained policy.
I think the humanoid was bound to be hard. There don't seem to be many cases of people directly controlling humanoids, so I'd guess there isn't much of a harness to plug a model into yet. People talk about putting human intelligence into humanoids, but a model trained on text would still have to learn walking separately, and that seems hard even for a frontier model. The paper does describe its subjects as general-purpose chat models with no robotics training. What it credits for the gains between generations, though, is better vision, numerical consistency, and 3D understanding.
The paper's conclusion is that a model's real-world influence can change by orders of magnitude depending on what it has access to, so evaluations and deployments should treat access level as a core part of the system.
