XDOF launches real-robot benchmark for frontier AI
XDOF launched Ember-0 and a public leaderboard measuring how frontier AI models perform on real bimanual robot tasks.
XDOF has launched Ember-0, a benchmark and leaderboard for testing frontier AI models on physical robots.
What happened
The benchmark covers 16 bimanual manipulation tasks performed on real robot stations. It evaluates capabilities including planning, perception, spatial reasoning, tool use, dexterity, memory and long-horizon execution.
The tested models often produced sensible plans and useful spatial targets. They performed less reliably when tasks required precise contact, deformable objects, tool mechanics or fast feedback.
Why it matters
A model can understand instructions without being able to execute them safely and consistently in the physical world. Real-robot testing exposes failures that cannot be measured through language or software benchmarks alone.
The bigger picture
Robotics needs evaluation systems that separate reasoning ability from manipulation skill. Public, repeatable benchmarks can give developers a clearer picture of where frontier models are improving and where dedicated robot-learning systems remain necessary.