ORA: Open Robot-Use Agents

Code Appendix Gallery · 153 episodes

TL;DR: ORA is an open robot-use agent harness that lets open-weight language models operate robots fully zero-shot: no VLA, no robot training and no task-specific memory, with results on RoboTwin and LIBERO-Pro (69.2% and 81.4% by the official success checks) competitive with agents built on frontier models. It gets there with a harness designed for robot tasks in the physical world: exact task and environment context, tools for perception and motion, and context management.

Figure 1. DeepSeek-V4.1-Flash, an open-weight model, operating robots through ORA, zero-shot. Estimated model-token cost at DeepSeek's off-peak API rates, using estimated cache hits: $0.13 per solved task on average across the 40 shown here, or $0.30–$0.35 per solved benchmark case including failed attempts. These are API-equivalent estimates, not actual self-hosting costs; GPU and simulator costs are excluded (see Cost).

(a) RoboTwin C2R

50 tasks · 250 cases

Success (%)

(b) LIBERO-Pro

8 Task and Swap suites · 800 cases

Success (%)

ORA harness (ours) + open-weight model, zero-shotRPent agent harness + language model + VLA + task-specific memoryVLA policies and other methods

Figure 2. Success rates on (a) RoboTwin C2R (250 cases) and (b) LIBERO-Pro (800 cases), using the same case lists as RPent^[1]. ORA runs every case zero-shot, with no demonstrations or task-specific memory. Baseline rates come from RPent's leaderboard. [a] Astra uses frozen memory batches. [b] Flash Mode uses released memory and reports the best of ten seeds. Full protocol and values: appendix.

Evaluation cases and baseline models

RoboTwin C2R (clean-to-randomized) covers 50 tasks with 5 seeds each in randomized scenes. LIBERO-Pro covers eight Task and Swap suites with 10 tasks and 10 seeds each. ORA rates count every case.

RPent combines a language model, VLA and task-specific memory collected in an earlier exploration pass. Models include GPT-5.5, Opus-4.7, GPT-6 Astra and open-weight Qwen3.6-27B^[2]. End-to-end VLA baselines are π0.5^[3], π_RLinf^[4], GR00T-N1.7^[5], LingBot-VLA^[6] and AtomVLA^[7]. RPent Flash Mode uses Molmo2-8B^[8] with released memory.

These are comparisons of complete systems with different tools, memory and protocols. Rates use official success checks, which can overstate task completion; see Limitations.

Introduction

ORA is an open robot-use agent harness that enables open-weight models like DeepSeek-V4.1-Flash^[9] to solve robot tasks (Figure 3). It leads agents built on closed frontier models on RoboTwin and comes close to them on LIBERO-Pro (Figure 2). Below we explain why robot use needs agents driven by open models and where ORA sits among existing methods.

Figure 3. ORA at work on a RoboDojo^[10] task. Left: the agent's terminal, where the model sees, reasons and calls tools. Right: the robot's cameras. Asked to stack the blocks blue, yellow, orange from the bottom up, the model succeeds in round 2, after round 1 failed. Replayed from the run's records, motion sped up.

Why robot-use agents

LLM agents now handle much digital work, including coding^[11] and computer use, as computer-use agents (CUAs)^[12]^[13]. However, most existing robot policies, such as vision-language-action models (VLAs)^[14]^[15] and world action models (WAMs)^[16], still rely on robot-specific training data, collected by teleoperation^[17], with handheld grippers such as UMI^[18] or from egocentric videos^[19]. An alternative is to build on the reasoning and tool calling of existing LLM agents. As a recent talk asked: could a strong enough agent operate a robot from its manual alone (Figure 4)^[20]?

Slide: General Robotics? Assume GPT-9.8 + Agentic Control + Any Robot + Its Manual. — **Figure 4.** From Jianyuan Wang's talk on general robotics at IMW 2026; credit^[20].

GPT-6 Astra versus Fable 5.1 on direct robot control: completion rate against API cost, with robot camera views. — **Figure 5.** Zero-shot direct robot control with GPT-6 Astra; credit^[21].

This suggests bringing the agent loop to the physical world, as in a recent robot-use agent^[22]: a tool call moves the robot, the model reasons about what it sees, and the next camera frame starts the next turn. Meanwhile, frontier models such as GPT-6 Astra have shown strong zero-shot direct robot control (Figure 5)^[21], and several recent demos operate robots this way^[23]^[24]^[25].

Open-weight models, however, have barely been tested on this, yet robot use needs them for two reasons. First, to understand how well language models manipulate objects and how to improve them, we need large amounts of reproducible data, which means a model we can serve ourselves and experiment with at an acceptable cost, rather than paying frontier-model API prices. Second, a deployed robot system built on an LLM must be fast and efficient enough to run on local hardware. That needs an open model small enough to deploy, and task data that only an open model makes affordable to collect.

Where ORA fits

Methods that control robots with learned models fall into four families. They differ in the role of the language model and in how often it makes a decision (Table 1).

Vision-language-action (VLA) models^[14]^[3]^[15] and, more recently, world action models that also predict future video^[16]^[26] map camera images and an instruction to actions at every control step and are trained on the largest robot datasets. Because the vision-language backbone is fine-tuned into a visuomotor policy that outputs actions, these systems use little of its ability to reason and act as an agent, for example to measure before moving, plan several steps ahead or change approach after a failure.

A common response is to add a language model as a planner on top of a learned policy, the System 2 / System 1 design^[27]^[28]^[29]^[5]. The language model splits the task into steps and the policy carries them out. In RPent^[1], a language model directs a VLA policy using task-specific memory. The model can also command some motions itself, but most of the manipulation is still done by the VLA, which is trained on robot demonstrations.

A third family has the language model write the policy as code. Code as Policies^[30] turns the scene and the task into a program that calls perception and control primitives. CaP-Agent0^[31], ASPIRE^[32] and RATs^[33] add an agent that runs the program, reads the outcome and revises it; ASPIRE and RATs also keep the skills that work. These methods need no robot demonstrations, and they are economical to run: the model is called once or a few times per task, and the program then runs without it. The price is that the program executes as one or a few open-loop segments. Getting it right often takes trial runs (ASPIRE, for example, learns its skills on separate seeds before it is tested), and between revisions no model looks at the scene: the program reacts only through the checks written into it, which limits how well it carries over to new scenes.

The fourth family runs the agent loop directly, in the ReAct style^[34]: the model reasons, calls a tool, reads the result and decides again. Frontier models have controlled robots this way^[21]^[23], and Guava^[35], OpenRUA^[36], Agent as Policy^[37] and Robo-Harness K1^[38] are harnesses for such agents. When the model issued every low-level command itself, the best model in one study reached 5.5% on LIBERO^[23]. ORA is in this family. Each tool call covers one bounded motion, which a controller executes, and the model sees the result before its next decision. It costs a model call at every step, so at today’s latencies and prices this family is slower and costlier to run than a program written once (see Cost and Limitations).

Family	What the language model does	How often it decides	Robot data needed	Examples
VLA and world action models	is trained into the motor policy	every control step	large robot datasets	RT-2^[14], π0, π0.5^[3], OpenVLA^[15], DreamZero^[16]
Planner + learned policy (System 2 / System 1)	directs a learned policy step by step	when the policy finishes a step	demonstrations to train the policy	SayCan^[27], Hi Robot^[28], Helix^[29], GR00T N1^[5], RPent^[1]
Code as policy	writes a program over perception and control primitives, runs it and revises it	once or a few times per task	none	Code as Policies^[30], CaP-Agent0^[31], ASPIRE^[32], RATs^[33]
Robot-use agent (ReAct loop)	calls a tool, reads the result and decides again	once per tool call	none	Guava^[35], OpenRUA^[36], Agent as Policy^[37], Robo-Harness K1^[38], ORA

Table 1. Four families of methods that control robots with learned models, by the role of the language model and how often it decides. Some systems combine several families.

How ORA works

ORA follows the practice of coding agents such as Claude Code^[39] and Pi^[40]: the task runs as a ReAct loop^[34] in which, every turn, the model reads its input, thinks and makes one tool call; the action changes the state of the world, and what comes back becomes the next turn’s input (Figure 6). Like those agents, it has a prompt bundle, tools and context management, but for a robot each has to do more.

First, the prompt bundle has to describe the body of each embodiment and the task: the robot’s frames, the point each command moves, the gripper’s constants, and the task’s success criteria written out in detail. Second, the tool suite has to help the model sense and act in the physical environment. In place of a shell and file reads and writes, the model sees the world only through cameras and the robot’s sensors, so it needs perception tools that measure the real scene and action tools that act on the environment and report, after every action, exactly what happened. Third, compared with a coding agent, a robot-use agent receives a large volume of redundant visual observations and environment data, so context management has to carry much heavier input: every turn adds camera images, and the model keeps returning to earlier results. The input must stay small without losing those results or breaking the prompt cache.

On the other side is the environment: a supervisor checks every motion against the robot’s limits, runs it and returns a measured receipt. Throughout, the model perceives the scene only through RGB and depth images and never sees privileged information from the simulator: the success check’s code, object ground truth or simulator contacts.

**Figure 6.** The interaction loop between the LLM, ORA and the environment. Every turn, the model reads the prompt bundle and the context, thinks, and calls one tool; an action changes the world, and the observation that comes back, like every tool result, joins the context for the next turn. The prompt bundle and the context, rebuilt every turn with the newest material first, make up everything the model receives; on the other side, the environment holds the robots, returns an observation every turn and keeps the success check. Click any part to see it in detail.

Environment and observation

We run ORA on three simulated benchmarks, each with its own robot (Figure 7). They differ in the simulator and its world frame, the arms, the point each command moves, the gripper’s geometry, the cameras and how success is judged. RoboDojo^[10] runs on Isaac Sim^[41] with a two-arm ARX X5; RoboTwin^[42] runs on SAPIEN^[43] with a two-arm ALOHA; LIBERO^[44] and LIBERO-Pro^[45] run on MuJoCo^[46] through robosuite^[47] with a one-arm Franka.

The loop, the kinds of tools and the model are shared; only the controllers are fitted to each robot. What differs reaches the model as text, through a benchmark boundary and a robot card. With these, one model runs all three robots. The card and the boundary give the agent the details of the body and the scene it needs to reason with, such as the point each command moves, where the fingers touch and how wide the jaws open; open a card below for them.

ARX X5RoboDojoFixed-base ARX X5: left 6-DoF arm, right 6-DoF arm; parallel-jaw grippers. Cameras: head fixed RGB-D; left_wrist, right_wrist moving RGB-D.

Simulator	Isaac Sim^[41]
Tasks we ran	42 multi-stage tabletop tasks, a first scan
Robot	ARX X5, two 6-DoF arms
Point a command moves	wrist flange; add no offset
Approach / jaw-closing axis	[1, 0, 0] / [0, 1, 0]
Where the fingers touch	jaw contact centre 0.150 m along the approach
Full jaw opening	88.8 mm
Cameras	one fixed, one on each wrist; 640 × 480
Official success	the benchmark's reward manager

ALOHARoboTwinFixed-base ALOHA-AgileX: left 6-DoF arm, right 6-DoF arm; parallel-jaw grippers. right_tcp is the controlled reference, 0.12 m from the flange; commands already target it.

Simulator	SAPIEN^[43]
Tasks we ran	50 tasks, 5 randomized scenes each
Robot	ALOHA-AgileX, two 6-DoF arms
Point a command moves	tool point 0.12 m beyond the flange
Approach / jaw-closing axis	[1, 0, 0] / [0, 1, 0]
Where the fingers touch	fingers from −50 to 36 mm around the tool point
Full jaw opening	94.9 mm, given in the `grasp_plan` help
Cameras	two fixed, one on each wrist; 320 × 240
Official success	the task's `check_success`

FrankaLIBERO / LIBERO-ProFixed-base Franka Panda: right 7-DoF arm; parallel-jaw grippers. right_grip_site is the controlled reference, 0.097 m from the end-effector root; commands already target it.

Simulator	MuJoCo^[46] (robosuite^[47])
Tasks we ran	40 tasks × 10 seeds; LIBERO-Pro 8 suites × 100
Robot	Franka Panda, one 7-DoF arm
Point a command moves	grip site 0.097 m beyond the end-effector root
Approach / jaw-closing axis	[0, 0, 1] / [1, 0, 0]
Where the fingers touch	fingers from −44 to 9 mm around the grip site
Full jaw opening	80 mm
Cameras	one fixed, one on the wrist; 256 × 256
Official success	the environment's termination signal

Figure 7. The three embodiments and the opening of each robot card, verbatim. Open a card for what it and the benchmark boundary tell the model; the model never sees the success code, only its yes or no.

After every action, the environment returns an observation (Figure 8). It holds an RGB image from each camera with its calibration, the measured pose and gripper of each arm, the receipt of the last action and the remaining budget, and it is paired with the observation the action was chosen from, so the model can compare before and after. Each camera’s depth image is stored with the observation, and the looking tools read it when the model measures.

Prompt Bundle

A coding agent reads the repository and its instructions before it edits anything; a robot-use agent likewise has to be told enough about its environment before it moves. We therefore give it a prompt bundle of seven layers (Figure 6). A new embodiment therefore needs no training: only the embodiment context is replaced. Open any layer to read it as the model did in the run of Figure 3.

System policy, the same for every robot and task: the turn protocol (read-only calls may be batched; a call that changes the world stands alone), every derived number goes to compute, how older evidence is replaced by references, and what the budgets mean.

System message, Native Robot Policy v4, excerpt

Turn protocol

Read-only tools may be called sequentially or in one independent batch. A
world-mutating tool must be the only tool call in its assistant response. Each
accepted mutation ends the current turn; wait for the next actual observation
before choosing another mutation.

Arithmetic belongs in the advertised calculation capability, not in your own
working. Whenever a number would be derived -- a residual, an offset along a
direction, a quaternion unpacked, a sum of extents, a dot or cross product, a
projection between image and world, a trigonometric value -- that derivation is
part of how the action was chosen […]

How this context reaches you

Current actual evidence arrives complete on every turn. Earlier evidence is not
guaranteed to stay in the form you first received it.

A result that exceeds its inline budget is replaced by a status record carrying
an exact reference to the complete original. […] Open the reference when you
need what a past result said; call again when you need what is true now.

What the reported allowances mean

Every turn reports what remains of the current session round […]: accepted world
mutations, model turns including read-only ones, and elapsed wall-clock seconds.
[…] keep enough in hand to witness the effect of your final mutation and to take
the advertised closing step: an unverified last action and an unclosed episode
are both worse than one attempt fewer.