ECCV 2026

Exploratory, Communicative, and Deployable

Vision-Driven Embodied Agents for Open-World Mobile Manipulation

Boyu Mi1,2* Mengchen Ma2,3* Yifei Yao1,2* Xing Gao2† Junting Chen4 Yangzi Li5 Zihou Zhu2 Guohao Li6 Zhenfei Yin7 Tai Wang2 Yao Mu1,2 Jiangmiao Pang2 Hanqing Wang2†
1Shanghai Jiao Tong University 2Shanghai Artificial Intelligence Laboratory 3Southeast University 4National University of Singapore 5Zhejiang University 6Camel AI 7University of Oxford

*Equal contribution  ·  †Corresponding authors

PaperComing soon Dataset & CheckpointComing soon Code Explore evidence

REAL in the physical world. The high-level policy controls an Ark LIFT2 mobile manipulator through the same tool interface used during simulation training.

Overview

Bridge the gap between what agents see and what they can do.

REAL makes the deployment constraints visible: evidence must be gathered from RGB observations, ambiguity must be resolved by dialogue, and every high-level action must remain executable on the robot.

The deployment gap

Many embodied benchmarks expose global object lists, poses, or success signals. They can also assume a fully specified instruction. Those shortcuts conceal the two actions a deployed agent actually needs: explore and ask.

The REAL response

REAL provides a shared, sim-to-real tool interface without oracle perception, a simulated user for clarification, and a dual-stage SFT + online RL pipeline that teaches the policy to recover, reason, and communicate in a closed loop.

Comparison between prior embodied agent frameworks using oracle perception and REAL using RGB exploration and natural-language interaction.
From privileged simulation to deployable interaction. REAL removes oracle perceptual APIs while retaining physically obtainable scene priors, so policies must explore RGB observations and resolve ambiguous instructions through dialogue.
01 / EXPLORE

Privilege-free perception

Navigation, detection, Set-of-Mark prompting, and visual grounding replace hidden simulator state.

Watch a discovery trajectory
02 / COMMUNICATE

Interactive intent alignment

A simulated user turns ambiguity into an observable decision: pause, ask, ground the response, then act.

Inspect SUL evidence
03 / DEPLOY

One interface, two backends

The same high-level policy transfers zero-shot from simulation to a dual-arm mobile robot.

View physical execution

Evidence explorer

Evidence, one trajectory at a time.

Choose a case to inspect one representative trajectory at its intended scale. Each connects a claimed capability to an observable sequence from the paper and supplementary materials.

Active discovery

Find what the first view misses.

The policy uses a deployable walk_around action to discover an object that is not initially visible, then grounds it before manipulation.

  1. Observe a partial scene
  2. Explore an unseen viewpoint
  3. Detect and ground the target

Framework

One visual-interactive loop from simulation to reality.

REAL routes a fixed MCP schema to simulation handlers or real robot controllers. The high-level policy never receives information that cannot be obtained by a deployed system.

REAL framework showing environment construction, simulated user interaction, data generation, SFT and reinforcement learning, and deployment.
REAL framework. A shared visual-interactive loop connects environment construction, automated trajectory generation, dual-stage policy optimization, benchmark evaluation, and physical deployment.
RGB

Privilege-free perception

Visual observations, not global object inventories, drive navigation and grounding.

ASK

Simulated human feedback

Context-aware clarification is available through a dedicated user interaction tool.

MCP

Backend-agnostic tools

The same high-level action schema dispatches to simulation or physical controllers.

Training

Turn diverse environments into closed-loop behavior.

REAL uses generated trajectories for tool alignment, then online reinforcement learning to improve recovery, exploration, and the timing of user queries.

A training scene used for generated visual-interactive mobile manipulation tasks.
Generated task instances vary rooms, furniture, object configurations, and language descriptions.
01

Compose tasks

Construct static and ambiguous task families across rooms, receptacles, objects, and distractors.

02

Generate & filter

Keep successful visual-interactive trajectories and reject invalid or non-deployable behavior.

03

SFT alignment

Align Qwen3-VL-8B with the MCP tool schema and long-horizon action format.

04

Online RL

Optimize exploration, recovery, and timely clarification from environmental feedback.

REAL-Bench

241 tasks that isolate four real-world capabilities.

Select a task family to connect its evaluation target with an illustrative trajectory. The full task count stays visible so the benchmark composition remains legible.

241total task instances72 FDP · 56 FODP · 48 FDO · 65 SUL

FDP / ACTIVE EXPLORATION

Object and furniture distractors make visual search necessary.

The agent must first identify the right source and destination among similarly named or visually similar furniture, then complete a cross-receptacle rearrangement.

What this split evaluates

Evidence must precede the pick-and-place action.

  1. 01
    Observe

    Separate similar furniture from the RGB view.

  2. 02
    Act

    Explore and ground the source and destination.

  3. 03
    Verify

    Complete the correct cross-receptacle placement.

Simulation results

RL restores exploration and proactive interaction.

Use the task selector to compare a one-epoch SFT policy, a two-epoch SFT policy, and the final SFT + RL policy across all four REAL-Bench capabilities.

56.9%SUL success

Best interactive-task result

+6.1RL gain on SUL

Over two-epoch SFT

33.9%FODP success

Recovered open-vocabulary exploration

FDP / TOOL ALIGNMENT

SFT establishes a usable tool policy.

One SFT epoch lifts the base model from 0.0% to 45.8%; the second epoch reaches 65.3%.

Open the complete REAL-Bench results table
ModelFDPFODPFDOSUL
Zero-shot prompting
Gemini 3 Pro Preview teacher81.939.350.053.8
Gemini 2.5 Pro66.741.143.849.2
GPT-568.130.447.952.3
GPT-4o30.628.627.140.0
Claude Haiku 4.545.816.152.147.7
Qwen3-VL-235B-A22B-Instruct22.214.314.618.5
Qwen3-VL-8B-Instruct0.00.00.01.5
REAL agent
REAL-8B · SFT 1 epoch45.830.435.436.9
REAL-8B · SFT 2 epochs65.328.643.850.8
REAL-8B · SFT + RL58.333.941.756.9

Failure analysis / 100 episodes

Most remaining errors are visual or procedural, not interface failures.

Manual analysis identifies object confusion as the largest source of failure, followed by missing key actions such as failing to ask for clarification.

  • 25% object confusion
  • 17% key action missing
  • 9% lost memory
Failure mode taxonomy for 100 evaluation episodes, including object confusion, key action missing, and lost memory.
Manual failure taxonomy for the best SFT + RL checkpoint.

Real-world deployment

Zero-shot transfer to a dual-arm mobile robot.

The high-level policy is frozen and runs through the same MCP tool interface as simulation, while specialized physical skills execute the continuous manipulation primitives.

REAL executing a long-horizon mobile manipulation trajectory on the Ark LIFT2 robot.
Physical deployment. The agent prepares the destination, explores candidate objects, asks for clarification, grounds the response, and completes the manipulation task.
78.3%end-to-end success60 physical episodes
63.1%overall SPLpath efficiency
85.3%primitive success600 VLA executions
1.78 cmodometry error30 repeatability trials
01

Prepare

Open the destination receptacle before fetching the object.

02

Explore

Navigate to the source and visually inspect multiple candidates.

03

Disambiguate

Pause and ask the user instead of guessing under ambiguity.

04

Execute

Ground the response, pick the object, return, and place it.

The first three real world execution steps.The next three real world execution steps.
Open real-world evaluation details
TaskSuccessStepsSPLAskVLM latency
FDO80.0%12.267.0%0.9071.1 s
FODP100.0%9.686.9%0.7053.5 s
SUL55.0%13.035.4%0.5580.5 s
Overall78.3%11.663.1%0.7268.4 s

Reference

Citation

If you find REAL or REAL-Bench useful, please consider citing our work.

@inproceedings{mi2026real,
  title     = {Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation},
  author    = {Mi, Boyu and Ma, Mengchen and Yao, Yifei and Gao, Xing and Chen, Junting and Li, Yangzi and Zhu, Zihou and Li, Guohao and Yin, Zhenfei and Wang, Tai and Mu, Yao and Pang, Jiangmiao and Wang, Hanqing},
  booktitle = {European Conference on Computer Vision},
  year      = {2026}
}