Seeing is Believing

Planning to Perceive with Foundation Models and Act Under Unknowns

Linfeng Zhao ^1,2

Willie McClinton ^3,*

Aidan Curtis ^3,*

Nishanth Kumar ³

Tom Silver ⁴

Leslie Pack Kaelbling ³

Lawson L. S. Wong ²

⁴

^*Equal contribution

The problem

Task-level planning is critical to long-horizon mobile manipulation.
Standard planning methods typically assume complete knowledge of the environment.
A robot entering a new room does not see everything: it must discover, perceive, and interact with objects while resolving uncertainties.
Robots must plan not only manipulation actions, but also information-gathering actions.

A Spot robot in a room next to a drawer cabinet with a red cup and a bowl on top, a table, and a cardboard bin on the floor — **Entering a new room: the cup and the bowl are visible, but not what is inside the closed drawers.** Frame from the Drawer Cleaning video below.

Spot robot in front of a closed drawer cabinet — **Much of what a task depends on is not in the image.** Left and middle: real frames of a Spot robot from the paper's drawer-cleaning example (Fig. 2). Right: an illustration of the two choices, not a measured result. Measured results are on the blog.

The same drawer opened by the robot, with a blue block inside — **Much of what a task depends on is not in the image.** Left and middle: real frames of a Spot robot from the paper's drawer-cleaning example (Fig. 2). Right: an illustration of the two choices, not a measured result. Measured results are on the blog.

Core design

Each fact about an object is known true, known false, or unknown. Unknown is not false: a closed drawer is not an empty drawer.
When the goal depends on an unknown fact, the planner adds an action to look, such as opening the drawer.
Vision-language models check facts in the camera images and translate the goal from text.

Illustration of the method

The abstraction: a symbolic belief

Each predicate is stored as two known-predicates (K-fluents).

`K_P`	`K_¬P`	Belief about Empty(drawer)
1	0	known true
0	1	known false
0	0	unknown

Unknown facts decide when to look

If the goal depends on a fact that is still unknown, the plan first looks: open the drawer, or view the cup from above. The robot plans as if the look gives the answer it needs, and replans if it does not.

The loop: observe, update, plan, execute

Observe Images and sensor inputs from the robot.
Update A VLM points to objects and evaluates predicates. Only answers it can give with confidence change the belief; known facts stay known.
Plan A symbolic planner (Fast Downward) plans over the belief, toward a goal translated from text by a language model.
Execute Run the first action as a robot skill.

If an observation differs from the outcome the plan assumed, replan from the updated belief.

Unknown facts are tracked explicitly, and they decide when the robot looks before it acts. Left: how one predicate holds three values, and how unknown facts lead to looking. Right: the runtime loop (Algorithm 1 in the paper).

Interactive · generated by the planner

Planning to look: an observation can come back either way, and the plan branches on it. The belief-space graph for the two-cup task, keeping every shortest plan. Because every state of knowledge is explicit, the same graph can check an agent’s progress step by step and give it an extra signal, such as a reward, for what it has found out. Drag to pan, scroll to zoom.

Connection to the major directions

POMDPs: plan in the space of beliefs (Kaelbling et al., 1998). We make the belief symbolic and object-centric.
Belief-space task and motion planning: mix acting and looking, plan online, replan (2012, 2013). Our belief update comes from vision-language models.
Classical replanning with knowledge fluents: known-true and known-false facts let a classical planner decide when to look (Bonet and Geffner, 2011).
Task and motion planning: the tradition our planner builds on (Garrett et al., 2021; Curtis et al., 2024).
Agentic robots and hierarchical systems: models ground instructions in skills (SayCan), write robot programs (Code as Policies) and use feedback (Inner Monologue). Ours is a hierarchy too: models estimate facts, a planner decides, skills act.

On a real Spot robot

Real robot · 4× speed

Empty Cup Removal

The robot cannot tell from the front whether a cup is empty. It looks from above, finds the yellow cup filled, replans, and moves the empty red cup to the bin. Played at 4×; model wait time is cut; the on-screen labels were added in editing.

Real robot · 4× speed

Drawer Cleaning

The robot opens the drawer, discovers the objects inside, and removes them. Played at 4×.

Each video is one demonstration of the system, not a success-rate measurement.

Cite this work

Linfeng Zhao, Willie McClinton*, Aidan Curtis*, Nishanth Kumar, Tom Silver, Leslie Pack Kaelbling, and Lawson L. S. Wong. Seeing is Believing: Belief-Space Planning with Foundation Models as Uncertainty Estimators. arXiv:2504.03245, 2025.

* Equal contribution.

Paper (arXiv) · PDF

@misc{zhao2025seeing,
  title = {Seeing is Believing: Belief-Space Planning with Foundation Models as Uncertainty Estimators},
  author = {Linfeng Zhao and Willie McClinton and Aidan Curtis and Nishanth Kumar and Tom Silver and Leslie Pack Kaelbling and Lawson L. S. Wong},
  year = {2025},
  eprint = {2504.03245},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url = {https://arxiv.org/abs/2504.03245}
}

Acknowledgments

We thank Jenny Barry, Tomás Lozano-Pérez, Will Shen, Aditya Agarwal, Ben Zandonati, Xiaolin Fang, Jiayuan Mao, and all members of the MIT Learning and Intelligent Systems (LIS) group and the Watch, Understand, Do team at the RAI Institute (formerly the Boston Dynamics AI Institute) for helpful discussions and support.