When a robot cannot see inside the drawer, what should it believe?

Why a robot that treats what it cannot see as absent gives up too early, and what it can keep track of instead.

6 November 2026

When a robot enters a new room, it does not see all the information in the room. Much of what its task depends on is out of view. Is the item in that drawer? Is the cup empty? Did something move after the robot left the room?

A real robot arm in front of a closed drawer. — Real robot, a closed drawer whose contents the camera cannot see.

When the robot cannot see something

Standard planning methods typically assume complete knowledge of the environment, so a planner has to decide what a closed drawer means. If it treats “not seen” as “not there”, a closed drawer becomes an empty drawer, and the robot gives up on a search it could have finished, or reports something as absent when it never looked.

So the question we keep coming back to is simple: when a robot cannot see something, what should it write down?

True, false, or unknown

In Seeing is Believing, the robot keeps a small set of facts about the objects that matter for the task, such as “the item is in drawer 2” or “the cup is empty”. Each fact is true, false, or unknown. Unknown is not the same as false.

A vision-language model fills in these facts from what the camera shows. A classical planner then decides what to do next, including when it is worth opening a drawer or looking from another angle. When what the robot sees contradicts what it expected, it updates the facts and plans again.

The core design, in more detail

Each task predicate is stored as a pair of known facts. For a drawer d1, emptiness becomes KnownEmpty(d1) and KnownFull(d1). If neither holds, emptiness is unknown. This symbolic belief is what the planner reads; it never keeps a probability distribution over world states.

Perception. An open-vocabulary detector points to the objects in view, and segmentation plus depth gives their positions. For each detected object, including ones the robot has just found, the VLM answers yes, no, or unknown for the relevant predicates, in one batched query across the camera views.
Looking as an action. An information-gathering action, such as observing whether a cup is empty, applies only while that fact is unknown. Its outcome is not known in advance, so it is compiled into two deterministic actions, one for each outcome (all-outcomes determinization). The classical planner picks the outcome that helps reach the goal; if the real observation differs, the robot replans.
Goal translation. The VLM also turns the instruction into a goal over these predicates, so a goal like “remove every empty cup” can depend on facts that start unknown.
Execution monitoring. After each action the robot observes again. If the result contradicts the outcome the plan assumed, it updates the belief and replans from there.

Illustration of the method

The abstraction: a symbolic belief

Each predicate is stored as two known-predicates (K-fluents).

`K_P`	`K_¬P`	Belief about Empty(drawer)
1	0	known true
0	1	known false
0	0	unknown

Unknown facts decide when to look

If the goal depends on a fact that is still unknown, the plan first looks: open the drawer, or view the cup from above. The robot plans as if the look gives the answer it needs, and replans if it does not.

The loop: observe, update, plan, execute

Observe Images and sensor inputs from the robot.
Update A VLM points to objects and evaluates predicates. Only answers it can give with confidence change the belief; known facts stay known.
Plan A symbolic planner (Fast Downward) plans over the belief, toward a goal translated from text by a language model.
Execute Run the first action as a robot skill.

If an observation differs from the outcome the plan assumed, replan from the updated belief.

Unknown facts are tracked explicitly, and they decide when the robot looks before it acts. Left: how one predicate holds three values, and how unknown facts lead to looking. Right: the runtime loop (Algorithm 1 in the paper).

Diagram of the method loop: perceive, update true, false and unknown facts, plan, act. — The loop: perceive, update true/false/unknown facts, plan, act.

On a real robot

Five frames of a Spot robot opening a drawer, picking a blue block and placing it in a bin, each with the predicates the VLM evaluated — **The robot opens the closed drawer because emptiness is unknown, then acts on what it finds.** Task: put any object in the drawer into the paper bin. Top: the actions. Bottom: object pointing and the predicates the VLM evaluated after each action. Before the drawer opens, `KnownEmpty(d1)` and `KnownFull(d1)` are both false, so emptiness is unknown and the plan starts with opening the drawer. Fig. 2 of the paper; tap the image for full size.

What changes when a model may say “unknown”?

What happens when a model has to answer yes or no about a drawer it cannot see? And what changes when it may answer “unknown” instead?

Explicit facts can be checked

Because every fact the robot holds is written down, each one can be checked. In simulation we know what is really inside every drawer, so we can tell when the robot reports a hidden fact as known, or reads a visible one wrong.

The same check works as a signal for an agent: it can verify each step, or give an extra reward for finding out what the task needs.

Interactive · generated by the planner

Every state of knowledge is a node, so progress can be checked step by step. The belief-space graph for the two-cup task, keeping the beliefs on every shortest plan. Each node is a symbolic belief (which facts are known true, known false or unknown); each edge is an action, and an observation can branch into the outcomes it may return. A trajectory can be scored against this graph: whether each step moves toward a goal belief, and which unknown facts it resolved. Drag to pan, scroll to zoom.

Results

Measured

Cup Pick-Place

fully observable

Random 0/10
VLM end-to-end 3/10
VLM captioning + LLM 10/10
VLM predicate labeling + LLM 9/10
Seeing is Believing (ours) 10/10

Path efficiency (SPL, 1 = optimal): ours 1.00; best other 0.69

Drawer Cleaning

contents hidden in a drawer

Random 0/10
VLM end-to-end 0/10
VLM captioning + LLM 1/10
VLM predicate labeling + LLM 0/10
Seeing is Believing (ours) 8/10

Path efficiency (SPL, 1 = optimal): ours 0.32; best other 0.04

Sort Weight

boxes must be weighed

Random 0/10
VLM end-to-end 0/10
VLM captioning + LLM 0/10
VLM predicate labeling + LLM 0/10
Seeing is Believing (ours) 7/10

Path efficiency (SPL, 1 = optimal): ours 0.46; best other 0.00

With the contents hidden, only the belief-space planner finishes most runs: 8/10 and 7/10, against at most 1/10 for every baseline. Successful runs out of n = 10 seeds per task and method, in a synthetic environment built from real images. All language and vision models are GPT-4o, zero-shot. Source: Table I of the paper.

Six real photos: open drawer, place green box, place white box, weigh white box, weigh green box, dispose green box — **Sort Weight: the boxes are sealed, so the only way to know which one is empty is to weigh it.** Six of the states the agent passes through; each state is a real photo. The shortest solution takes 14 actions. Fig. 4 of the paper; tap the image for full size.

Model, determinization, and assumptions

The key idea

Turn partial observability into repeated classical planning

The method has one modeling step and one online loop. Keeping these separate makes the role of three-valued predicates precise.

Specify once A symbolic world model

Predicates and goal: Empty(?c), Inside(?x, ?c)
World-changing operators: Open, Pick, Place
Information operators: ObserveEmpty(?c)

Predicate memory Every fact is true, false, or unknown

true false unknown

P = ? when neither K(P) nor K(not P) is stored

All-outcomes determinization Expose both possible observations to a classical planner

ObserveEmpty(drawer)

ObserveEmpty+ assume Empty(drawer) = T

ObserveEmpty- assume Empty(drawer) = F

The planner may choose the useful outcome optimistically. Execution does not: the observation determines which fact is written to memory.

Run online Check each optimistic plan against reality

1PlanCall a classical planner on the determinized model.
2ExecuteRun the next symbolic action as a robot skill.
3Observe and updateUse perception to update only the relevant predicates.
4Check and replanIf reality differs from the assumed outcome, discard the suffix and plan again.

Planning graphs for each task

These are the belief-space graphs the planner works with. Each node is a belief; an observation can come back either way, so the graph branches there.

Two cups: every shortest plan (also on the project page).
Cup emptiness: shortest plan.
Two cups: the belief policy, one branch per observation outcome.
Drawer cleaning: transition graph.
Two-cup pick and place: transition graph.
Sort by weight: transition graph.
Sort by weight: the belief policy.

Questions we are still asking

Which tasks actually need a robot to keep track of what it has not seen, and which can be solved by a simple search rule?
As models get better, which perception mistakes go away, and which stay?
Can we predict how a whole robot system will do from how each of its parts does?

Paper: arXiv. Project page: Seeing is Believing.