Knowing Where to Look

“Find a roll of blue duct tape”, hard viewpoint: two exploration steps, opening the drawer, and the pick. Bottom-left inset: the belief map during exploration. Third-person time-lapse.

Belief, step by step

The same duct-tape run as above, decomposed. The robot starts from the hard viewpoint, facing the snack area; the tape is inside a closed storage drawer on the office side. Pick a step to see the belief that selected it.

The robot stands idle in the middle of the room, facing the snack side.

Step 1, explore: the robot turns toward the office side while the belief-map inset shows one red high-belief blob.

Step 2, explore: the robot drives further into the office and the belief-map inset now shows two mapped surfaces.

Step 3, open drawer: the robot stands at the white storage drawer and pulls the top drawer open.

Step 4, pick blue tape: the drawer is open with the roll of blue tape visible and the robot reaches in.

Belief map after the initial observation: a single wedge of floor and wall with one small red high-belief blob at the snack-side drawer.

after the first observation

Action initial belief

The robot starts facing the snack area. Its initial belief places mass on the snack-side storage drawer, alongside geometric mass on the still unexplored office (not visualised, for clarity).

Belief map after the first explore: two viewing wedges now cover the left wall, the desk with a monitor, and the right side, with red blobs at the drawer unit and the far cabinet corner.

after explore 1

Action explore_at

The observation action toward the unexplored region carries the largest expected revealed belief mass, so the robot takes it, revealing part of the office.

Belief map after the second explore: one contiguous swept floor region with the desk fully mapped and the strongest red blob at the office drawer unit.

after explore 2

Action explore_at

The updated belief drives a second observation action; belief now concentrates on the two container blobs, the strongest at the office storage drawer.

The belief map before opening: the same map as after the second explore, with the largest red blob at the office drawer unit.

before opening (same map)

Action open_drawer

An open action on the office storage drawer now carries the largest expected revealed belief mass, so the robot executes the open_drawer skill, revealing the duct tape.

The last belief map shown, after the second explore, with the largest red blob at the office drawer unit; the paper does not show the belief after the drawer is opened.

last map shown (after explore 2)

Action pick_from_drawer

The observation from the opened drawer updates the memory, which now holds an object satisfying the query, so the loop ends with a pick and the robot retrieves the tape. The paper shows no belief map after this update.

Start

Arrow keys move between steps.

Action selection at every step: $a^{*} = \underset{a}{argmax} M_{rev} (a)$ , $M_{rev} (a) = \sum_{i \in F (a)} σ (B_{i})$ a* = argmax_a M_rev(a), M_rev(a) = Σ_i∈F(a) σ(B_i), where F(a) is the set of not-yet-observed cells the action would reveal and σ converts log-odds to probability.

Three red voxel renderings of the office desk, bookshelf and drawer: a dense occupancy belief, a blocky semantic belief concentrated on the desk and drawer, and their combination, with a low-to-high belief colour bar. — Left to right, for *“find the stapler”*: occupancy belief G, semantic belief S, and their sum, the combined belief B.

Abstract

When searching for an object, humans naturally combine semantic intuition — reasoning about where the object is likely to be stored — with geometric awareness of what parts of the space are visible, occluded, or unexplored. Effective robot object search requires the same joint reasoning, yet existing methods rely on only one: geometric methods maintain principled spatial beliefs but lack deep semantic understanding, while language-model based methods leverage commonsense knowledge but maintain no explicit belief over space. We argue that jointly reasoning over semantic and geometric evidence in an explicit 3D belief enables human-like interactive object search that outperforms either modality alone. To test this claim, we build a system that accumulates per-object VLM plausibility scores and geometric occupancy evidence via Bayesian log-odds filtering into a unified 3D voxel belief field. High-belief regions in the combined belief field drive action selection at each step: taking a closer look from a different viewpoint, navigating toward an unexplored region of the workspace, or physically interacting with a closed cabinet. We deploy this system end-to-end on a bimanual mobile humanoid and find that it outperforms geometry-only search and LLM-based planning methods.

How it decides

Given “find the stapler”, the robot opens a drawer, explores, looks at the desk and picks the stapler from the table, with an observation photo taken after each action. — **Belief-driven interactive object search.** Given the query *“find the stapler,”* SSF maintains a combined semantic-geometric belief field that drives action selection. The robot takes observations (blue) after each action (green), updating its belief until the target is retrieved.

Object-based memory. SSF keeps a data-association filter over 3D object hypotheses; each holds a partial point cloud, a bounding box, a semantic feature vector, the images and masks of its detections, and a text label and caption. The memory answers whether the target has already been seen and also represents the larger objects (cupboards, cabinets, drawers) that might contain it. Whenever a new view is associated with an object o, the VLM is asked for the probability p_o that o contains, conceals, or is an object satisfying the query. Because such outputs are unreliable, they are treated as observations and aggregated by a Bayes filter over per-object log-odds, ℓ_o := ℓ_o + log(p_o/(1 − p_o)), starting from an uninformed prior of 0 and with p_o clamped to [10⁻³, 1 − 10⁻³] so that no single answer can eliminate or confirm a hypothesis. Only substantially different viewpoints (at least 0.15 m in base position, 15° in yaw, or 10° in head tilt) contribute, to limit correlated observations.

Belief map. The belief map is a 3D voxel grid. Each cell i holds a separate Bernoulli belief that an object satisfying the query overlaps it: these are per-cell beliefs, not a distribution over the location of one particular object. Geometric and semantic evidence are treated as conditionally independent, so in log-odds and with a uniform prior the two simply add, $B_{i} = G_{i} + S_{i}$ B_i = G_i + S_i. G is a persistent occupancy grid updated by raycasting each depth image; each cell takes a fixed geometric log-odds by observation state: −6.91 when observed free, −4.00 when occupied by something other than the target, −2.20 when unexplored, and −0.85 when it was occluded in some observation and has not yet been seen. S_i is the largest ℓ_o among the memory objects that overlap cell i, and 0 where no object does. Because the interior of a closed container keeps positive belief, the field can drive manipulation and not only viewpoint changes.

Action selection. If the memory already contains an object satisfying the query, the robot picks it and stops. Otherwise high-belief cells are clustered into connected components, and each selected cluster proposes one candidate: an observation from a new viewpoint, or, if the cluster lies inside a closed object in memory, an opening action on that object. Every candidate a has a footprint F(a), the not-yet-observed cells it would reveal (the cone a viewpoint would see, or the interior an opening would expose), and the robot greedily executes the candidate with the highest expected revealed belief mass, the sum of σ(B_i) over that footprint. If no remaining footprint could still contain the target, the search ends in failure. The candidates are dispatched as skills: look_at, explore, open_cabinet, open_drawer, and the four pick_* skills for the final retrieval.

System diagram: the query and RGB-D observation feed an object-based memory and a belief map that adds occupancy and semantic belief; the action module clusters high-belief cells, generates candidate actions, and dispatches the argmax skill, whose new observation closes the loop. — **System overview.** Given a natural-language query q (e.g., *“find the stapler”*) and an RGB-D observation *I_t*, SSF maintains two persistent structures: the object-based memory O, which tracks object hypotheses and their per-object semantic log-odds, and the belief map B, which fuses geometric evidence G from occupancy raycasting with semantic evidence S derived from the memory's VLM scores. The action module spatially clusters high-belief cells in B, generates one candidate action per selected cluster, and selects the action a* with the highest expected revealed belief mass M_rev. The corresponding skill is dispatched from the skill repository (here, `look_at`). The observation *I_t+1* returned by the action updates O and then B.

Implementation: a Rainbow Robotics RB-Y1 bimanual wheeled humanoid with a pan-tilt ZED 2i head camera and a base-mounted Livox LiDAR; CLIP ViT-H/14 for features, SAM3 for segmentation, Gemini 3.1 Pro as the VLM.

Same task, three searchers

The action logs are the complete recorded runs; the clips are the edited video comparisons (third-person time-lapse) and run each method to its end state: in Example 1 the geometric baseline is shown eventually reaching the banana, in Example 2 the LLM planner is cut at the time limit. The paper counts a retrieval after 15 minutes as a failure (†). Chips are the recorded actions in order: E explore, L look, O open, P pick. The count excludes the terminal pick.

Visible: book in bookshelf

(a) · hard viewpoint · the robot starts facing away from the office side

OursELP2 search actions found
Geometry-onlyELP2 search actions found
LLM plannerLOOLLOLELP9 search actions found after budget †

For the book inside the bookshelf from the hard viewpoint, the geometry-only method uncovers the unseen volume first, by exploring and looking inside the bookshelf, as it dominates the occluded space, and SSF follows the same sequence; the LLM planner struggles with this initialisation, favouring interacting with and looking at objects already in its scene graph, and exhausts the search budget.

“Find the black book”, three runs side by side. Our system and the geometric baseline reach the bookshelf; the semantic baseline keeps circling the already-seen part of the room until its panel turns red with a time-limit message. Our System = Ours · Geometric Baseline = geometry-only · Semantic Baseline = LLM planner · third-person time-lapse.

Clutter: banana in fruit basket

(b) · easy viewpoint · most of the workspace is visible from the start

OursLLP2 search actions found
Geometry-onlyEEOLLOELLLLP11 search actions found after budget †
LLM plannerLLP2 search actions found

For the banana in the fruit basket from the easy viewpoint, both SSF and the LLM planner identify the small fruit basket as a plausible target location; the geometry-only method instead tries to uncover the occluded space, performing 11 search actions before reaching the basket, well beyond the search budget.

“Find the banana”, three runs side by side. Our system and the semantic baseline drive to the snack table and report the banana found early in the run, while the geometric baseline keeps wandering the room before it finally succeeds. Our System = Ours · Geometric Baseline = geometry-only · Semantic Baseline = LLM planner · third-person time-lapse.

Container: duct tape in drawer

OursEOP2 search actions found
Geometry-onlyEEOLLLLLLOP10 search actions found after budget †
LLM plannerLLLEOLEOP8 search actions found after budget †

For the duct tape in the drawer from the easy viewpoint, both baselines exhaust their search budget: the LLM planner investigates visible scene-graph objects, including desks and another storage drawer, before opening the correct drawer, and geometry-only spends actions exploring and observing occluded regions before selecting the target drawer; SSF explores and then opens the correct drawer, requiring only two actions.

Action log only: no baseline footage for this configuration.

Show the paper figure

Action sequences across matched search configurations. Each panel compares the three methods on the same target placement and initial viewpoint. Cells show actions in execution order; the right column counts search actions, excluding the terminal pick. Complete recorded sequences are shown, with a dagger marking picks after the 15-minute evaluation budget.

Results

14/16
Ours
8/16
Geometry-only
7/16
LLM planner

16 trials per method · 15-minute budget · 4 settings × 2 object categories × 2 viewpoints

Quantitative results with a 15-minute search budget. SR (success rate, higher is better) is the number of successful retrievals over total trials; each setting comprises 2 object categories × 2 viewpoints. NA (number of actions, lower is better) is averaged over successful trials only.
Setting	SR ↑	NA ↓	SR ↑	NA ↓	SR ↑	NA ↓
	Ours		Geometry-only		LLM planner
Visible	4/4	2.5	4/4	2.5	2/4	2.0
Clutter	4/4	4.0	2/4	6.0	3/4	3.0
Container	4/4	3.5	2/4	4.5	1/4	5.0
Floor	2/4	5.0	0/4	--	1/4	4.0
Overall	14/16	3.6	8/16	3.9	7/16	3.1

Paired outcomes were compared with a two-sided exact McNemar test. Against geometry-only, SSF succeeds in six configurations where the baseline fails, with no configurations where the baseline succeeds and SSF fails (p = 0.031). Against the LLM planner, the corresponding counts are seven and zero (p = 0.016).

Ours: of 16 trials, 14 succeed (4 visible, 4 clutter, 4 container, 2 floor) and 2 floor trials exceed the budget because revealed mass ranks large container interiors over the small floor pocket. — SSF, 16 trials.

Geometry-only: 8 successes (4 visible, 2 clutter, 2 container) and 8 budget failures (2 clutter, 2 container, 4 floor), with no semantic prior it opens the largest occluders first. — Geometry-only baseline, 16 trials.

LLM planner: 7 successes (2 visible, 3 clutter, 1 container, 1 floor) and 9 failures split into budget exceeded, wrong object picked and no free-space reasoning, 3 each. — LLM planner, 16 trials.

Outcomes and failure modes, one panel per method: SSF, the geometry-only baseline and the LLM planner, with 16 trials per method, excluding the hard-floor setting. Flows connect trial outcomes to task settings and, for failed trials, to failure categories. Flow widths represent trial counts; percentages are computed over the 16 trials shown for each method.

Failure modes

The geometry-only baseline lacks semantic guidance and spends the search budget exploring regions or interacting with containers unrelated to the target; it succeeds only when the target is clearly visible or happens to lie in the largest occluded container. The LLM planner favours investigating already-observed objects in its scene graph over exploring unseen space, and struggles under hard-viewpoint initialisation: searching for the book in the bookshelf while facing the kitchen area, it opens the snack drawer and the kitchen cabinet, then inspects both tables, and times out before reaching the bookshelf. For retrieval it relies on scene-graph nodes whose labels match the query, produced by an open-vocabulary detector and verified by the VLM; from a distant view an orange textbook can be labelled a snack box, leading to a wrong pick. SSF's two failures are the glue-bottle trials of the floor setting: the bottle lies in a small occluded pocket of floor beside the office drawer, which overlaps no object in memory, receives no semantic evidence, and carries only a small geometric prior. The fused belief is not wrong about where the target could be, but because the expected revealed mass sums belief over a footprint, actions revealing large belief regions outrank those exposing smaller ones supported mainly by geometric evidence.

Limitations

Greedy action selection is used as a practical heuristic, choosing at each step the action with the highest expected revealed belief mass. In the two floor failures, a small occluded pocket of floor beside the office drawer receives little belief mass, so actions that reveal larger regions are chosen first.

More runs

Third-person time-lapse. In the four inset runs a yellow star marks the target.

“Find the stapler”: the robot looks over the cluttered office desk, grasps the stapler and turns to the camera.

Clips: stapler · banana · peanut butter · black book · blue duct tape (MP4).

Skills

Eight photographs of the robot performing open_cabinet, open_drawer, explore, look_at, pick_from_cabinet, pick_from_drawer, pick_from_shelf and pick_from_table. — **Skill library.** `open_cabinet` and `open_drawer` are learned via behaviour cloning; `explore` navigates to frontier viewpoints; `look_at` repositions the base, torso, and head to uncover occluded voxels. The four `pick_*` skills (bottom row) retrieve objects from cabinets, drawers, shelves, and tables.

open_cabinet: Learned by behaviour cloning with Diffusion Policy, tailored to the specific container.
open_drawer: Learned by behaviour cloning with Diffusion Policy, tailored to the specific container.
explore: Navigates to frontier viewpoints that reveal unexplored space.
look_at: Repositions base, torso and head to uncover occluded voxels.
pick_from_cabinet: Adapted TipTop, an open-vocabulary manipulation system, parameterised over the torso and the 7-DOF right arm.
pick_from_drawer: Adapted TipTop.
pick_from_shelf: Adapted TipTop.
pick_from_table: Adapted TipTop.
navigation: A* on a 2D occupancy map down-projected from the 3D voxel grid, executed by a feedback controller; used inside every other skill to move the base into place.

open_drawer: the robot approaches the drawer unit and pulls the top drawer open, revealing the blue tape.

open_cabinet: the robot opens the pantry cabinet doors, reaches in and retrieves the peanut-butter jar.

Setup and platform

Top: the robot at the Easy, Hard (Office) and Hard (Snack) initial viewpoints; bottom: visible, cluttered and container tasks in the snack and office areas. — **Experimental setup and task settings.** Top: easy and hard initial viewpoints. The easy viewpoint exposes most of the workspace; hard viewpoints initially face away from the target object's office or snack area. Bottom: examples of visible, cluttered, and container tasks in both areas.

All experiments run in a real office of about 87 m³ (6.04 × 4.85 × 2.97 m), divided into a kitchen pantry area and an office-supply area. The kitchen contains a snack table (Cheez-Its, a fruit basket, crackers), a pantry cabinet of breakfast items (peanut butter, jam, tea), and a storage drawer with a snack box on top. The office contains a desk cluttered with supplies (glue, tape, staplers, binder clips, paper), a bookshelf with books and folders, and a storage drawer holding duct tape with supplies on top. An empty storage drawer separates the two areas. Each setting is evaluated under an easy and a hard initial viewpoint, for 48 real-robot trials in total.

Visible: The target is in plain sight (a book on the bookshelf; a Cheez-It box on the snack desk), requiring only retrieval.
Clutter: The target is occluded within a pile of similar objects (a stapler among office supplies; a banana inside a fruit basket), requiring a closer look and visual disambiguation.
Container: The target is inside a closed container (duct tape in the storage drawer; a peanut-butter jar in the pantry cabinet); the robot must identify the correct container and open it.
Floor: The target has fallen near a semantically related surface rather than into standard storage (a glue bottle from the office desk; a pear from the snack table). The system must place semantic mass on that surface so the object is incidentally observed when the robot orients toward it.

Baselines

Geometry-only. GenMOS adapted to the full action set, with its POUCT planner replaced by greedy action selection. It keeps the multiplicative voxel belief update, initialised by occupancy state (2.0 occluded, 0.3 unexplored, 1.0 occupied); raycasting updates occupancy, VLM detections of the target multiply the corresponding voxel beliefs by 10³, and otherwise visible voxels decay by 10⁻³. Retrieval triggers when any voxel belief exceeds 10⁶; the search otherwise ends when the time budget is exhausted.

LLM planner. Follows CuriousBot: a semantic scene graph of detected objects and VLM-estimated spatial relations from RGB-D observations. Given the graph, the query and the action history, the LLM selects an object to investigate, an observation action to explore unseen space, or a pick for a visible, VLM-verified target. Exploration was added so the planner can act from initial views with no plausible target locations; following VLFM, frontiers are annotated with visible scene-graph objects and the LLM picks the one most relevant to the query. Search ends on retrieval or when the step or time budget runs out.

Hardware and perception

The RB-Y1 wheeled humanoid standing in the office with its head camera and wrist cameras. — The Rainbow Robotics RB-Y1 bimanual wheeled humanoid in the workspace.

The RB-Y1 has parallel-jaw grippers; a base-mounted Livox LiDAR provides odometry, a head-mounted ZED 2i stereo camera is actuated in pan and tilt, and wrist-mounted monocular ZED cameras on each arm support the learned cabinet- and drawer-opening policies. CLIP ViT-H/14 provides semantic features, SAM3 the segmentation masks, and Gemini 3.1 Pro the VLM judgements; with 3D sensors on board no depth-recovery model is needed.

Because the semantic belief depends on VLM outputs, their repeatability was measured: querying Gemini on 10 frames (5 per environment) 10 times each, 54% of per-object scores were identical across runs and 91% varied by at most 0.1, with a near-deterministic ranking of objects, and the median latency was 10 s per scoring call.

Full video

Here's the 3 min video.

Setup, the duct-tape run, belief fusion, architecture, baselines, two side-by-side comparisons, more runs, and a summary. Silent, 2:56.

Transcript

On-screen text, verbatim, with the time it appears; each chapter ends with a description of the visuals. The video has no narration.

1. Setup — 0:00

0:00.0: We set up a scene
≈0:01.0: with food items on one side
≈0:02.3: and office suppplies on the other (sic)
≈0:03.8: We ask the robot to / search for a specific object
0:06.9: A 2×4 grid of robot photos labelled open_cabinet, open_drawer, explore, look_at, pick_from_cabinet, pick_from_drawer, pick_from_shelf, pick_from_table; caption: using these skills
Visual: A washed-out wide view of an office-like room in which first the left, food side (pantry cabinet, snack table, storage drawer) and then the right, office side (desk with monitor, bookshelf, storage drawer) is highlighted in full colour, followed by a grid of eight photographs of the wheeled humanoid performing one skill each.

2. “find a roll of blue duct tape” (hard viewpoint) — 0:09

0:09.3: “find a roll of blue duct tape”
0:14.4: Robot run begins, no text
≈0:17.0: Step 1: Explore; picture-in-picture panel at bottom left titled Belief Map
≈0:20.5: Step 2: Explore
≈0:22.8: Step 3: Open Drawer
≈0:31.0: Step 4: Pick blue tape
Visual: A person places the tape in the white storage drawer and leaves. Starting from the hard viewpoint facing the snack side, the black-and-white wheeled humanoid turns and drives toward the office side while a small belief-map inset grows from a single red blob to a point cloud of the office furniture, then it rolls up to the white storage drawer, pulls the drawer open with one arm, reaches in and lifts out the roll of blue tape.

3. Belief fusion — 0:39

0:39.2: Our system uses a combination of semantic and geometric belief
≈0:41.0: Query: “find the stapler”
≈0:43.5: Observation; panels RGB-D and Depth
≈0:46.3: Occupancy Belief, B_t^occ; colour bar low belief … high belief
≈0:48.0: Semantic Belief, B_t^sem
≈0:49.5: Occupancy Belief ⊕ Semantic Belief = Combined Belief, with B_t^occ, B_t^sem, B_t
Visual: A sequence of white slides shows the query card, a side-by-side RGB image and colour depth map of the office desk, then red voxel renderings of the desk, bookshelf and drawer: a dense occupancy belief, a blocky semantic belief concentrated on the desk and drawer, and their combination, before the slide clears.

4. Architecture — 0:52

0:52.3: Query: “find the stapler”
≈0:53.3: Observation, RGB-D, Depth, I_t
≈0:54.3: O_t, Object-Based Memory
≈0:55.3: Belief Map: Occupancy Belief ⊕ Semantic Belief = Combined Belief, B_i^G, B_i^S, B_i
≈0:56.3: Action Module: Generate Candidate Belief Clusters; Generate Candidate Actions (look_at(desk) a₁, look_at(shelf) a₂, …, open_drawer a_K; a* = argmax_a M_rev(a)); Skill Repository (explore, open_drawer, pick_from_drawer, pick_from_shelf, pick_from_table, pick_from_cabinet, look_at, open_cabinet)
≈0:57.3: Executed Action, a*
≈0:58.3: Feedback arrow I_t+1
≈0:59.3: not found (arrow back to the observation) and found (arrow to a robot icon)
Visual: The full system diagram assembles piece by piece on white: query card, RGB-D observation, object-based memory, the pink belief-map box combining occupancy and semantic beliefs, the blue action module with candidate clusters, candidate actions and the skill repository, and finally the executed-action robot photo with found and not-found feedback arrows.

5. Baselines — 1:02

1:02.2: We compare our method against 2 baselines
≈1:04.3: A geometric baseline that maintains / a 3D occupancy-based belief
≈1:06.3: and a semantic baseline that uses / LLM common sense over observed objects.
Visual: Three plain black-on-white text slides introduce the two baselines.

6. Example 1: banana — 1:09

1:09.6: Example 1; Query:“find the banana”; centre photo captioned Start Position
≈1:10.3: Insets Initial Observation and Object Placement; a yellow star marks the target
≈1:13.8: Three panels: Our System, Geometric Baseline, Semantic Baseline
≈1:15.5: Our System: Object Found
≈1:17.5: Semantic Baseline: Object Found
≈1:19.0: Because the geometric baseline lacks contextual reasoning, / it fails to infer that the banana is most likely on the snack table.
≈1:24.0: Geometric Baseline: Object Found
Visual: A setup slide shows the robot's start position in the centre of the room with the banana placed in a tray on the snack table, then three time-lapse panels run side by side in which our system and the semantic baseline drive to the snack table and report the banana found within seconds while the geometric baseline keeps wandering the room before it finally succeeds.

7. Example 2: black book — 1:26

1:26.8: Centre photo captioned Start Position
1:27.5: Example 2; Query:“find the black book”
≈1:29.3: Insets Initial Observation and Object Placement; yellow star on the bookshelf
1:31.8: Three panels: Our System, Geometric Baseline, Semantic Baseline
≈1:35.3: Our System: Object Found; ≈1:35.5 Geometric Baseline: Object Found
≈1:41.0: The semantic baseline fails to effectively explore / unseen parts of the scene, causing it to exceed the time limit.
≈1:50.0: Semantic Baseline: Time Limit Exceeded
Visual: The setup slide shows the robot starting near the camera with the black book placed on the bookshelf at the right, then in the three side-by-side runs our system and the geometric baseline reach the bookshelf and report the book found while the semantic baseline keeps circling the already-seen part of the room until its panel turns red with a time-limit message.

8. More examples — 1:52

1:52.3: More examples on / varied scene configurations
1:55.1: “find the stapler”; star on the cluttered office desk at the left
2:05.3: “find the banana”; star on the centre desk
2:22.3: “find the jar of peanut butter”; star on the pantry cabinet; 2:28.4 cuts to a close view of the cabinet
2:37.0: “find the black book”; star on the bookshelf at the right
Visual: Four inset time-lapse runs in differently arranged rooms, each opening with the query and a yellow star on the target's location: the robot looks over a cluttered desk and picks up the stapler, drives across the room to a desk and lifts the banana, rolls to the pantry cabinet, opens both doors and takes out the peanut-butter jar, and finally crosses to the bookshelf and reaches for the black book.

9. Summary — 2:47

2:47.5: We presented a belief-driven interactive object search system / that fuses VLM-guided priors and geometric evidence from occupancy / raycasting into a unified 3D voxel field.
≈2:49.8: We deploy end-to-end on a bimanual wheeled humanoid with learned / interaction skills, open-vocabulary pick-and-place, and mobile navigation.
≈2:51.5: We achieve a 87.5% success rate outperforming geometry-only (50%) and / LLM-based planning methods (44%).
≈2:53.5: We show that jointly reasoning over semantics and geometry in an explicit 3D / voxel belief field enables more efficient interactive object search than either / modality alone.
Visual: Four left-aligned text bullets summarising the contributions appear one after another on a white slide and remain until the video ends at 2:56.

BibTeX

@misc{agarwal2026knowing,
  title  = {Knowing Where to Look: Interactive Object Search via Spatial-Semantic Fusion},
  author = {Agarwal, Aditya and Cai, Lucy and Lu, Kelly and Manderson, Travis and Roy, Nicholas and Lozano-P{\'e}rez, Tom{\'a}s and Kaelbling, Leslie Pack},
  year   = {2026}
}