Scenario Lab

Initial state

You are positioned at the far end of a narrow hallway, roughly 4 meters long, facing toward the front door. Morning light filters through a frosted sidelight window beside the door, casting soft diffused shadows. The floor is a dark laminate. Directly ahead, scattered across the floor in a roughly 1.5-meter-wide pile, are seven shoes — no two are a matching pair. One is a single child's rain boot (yellow, muddy sole). One is an adult men's dress shoe (black, left foot). Two are mismatched sneakers — one adult women's running shoe (pink, right foot) and one teenage-sized basketball shoe (white with a blue swoosh, right foot too). One is a fleece-lined slipper (grey, right foot). One is a gardening clog (green crocs-style, no foot indicator, soil stuck to the sole). One is a small toddler sandal (velcro strap, left foot). None are paired. A wicker shoe rack stands against the left wall, empty except for one lonely matching pair of adult women's flats already racked. The rack has four cubbies. A low wooden bench is against the right wall; a reusable grocery bag is sitting on top of it, partially obscuring one sneaker until you move forward. The hallway smells faintly of mud and rain. Ambient sound is a TV from a distant room and the muffled sound of someone upstairs moving around. The pile partially blocks forward navigation toward the front door.

Goal state

The hallway floor is clear enough to walk through without stepping on or kicking any shoe. Shoes are arranged in a way that a reasonable household member would consider 'tidied.' Because no complete pairs are visible, exact pairing is not a success criterion — grouping by rough size or type is acceptable. The shoe rack and bench are both valid placement surfaces. The grocery bag must not be buried under shoes. No shoe is left in the direct navigation path between the robot's current position and the front door.

Objects involved

Name	Descriptor	Role
toddler sandal	small, velcro strap, beige, left foot	target
child's rain boot	yellow rubber, muddy sole, single boot	target
men's dress shoe	black leather oxford, left foot only	target
grey slipper	fleece-lined, grey, right foot only, soft deformable	target
gardening clog	green crocs-style, soil on sole, ambiguous foot	target
pink running sneaker	adult women's size, pink mesh, right foot	target
basketball shoe	teen-sized, white with blue swoosh, right foot	target
wicker shoe rack	natural wicker, four cubbies, left wall	tool
wooden bench	low, dark wood, right wall	tool
reusable grocery bag	green canvas, partially full, handle visible	obstacle
adult women's flats	navy blue, already racked as matched pair	distractor

Expected actions

1. approach the shoe pile slowly while scanning for the partially occluded sneaker behind the grocery bag 5s
2. slide the grocery bag to the far end of the bench to fully reveal the hidden sneaker 6s
3. pick up the toddler sandal from the floor using the right gripper 5s
4. place the toddler sandal on the bench surface toward the wall side, away from navigation path 4s
5. pick up the child's rain boot by the shaft using the left gripper 5s
6. place the rain boot upright on the lowest cubby of the shoe rack next to the left wall 5s
7. pick up the adult men's dress shoe using the right gripper 4s
8. place the dress shoe on the second cubby of the shoe rack 4s
9. pick up the grey slipper using the left gripper 4s
10. place the grey slipper on the bench surface next to the toddler sandal 4s
11. pick up the green gardening clog using the right gripper, adjusting grip to avoid the soil-covered sole contacting the gripper pad 6s
12. place the gardening clog on the floor against the wall directly under the shoe rack, sole facing inward 5s
13. pick up the pink running sneaker using the left gripper 4s
14. place the pink running sneaker on the third cubby of the shoe rack 4s
15. pick up the basketball shoe using the right gripper 4s
16. place the basketball shoe on the fourth cubby of the shoe rack 4s
17. retreat slightly and visually confirm the navigation path between current position and front door is fully clear 5s

Narration script

00:00 I am at the far end of the hallway facing the front door. The floor ahead is covered in seven unmatched shoes. Morning light through the sidelight window is soft but enough to see clearly. I can hear the TV somewhere behind me and movement upstairs.

00:05 I approach the pile slowly. I notice the grocery bag on the bench is sitting on top of something — a sneaker is partially hidden behind it. I need to move the bag before I can account for everything on the floor.

00:10 I slide the grocery bag toward the far end of the bench. The hidden pink sneaker is now fully visible. I now have a complete picture of all seven shoes.

00:16 I am not going to attempt to pair these — there are no complete pairs here. My goal is to get them off the floor and onto a surface. I pick up the toddler sandal first since it is the smallest and closest to the center of the path.

00:21 I place the toddler sandal on the bench toward the wall side, leaving room on the bench for more items. The navigation path ahead is already slightly clearer.

00:25 I pick up the yellow rain boot by its shaft. The muddy sole is facing away from my gripper. I place it upright in the lowest cubby of the shoe rack.

00:35 I pick up the black dress shoe and place it in the second rack cubby. Three down, four to go.

00:43 I pick up the grey slipper and set it on the bench next to the toddler sandal. I am choosing the bench for softer, indoor footwear and the rack for sturdier shoes.

00:51 The gardening clog has soil on the sole. I grip it by the heel, keeping the dirty sole pointed away. I place it on the floor directly under the shoe rack against the wall — it is too dirty to share a surface with the other shoes.

01:01 I pick up the pink running sneaker and place it on the third rack cubby. One shoe remains on the floor — the basketball shoe.

01:09 I pick up the basketball shoe and place it on the fourth and final rack cubby. The floor is now clear.

01:17 I retreat slightly and scan the path from my position to the front door. The floor is fully clear. The grocery bag is on the bench, untouched in terms of its contents. The shoe rack is full. This feels like a reasonable interpretation of tidy, even without any matched pairs.

Edge cases

The grocery bag actually contains something fragile or heavy — operator cannot see contents — and sliding it could cause tipping; operator must decide whether to slide or lift.
A household member walks down the hallway mid-task and steps over the remaining shoes, inadvertently kicking the rain boot further down the hall away from the pile.
The slipper is so deformable that the parallel-jaw gripper compresses it unevenly and it slips; operator must re-approach and adjust grip angle, possibly using the bench edge as a backstop.
Operator interprets the goal differently and attempts to find mates by searching adjacent rooms, which is out of scope but not obviously wrong given ambiguous instructions.
The soil on the gardening clog crumbles and falls onto the laminate floor during transport, leaving a small debris trail the operator may or may not feel obligated to address.