Unit Ready · evaluation harness
How wrong is the model, and where
96%accuracy on decided checks
0misses: problem called ready
4false alarms: ready called a problem
7abstained (sent to review)
10.7slatency p50 per frame
73.87slatency p95 per frame
$0.0034list price per frame
$0.041per unit at 12 kept frames
0schema failures (retried)
Per item
| Item | Checks | Real problems | Caught | Catch rate | False alarms | False-alarm rate | Abstain | Accuracy |
|---|---|---|---|---|---|---|---|---|
bed_made | 16 | 6 | 6 | 100% | 0 | 0% | 0 | 100% |
towels_present | 24 | 9 | 9 | 100% | 0 | 0% | 0 | 100% |
bin_empty | 24 | 9 | 9 | 100% | 1 | 12% | 7 | 94% |
surfaces_clear | 24 | 8 | 8 | 100% | 3 | 19% | 0 | 88% |
toilet_paper | 8 | 3 | 3 | 100% | 0 | 0% | 0 | 100% |
Catch rate: of the frames where the item was really not ready, the share the model flagged. A miss is the expensive error: a guest walks into the problem. A false alarm costs a cleaner trip. Abstains are neither; they cost a human a look.
Every wrong answer 4 of 89 decided
b06.jpg · bedroom
FALSE ALARM bin_empty label ready, model said missing at 0.90
Model: The trash can by the window has a liner and appears to contain discarded items.
Staged as: unmade, no towels
b06.jpg · bedroom
FALSE ALARM surfaces_clear label ready, model said missing at 0.95
Model: A used mug and item are left on the bedside table and carpet has visible spots.
Staged as: unmade, no towels
b07.jpg · bedroom
FALSE ALARM surfaces_clear label ready, model said missing at 0.99
Model: Trash and bottles are spilled across the floor and items sit on the desk.
Staged as: unmade, towels on bed, bin overflowing
b12.jpg · bedroom
FALSE ALARM surfaces_clear label ready, model said missing at 0.90
Model: A pillow is discarded on the floor and items are left on the nightstand.
Staged as: hard case: duvet pulled up but lumpy, pillows strewn
7 abstains — the model said unclear
b03.jpgbin_empty (label ready): A black bin is visible on the floor, but its interior cannot be seen clearly.b05.jpgbin_empty (label ready): The trash can by the desk is partially visible but the inside cannot be clearly seen.b10.jpgbin_empty (label ready): The small basket next to the nightstand is in shadow and its interior cannot be seen.b13.jpgbin_empty (label ready): The small wastebasket is visible near the wall, but its interior cannot be seen clearly.w01.jpgbin_empty (label ready): The trash can has a closed lid, so its contents cannot be seen.w02.jpgbin_empty (label ready): The small metal step bin has its lid closed, so the interior cannot be seen.w06.jpgbin_empty (label ready): The trash can is closed with a lid, so its contents cannot be seen.
All frames and answers
| Frame | Room | Staged as | Answers | Latency |
|---|---|---|---|---|
| b01.jpg | bedroom | all ready, grey modern room | bed_made: present 0.95 towels_present: present 0.95 bin_empty: present 0.85 surfaces_clear: present 0.95 | 6.5s |
| b02.jpg | bedroom | unmade, towels on armchair | bed_made: missing 0.99 towels_present: present 0.98 bin_empty: present 0.92 surfaces_clear: present 0.95 | 26.75s |
| b03.jpg | bedroom | made, no towels at all | bed_made: present 0.95 towels_present: missing 0.98 bin_empty: unclear 0.40 surfaces_clear: present 0.90 | 10.79s |
| b04.jpg | bedroom | bin overflowing beside bed | bed_made: present 0.95 towels_present: present 0.95 bin_empty: missing 0.99 surfaces_clear: present 0.90 | 10.62s |
| b05.jpg | bedroom | pizza box, clothes on floor | bed_made: present 0.95 towels_present: present 0.95 bin_empty: unclear 0.40 surfaces_clear: missing 0.99 | 8.22s |
| b06.jpg | bedroom | unmade, no towels | bed_made: missing 0.99 towels_present: missing 0.95 bin_empty: missing 0.90 surfaces_clear: missing 0.95 | 15.15s |
| b07.jpg | bedroom | unmade, towels on bed, bin overflowing | bed_made: missing 0.99 towels_present: present 0.95 bin_empty: missing 0.99 surfaces_clear: missing 0.99 | 45.93s |
| b08.jpg | bedroom | beige blanket is a towel confuser; bin and nightstand bad | bed_made: present 0.85 towels_present: missing 0.95 bin_empty: missing 0.99 surfaces_clear: missing 0.99 | 73.87s |
| b09.jpg | bedroom | checkout state, everything wrong | bed_made: missing 0.99 towels_present: missing 0.95 bin_empty: missing 0.99 surfaces_clear: missing 0.99 | 56.28s |
| b10.jpg | bedroom | all ready, dim lamp light | bed_made: present 0.95 towels_present: present 0.95 bin_empty: unclear 0.30 surfaces_clear: present 0.90 | 39.63s |
| b11.jpg | bedroom | open suitcase, shoes, backpack on floor | bed_made: present 0.95 towels_present: present 0.95 bin_empty: present 0.85 surfaces_clear: missing 0.99 | 53.18s |
| b12.jpg | bedroom | hard case: duvet pulled up but lumpy, pillows strewn | bed_made: missing 0.99 towels_present: present 0.95 bin_empty: present 0.90 surfaces_clear: missing 0.90 | 6.09s |
| b13.jpg | bedroom | grey blanket is a towel confuser; no towels | bed_made: present 0.95 towels_present: missing 0.95 bin_empty: unclear 0.40 surfaces_clear: present 0.90 | 7.52s |
| b14.jpg | bedroom | hard case: bin has a few tissues only | bed_made: present 0.95 towels_present: present 0.95 bin_empty: missing 0.99 surfaces_clear: present 0.90 | 38.52s |
| b15.jpg | bedroom | all ready, doorway wide angle | bed_made: present 0.95 towels_present: present 0.95 bin_empty: present 0.90 surfaces_clear: present 0.95 | 5.32s |
| b16.jpg | bedroom | everything wrong, dark and grainy | bed_made: missing 0.99 towels_present: missing 0.95 bin_empty: missing 0.99 surfaces_clear: missing 0.99 | 7.23s |
| w01.jpg | bathroom | all ready | towels_present: present 0.95 toilet_paper: present 0.95 bin_empty: unclear 0.80 surfaces_clear: present 0.90 | 22.18s |
| w02.jpg | bathroom | rail empty | towels_present: missing 0.98 toilet_paper: present 0.99 bin_empty: unclear 0.40 surfaces_clear: present 0.95 | 6.53s |
| w03.jpg | bathroom | empty cardboard tube on holder | towels_present: present 0.95 toilet_paper: missing 0.98 bin_empty: present 0.90 surfaces_clear: present 0.90 | 5.19s |
| w04.jpg | bathroom | bin overflowing | towels_present: present 0.98 toilet_paper: present 0.98 bin_empty: missing 0.99 surfaces_clear: present 0.90 | 6.15s |
| w05.jpg | bathroom | toothbrush, razor, wipes on counter | towels_present: present 0.98 toilet_paper: present 0.99 bin_empty: present 0.92 surfaces_clear: missing 0.99 | 10.45s |
| w06.jpg | bathroom | no towels, empty tube | towels_present: missing 0.99 toilet_paper: missing 0.99 bin_empty: unclear 0.10 surfaces_clear: present 0.95 | 17.46s |
| w07.jpg | bathroom | bin overflowing, counter cluttered | towels_present: present 0.98 toilet_paper: present 0.99 bin_empty: missing 0.99 surfaces_clear: missing 0.99 | 222.04s |
| w08.jpg | bathroom | checkout state: towels on floor, empty tube, bin full, counter messy | towels_present: missing 0.99 toilet_paper: missing 0.99 bin_empty: missing 0.99 surfaces_clear: missing 0.99 | 4.97s |