How we test
What Montréal v1.0 measures
Montréal v1.0 scores vision models on still port-camera images. Each model answers two questions: truck presence, and whether a shipping container or empty chassis is attached to a road truck. Answers are Yes, No, or Unsure. One qualifying truck anywhere in the image is enough for Yes. The benchmark does not measure vehicle counts, movement, or waiting time.
The two tasks
Truck presence asks whether a road truck is visible. Pickups count. Container attachment asks whether a shipping container is physically attached to a road truck, or whether a truck is towing an empty container chassis. A container standing nearby, parked on the ground, or sitting on rail does not count.
Reference labels
Human labels are the answer key. At least two reviewers submit their own judgments, with model answers hidden. Reviewers resolve disagreements and keep Unsure when the image cannot support a clear answer.
Frozen version
The reference answers, image manifest, and scoring protocol are fixed before models are scored. A correction creates a new version. It does not overwrite a published score.
Model runs
Each model receives the same images and questions under recorded conditions. The platform records the model identity, prompt, output limits, and any failed requests, then scores the answers against the frozen reference.
How scores are read
Found means the model answered Yes where the human reference says Yes. A false alarm means it answered Yes where the reference says No. Missed positives are Yes-reference answers that the model marked No. Uncertain model outputs are counted separately from misses and from failed calls. Uncertain reference labels are excluded from the Yes/No denominators and remain visible in the report.
Publication
Public reports show aggregate counts against the frozen version. Private images and raw model responses stay outside the public report. You do not label the dataset again for every model. New ports or changed images, labels, or rules have their own benchmark versions.
Earlier studies
These pilots informed Montréal v1.0. Each used its own setup and one human reference. Their scores stay separate from the versioned release.
Guided inspection
July 28, 2026 · 23 scored task rows · One human reference
20 unique images plus 4 repeat tasks. One undecidable task excluded, leaving 23 binary rows and 19 unique decidable images. Container labels include 8 positive and 15 negative rows. Repeats are not independent evidence.
Explore this study →Image-labeling pilot
Published pilot summary · 23 binary task rows · One human reference
20 unique images and 4 hidden repeats. One undecidable task excluded. These are the published pilot scores; original response traces have not been verified. The setup differs from the guided inspection run.
Explore this study →Disagreement review
72 selected disagreement images · One human relabel
GPT-5.5 and Grok 4.5 used different evidence workflows. Agreement excludes abstentions. These selected disagreement images are not a representative sample of port traffic.
Explore this study →Tool-use run
July 28, 2026 · 23 binary task rows · Mixed inspection workflows
20 unique images plus 4 repeats. GPT-5.5 used strict autonomous inspection; other models used guided steps. GPT-5.5 completed 14 of 24 tasks. Label agreement alone does not capture protocol errors, runtime failures or stopped turns.
Explore this study →