Montréal v1.0

This small study used 96 photos. It's retired, and the new study doesn't use it.

The question was yes or no for the whole photo, with pickups counted as trucks. Attachment meant a shipping container attached to a road truck, or an empty container chassis being towed. Parked containers and rail containers alone did not count.

Published: 2026-09-25

Frozen reference hash:

6697e8c878ae0974f94d1700a12a5ede61d5ad6a975b36be319e0fd6dfb2ec3f

Publication hash:

5dce53bec5a5af3403e07ac4e6fc7c4589d897b12e20c619f476ea689064097d

Truck presence

Both tasks had 91 definite reference answers and 5 unsure references. Found and false alarms below are photo counts.

Model	Found / positive	False alarms / negative	Correct / answered	Reference unsure	Total	Agreement	Coverage
GPT-6 Astra	54 / 54	21 / 37	70 / 91	5	96	76.9%	100.0%
Gemma 4 26B A4B	28 / 54	0 / 37	65 / 91	5	96	71.4%	100.0%
Llama 4 Scout 17B	39 / 54	6 / 37	70 / 91	5	96	76.9%	100.0%
Mistral Small 3.1 24B	48 / 54	7 / 37	78 / 91	5	96	85.7%	100.0%
Kimi K2.6	33 / 54	2 / 37	68 / 91	5	96	74.7%	100.0%
Kimi K2.7 Code	33 / 54	1 / 37	69 / 91	5	96	75.8%	100.0%

Container attachment

Both tasks had 91 definite reference answers and 5 unsure references. Found and false alarms below are photo counts.

Model	Found / positive	False alarms / negative	Correct / answered	Uncertain	Reference unsure	Total	Agreement	Coverage
GPT-6 Astra	35 / 37	8 / 54	79 / 88	3	5	96	89.8%	96.7%
Gemma 4 26B A4B	16 / 37	1 / 54	69 / 91	0	5	96	75.8%	100.0%
Llama 4 Scout 17B	6 / 37	0 / 54	60 / 91	0	5	96	65.9%	100.0%
Mistral Small 3.1 24B	7 / 37	0 / 54	61 / 89	2	5	96	68.5%	97.8%
Kimi K2.6	14 / 37	0 / 54	68 / 85	6	5	96	80.0%	93.4%
Kimi K2.7 Code	16 / 37	0 / 54	70 / 89	2	5	96	78.7%	97.8%