How well do models understand a port?
PortMind evaluates vision models on port camera images. We ask models to identify trucks and container attachments, then compare their answers with human labels.
Earlier studies
Grok 4.5 
Mistral Small 3.1 
Llama 4 Scout 
LLaVA 1.5 No score available: Moondream 3.1.
July 28, 2026 · 23 scored task rows · One human reference
Always answering “No” scores 65.2% here. Compare misses and false alarms below.
Study setup and limitations
20 unique images plus 4 repeat tasks. One undecidable task excluded, leaving 23 binary rows and 19 unique decidable images. Container labels include 8 positive and 15 negative rows. Repeats are not independent evidence.
Read about this study →Misses and false alarms
In guided inspection, Llama 3.2 Vision found all 8 positive tasks but also flagged 14 of 15 negative tasks.
Select a model to open its results. 8 positive and 15 negative task rows, including repeats.
Tasks
Truck presence
Can you see at least one truck? Pickups and larger road trucks count.
Container attachment
Is any truck carrying a shipping container or towing an empty chassis? A container standing nearby does not count.
Each question has three answers: Yes, No or Unsure. The benchmark tests what is visible in an image, not traffic flow or waiting time.
See labeling examples →How we evaluate models
Label the images
Reviewers independently answer the same questions, with model answers hidden.
Review the answers
Reviewers resolve disagreements before we finalize the reference labels.
Run the models
Each model receives the same images and questions. We record its answers and any failed requests.
Report the results
We compare model answers with the reference labels and report agreement, misses and false alarms.
Contact
For questions about the benchmark or evaluating a model, get in touch.