SABRE: Scalable and Automated Benchmarking of VLMs under Stress
SABRE turns a Markdown test design into generated VLM stress tests, then filters and repairs candidates. It offers a repeatable pattern for refreshing evals as models improve.
SABRE converts a Markdown task design into specifications, images, and question-answer pairs, then applies model filtering and human review. SABRE-Prior includes **600 images** and **1,000 questions** testing whether models follow visual evidence over learned expectations.
Teams evaluating vision agents can encode test intent and schema first, generate candidates, discard easy cases, then reserve human effort for validity checks, annotation fixes, and localized image repair.
SABRE converts a Markdown task design into specifications, images, and question-answer pairs, then applies model filtering and human review. SABRE-Prior includes **600 images** and **1,000 questions** testing whether models follow visual evidence over learned expectations. Teams evaluating vision agents can encode test intent and schema first, generate candidates, discard easy cases, then reserve human effort for validity checks, annotation fixes, and localized image repair. Across **six VLMs**, macro-average accuracy ranged from **17.8% to 31.3%**. A real-image control was comparably difficult for the filtering model, so low scores cannot be attributed only to counterfactual generated imagery.
SABRE supplies a scalable construction pipeline for adversarial visual-evidence tests, combining specification-first generation, model-based difficulty filtering, and targeted human repair. Relative to the candidates, it makes expectation-versus-evidence conflict directly testable and reports broad failure across six VLMs, while its real-image control narrows the explanation for low scores beyond synthetic-image artifacts.