Agentsia Labs · Research surface
Frontier leaderboards measure SWE-bench, MMLU-Pro, GPQA Diamond, HLE. None of them test adtech, fintech, legaltech, automotive, or clinical decisioning. Agentsia Labs publishes rigorous, open, reproducible evaluations of frontier APIs and open-weights specialists on the commercial workflows that matter to regulated enterprises.
Quarterly cadence. Automatic retest on frontier releases. Open-source harness. Published datasets. Invited expert review.
The programme
Every scenario, every rubric, every score aggregation, every raw model output is published. Datasets ship under Apache 2.0. The harness is on GitHub. Practitioners can reproduce any number we publish, end to end, on their own hardware.
Each release is tagged with a git SHA on the dataset and the harness. Model responses are recorded with version, date, temperature, and system prompt. Frontier updates trigger automatic retests. Stale numbers do not survive.
Rubrics are co-authored with invited domain operators (DSP traders, credit decision scientists, clinical informaticians). Methodology is readable in prose, not just code. If you disagree with a scoring choice, the argument to have is visible.
The cadence
A benchmark is only credible if it is current. Every active vertical is refreshed each quarter with the latest frontier versions and new open-weights releases. When a frontier model ships with a material capability delta, we retest within two weeks and publish. Announced cadence creates external accountability and gives practitioners a reason to return.
The roadmap
RTB reasoning, bid shading, pre-bid MFA filtering, supply-path optimisation, blocklist decisioning, latency-constrained reasoning.
Fraud typology, credit decisioning, transaction-pattern reasoning, compliance-aligned reasoning under regulatory constraints.
Statutory interpretation, contract analysis, regulatory reasoning under jurisdiction-specific constraints. In design.
Clinical triage, decision support, documentation quality under HIPAA-aligned constraints. In design.
ADAS perception edge cases, in-vehicle agent reasoning, driving-telemetry interpretation. In design.
Colophon
Every benchmark release is authored end to end by Modelsmith, the specialisation control plane that Agentsia licenses to regulated enterprises. The synthetic scenarios come out of the data-generation agent. The evaluations run through the same eval harness practitioners use on their own workflows. The post-trained specialists ship with real promotion records. The surface is the receipts.
If you want the platform behind the numbers, visit agentsia.uk/platform.