Model selection

Public benchmarks are used as a screening layer for model selection, not as a substitute for application-specific evaluation. Models that perform well across capabilities relevant to analytics are shortlisted for deeper evaluation on representative domain tasks.

The workload is an analytics agent: it searches a semantic model, writes SQL, validates its own answer, and builds charts and dashboards through a multi-step tool loop. So the capabilities tracked here are the ones that loop leans on, each mapped to one or two public benchmarks, normalized onto a common scale and combined with weights that are stated in full below. Nothing is collapsed into a generic intelligence number, and a missing result stays missing rather than counting as zero.

This tracker is the public-screening layer of the Omni Evaluations project.

36 models · 152 benchmark observations · 11 currently pass every gate · last retrieval Oct 1, 2026

How the composite is computed

1. Normalize. Each observation is mapped onto 0–1 by its benchmark's rule. Pass-rate and accuracy benchmarks use the declared scale linearly (a 0–100 score divided by 100), clamped. Price uses a log-inverse rule so that a 10× cheaper model gains the same credit whether it moves from $20 to $2 or from $2 to $0.20.
2. Capability score. Within a capability, normalized observations are averaged with the within-capability weights in the mapping table. A benchmark with no observation for the model is dropped from that average, not scored as zero.
3. Composite. Σ(weight × capability score) over the capabilities that have a score, divided by the sum of those weights. The table shows that covered share next to each composite. If less than 75% of the weight is covered the composite is withheld. Two composites are only comparable when they cover the same capabilities. LiveSQLBench has no rows yet for models released after mid-2026, so the few models that carry an SQL score are pulled down by it relative to the rest. That gap is the first thing internal evaluation has to fill.
4. Weights. Data analysis 20%, Instruction following 20%, SQL / structured data 20%, Reasoning 15%, Long context 10%, Cost / efficiency 10%, General knowledge 5%. Tool / agent is tracked but carries no weight. These are screening weights for an analytics workload, not claims about general model quality. They live in src/data/models/capabilities.ts.

Everything on this page is generated from records under src/data/models/: model records, benchmark definitions with version and normalization rule, raw observations with source URL and retrieval date, list prices, capability weights and gate thresholds. Scores can be refreshed by rewriting the observation and pricing files from a benchmark's export without touching the page.

Public evidence versus internal evaluation

Two kinds of evidence are kept apart. Public benchmark evidence is everything on this page: reproducible, comparable across models, and only loosely related to any one production workload. Application-specific evaluation is the internal suite run on representative analytics tasks against the real semantic model and tool surface. The public composite says which models are worth the cost of that run. It does not predict production quality, and a high composite is not a recommendation.

Once enough shortlisted models have been through the internal suite, the pairing of public scores and internal outcomes becomes data of its own. The plan is to measure which public benchmarks correlate with internal performance on this workload and to move the capability weights toward the ones that do, or drop a benchmark that does not. The intended loop:

public benchmarks → candidate shortlist → internal evaluation → observed correlation → improved screening model

Until that correlation exists, the weights above are priors chosen from what the harness does most, not fitted values.