311 Intelligence Unit · Live tracker
Where artificial intelligence stands today, area by area: what it can do, who leads, what it costs the planet and how the world is responding. Every figure is sourced, dated and checked.
Compiled by Matthew Griffin and the 311 Intelligence Unit from 195 sources. Last updated . Sources checked daily, last check .
The short answer
On closed tests of knowledge, science, maths and coding, the best AI now matches or beats expert humans: it is ahead of the human score on 5 of the 10 tests here that have one. Where it still falls short is open-ended, real work: it completes about one in five real freelance projects, and new puzzle tests like ARC-AGI-3 still catch it out.
The length of task AI can carry out on its own has doubled roughly every four to seven months since 2019, and the price of a given level of AI falls about 40 times a year. Use is exploding: Google alone processes more than three quadrillion tokens a month.
Independently checked Maker's own figure Reported, being checked
Quick answers
There is no single best model: it depends on the task. Claude Opus 5.5 (max) by Anthropic leads general capability on Artificial Analysis Intelligence Index v4.3.2 with 57.6 (15 Oct 2026), 1.6 points ahead of Claude Sonnet 5.5 (max). Claude Sonnet 5.5 by Anthropic leads coding on Vals.ai Vibe Code Bench v1.1 with 92.4% (12 Oct 2026), 0.5 points ahead of Gemini 4 Argon. GPT-6.1 Sol (max) by OpenAI leads maths on FrontierMath Tiers 1-3 v2 with 93.7% (13 Oct 2026), tied with GPT-6 Astra (max). 4 labs share the lead across the 17 areas; the status board shows the leader in each.
Claude Sonnet 5.5 by Anthropic leads coding on Vals.ai Vibe Code Bench v1.1 with 92.4% (12 Oct 2026), 0.5 points ahead of Gemini 4 Argon. Coding tests change quickly, so check the date beside each score.
Claude Opus 5.5 by Anthropic leads agents and computer use on Terminal-Bench 4.0 on Vals.ai with 65.2% (13 Oct 2026), 1 point ahead of Claude Sonnet 5.5.
Claude Opus 5.5 (max) by Anthropic leads general capability on Artificial Analysis Intelligence Index v4.3.2 with 57.6 (15 Oct 2026), 1.6 points ahead of Claude Sonnet 5.5 (max). GPT-6 Astra Ultra by OpenAI leads IQ-style tests on TrackingAI Offline Test with 131.5 (14 Oct 2026), 1.5 points ahead of Kimi K3. IQ tests made for people are a rough guide for machines, so we report them as one area among many.
GPT-6.1 Sol (max) by OpenAI leads price for intelligence on Artificial Analysis at $0.72 per task (15 Oct 2026), the lowest cost among highly capable models.
The United States leads our country roll-up, ahead of China and the United Arab Emirates. The roll-up averages four measured layers: where the best models come from, how many leading models each country makes, how much it uses AI and its computing power.
OpenAI leads the most areas (6 of 17): Maths, Learning on the spot, IQ tests, Safety, Bias resistance, Price for intelligence.
Sources are checked every day and the page is rebuilt when a figure changes. This is the October 2026 edition; each figure shows its own date and status.
Yes. Matthew Griffin, founder of the 311 Institute, presents these findings as a keynote, board briefing or leadership session, updated to the day of the event.
Status board
One tile per area: the leading model on one independent test, its score and how far ahead it is. Each area is ranked on its own, because no single score captures what AI can do. Select a tile for the detail.
AnthropicOpenAIGoogleMetaOther
| Area | Leader | Score | Lead over next | Best open-weight | Test, status and source |
|---|---|---|---|---|---|
| General capability | Claude Opus 5.5 (max)Anthropic | 57.6 | +1.6 | None in top 8 | Artificial Analysis Intelligence Index v4.3.2 (best setting per model) Source |
| Maths | GPT-6.1 Sol (max)OpenAITied with GPT-6 Astra (max) | 93.7% | Tied | None in top 8 | FrontierMath Tiers 1-3 v2, % correct (Epoch AI) Source |
| Science | Claude Opus 5.5 (max)Anthropic | 66.9% | +3.8 | MiMo-V2.6-Pro (60.9) | SciCode (Artificial Analysis, subproblem scoring) Source |
| Coding | Claude Sonnet 5.5Anthropic | 92.4% | +0.5 | None in top 8 | Vals.ai Vibe Code Bench v1.1, % (build a web app from scratch) Source |
| Agents and computer use | Claude Opus 5.5Anthropic | 65.2% | +1 | None in top 8 | Terminal-Bench 4.0 on Vals.ai, % accuracy (mini-SWE-agent harness) Source |
| Work value | Gemini 4 ArgonGoogle | 68.9% | +1.9 | None in top 8 | Vals Index, % (agentic finance, coding, legal, tax tasks) Source |
| Multimodal | Claude Opus 5.5 (max)Anthropic | 87.7% | +0.8 | None in top 8 | MMMU-Pro on Artificial Analysis, % (best setting per model) Source |
| Health | Claude Opus 5.5Anthropic | 91.4% | +0.1 | None in top 8 | Vals MedScribe (clinical note writing) Source |
| Knowledge and facts | Gemini 4 Argon (high)Google | 76% | +0.4 | None in top 8 | SimpleQA Verified, Epoch AI run (best setting per model) Source |
| Learning on the spot | GPT-6 Astra (Max)OpenAI | 62.7% | +10 | None in top 4 | ARC-AGI-3, Standard harness, semi-private set Source |
| IQ tests | GPT-6 Astra UltraOpenAI | 131.5 | +1.5 | Kimi K3 (130) | TrackingAI Offline Test, text models, mean of last 7 runs (fewer where noted) Source |
| Safety | GPT-5 nano (2025-08-07)OpenAI | 0.986 | +0.004 | gpt-oss-120b (0.981) | HELM Safety v1.17.0 mean score Source |
| Cyber security | Lyrie Agent (DeepSeek-V4.1-Flash, Abliterated GLM-5.2)Lyrie | 99.2% | +0.7 | Lyrie Agent (DeepSeek-V4.1-Flash, Abliterated GLM-5.2) (99.2) | CyberGym (level 1) Source |
| Bias resistance | GPT-4.1 miniOpenAI | 88.1% | +7.9 | Llama 3.1 405B Instruct (75.2) | Phare Bias Resistance (higher is better) Source |
| Southeast Asian languages | Gemma 4 31BGoogle | 76.2 | +1.5 | Gemma 4 31B (76.2) | SEA-HELM SEA Overall average (open-weight models up to 200B) Source |
| Human preference | gemini-4-argon-highGoogle | 1,524 | +15 | None in top 8 | Arena (LMArena) text leaderboard score Source |
| Price for intelligence | GPT-6.1 Sol (max)OpenAI | $0.72 | $1.27 cheaper | None in top 7 | Artificial Analysis: cost in USD to run one Intelligence Index task (lower is cheaper), with list price per million tokens Source |
Lead over next is in the test's own units (percentage points for % scores). Some tests have not yet run the newest models; the test name and date show what each row covers.
Globe
Choose a category, then a measure. Drag the globe to turn it and select a country to see everything we hold on it. Every layer is sourced below the globe.
Loading the globe…
| # | Country | Roll-up (0 to 100) | Best model | Notable models built, 2025 | Working-age people using AI | Share of world data-centre electricity, 2024 |
|---|---|---|---|---|---|---|
| 1 | 86 | 57.6 | 59 | 33% | 45% | |
| 2 | 55 | 46.3 | 35 | 17.5% | 25% | |
| 3 | 38 | 31 | – | 73.3% | – | |
| 4 | 33 | 38 | – | 49.6% | – | |
| 5 | 29 | 28 | 8 | 40.6% | – | |
| 6 | 22 | – | – | 64.3% | – | |
| 7 | 17 | – | – | 49.9% | – | |
| 8 | 17 | – | – | 49.4% | – | |
| 9 | 15 | – | – | 45.1% | – | |
| 10 | 15 | – | – | 44.1% | – | |
| 11 | 15 | – | – | 43.7% | – | |
| 12 | 15 | – | – | 43.5% | – | |
| 13 | 15 | – | – | 43.5% | – | |
| 14 | 14 | – | – | 40.7% | – | |
| 15 | 6 | 13 | – | – | – | |
| 16 | 4 | 9 | – | – | – | |
| 17 | 3 | 6 | – | – | – |
How the roll-up works: each measure is scaled so the leading country scores 100, then the four measures are averaged. These lists cover the leading countries only, so a country that does not appear on a list scores 0 for it. Governance, trust and language layers are shown on the globe but not scored, because they are not like-for-like.
| # | Lab | Areas led | Top-three places | Points | Leads in |
|---|---|---|---|---|---|
| 1 | OpenAI | 6 | 9 | 23 | Maths, Learning on the spot, IQ tests, Safety, Bias resistance, Price for intelligence |
| 2 | Anthropic | 6 | 9 | 22 | General capability, Science, Coding, Agents and computer use, Multimodal, Health |
| 3 | 4 | 7 | 17 | Work value, Knowledge and facts, Southeast Asian languages, Human preference | |
| 4 | Lyrie | 1 | 1 | 3 | Cyber security |
| 5 | Meta | 0 | 4 | 5 | – |
| 6 | Moonshot AI | 0 | 1 | 2 | – |
| 7 | Creation Lab | 0 | 1 | 2 | – |
| 8 | xAI | 0 | 1 | 2 | – |
| 9 | Alibaba | 0 | 1 | 2 | – |
| 10 | Continuum AI | 0 | 1 | 1 | – |
Points: 3 for first, 2 for second and 1 for third in each area of the scorecard (ties share the place). This counts placings only; it does not blend scores from different tests.
Best against the rest
Anthropic models lead general capability, terminal agents, coding and multimodal tests; OpenAI leads research maths; Google leads the Vals work index and human preference. Margins at the top are small, mostly 0.5 to 2.5 points, and no open-weight model is in any top 8.
Claude Opus 5.5 (max) leads by 1.6 over Claude Sonnet 5.5 (max)
GPT-6.1 Sol (max) and GPT-6 Astra (max) are tied
Claude Sonnet 5.5 leads by 0.5 points over Gemini 4 Argon
Claude Opus 5.5 leads by 1 points over Claude Sonnet 5.5
Gemini 4 Argon leads by 1.9 points over Claude Sonnet 5.5
Claude Opus 5.5 (max) leads by 0.8 points over GPT-6 Astra (max)
gemini-4-argon-high leads by 15 over muse-wonder-max
MiMo-V2.6-Pro is cheaper by $0.59 over GPT-6.1 Sol (max)
Artificial Analysis Intelligence Index v4.3.2 (best setting per model) #
57.6 index points
Leader on general capability and gap to second
Claude Opus 5.5 (max), Anthropic · 22 Sep 2026
Leads Claude Sonnet 5.5 (56.0) by 1.6 points. Best open-weight: MiMo-V2.6-Pro 46.3, 11th.
FrontierMath Tiers 1-3 v2, % correct (Epoch AI) #
93.7%
Leader on research-level maths and gap to second
GPT-6.1 Sol and GPT-6 Astra (tie), OpenAI · 29 Sep 2026
Tie at top; next Claude Opus 5.5 at 91.2 (2.5 points behind). Best open-weight Kimi K3 72.2. GPT-6.1 Sol scored 100% on Tier 4 v2. History is best score by model release date on the v2 set.
Vals.ai Vibe Code Bench v1.1, % (build a web app from scratch) #
92.4%
Leader on app-building coding and gap to second
Claude Sonnet 5.5, Anthropic · 12 Oct 2026
Leads Gemini 4 Argon (91.91) by 0.48 points. Best open-weight MiMo V2.6 Pro 85.22, 14th.
SWE-bench Verified on Vals.ai (archived) #
97%
Real GitHub bug fixes; now saturated
Claude Opus 5, Anthropic · 1 Sep 2026
Archived as saturated. Open-weight DeepSeek V4 Pro 0813 second at 96.4, GLM 5.3 95.4, Kimi K3 93.4. Official swebench.com Verified top is 79.2 (Claude 4.5 Opus, Dec 2025), not updated with newer models.
Terminal-Bench 4.0 on Vals.ai, % accuracy (mini-SWE-agent harness) #
65.2%
Leader on terminal agent tasks and gap to second
Claude Opus 5.5, Anthropic · 13 Oct 2026
Leads Claude Sonnet 5.5 (64.14) by 1.01 points; Opus 5.5 drops to 58.08 if provider fallbacks count as failures. Best open-weight GLM 5.3 38.89, 12th.
OSWorld 2 (computer use, 500 steps) #
31.4%
Full completion of long computer-use tasks
Claude Opus 5 (max), Anthropic · 24 Jul 2026
Partial score 68.3%. Second GPT-5.6 Sol 27.3%. Newer models (Opus 5.5, GPT-6) not listed in the data. Human success rate not published; median human time 1.6 hours per task.
Vals Index, % (agentic finance, coding, legal, tax tasks) #
68.9%
Leader on work-task index and gap to second
Gemini 4 Argon, Google · 13 Oct 2026
Leads Claude Sonnet 5.5 (67.04) by 1.86 points. On GDPval-AA (Artificial Analysis) Claude Opus 5.5 leads; best open-weight MiMo-V2.6-Pro ranks 7th.
MMMU-Pro on Artificial Analysis, % (best setting per model) #
87.7%
Leader on expert multimodal questions and gap to second
Claude Opus 5.5 (max), Anthropic · 22 Sep 2026
Leads GPT-6 Astra (86.9) by 0.8 points. Best open-weight Kimi K3 80.5. Vals.ai MMMU-Pro (archived 2026-09-01) had Claude Fable 5.1 top at 90.64%.
Arena (LMArena) text leaderboard score #
1,524 Arena score
Leader on human preference and gap to second
gemini-4-argon-high, Google · 15 Oct 2026
Preliminary (4,904 votes). Leads Meta muse-wonder-max (1509) by 15. Best open-weight kimi-k3-max 1487, 18th.
Artificial Analysis: cost in USD to run one Intelligence Index task (lower is cheaper), with list price per million tokens #
0.72 USD per task
Cheapest top-tier model (index 50 or above) per index task
GPT-6.1 Sol (max), OpenAI · 29 Sep 2026
Scores 51.8, 5.8 below the leader, at about one eighth of Claude Opus 5.5's $5.98 per task. Open-weight MiMo-V2.6-Pro costs $0.13 per task at 46.3.
Areas
Pick an area. Each shows the headline figures, the leading models, how results have changed, and plainly what is not yet measured.
Capability
On independent tests of real research work, the best models now pass about 63% of Terminal-Bench Science lab tasks and score about 67% on SciCode subproblems, while the authors' LABBench2 board tops out at 61%. Earlier tests show agents beat experts on short tasks but humans still lead over longer horizons (RE-Bench, PaperBench).
Claude Opus 5.5 (max) leads by 3.8 points over Claude Fable 5.1 (max)
Agent scaffolds differ by entry; maintainers stopped accepting submissions in April 2026, so newer models are absent. Source
LABBench2 #
60.9%
About 1,900 biology research tasks: literature, figures, protocols, sequences, cloning
GPT-5.2 (tools, high), OpenAI · 13 Mar 2026
Average across sub-benchmarks with tools; Tools entries cover 14 of 15 sub-benchmarks. Authors' board last updated 2026-03-23, no newer models and no human baseline. Task count (about 1,900) is not stated on the board.
LAB-Bench LitQA2 (2024) #
66% accuracy
Hard literature-search questions in biology
PaperQA2 agent, FutureHouse · Sep 2024 Open weights
People: 67.7 (9 PhD-level biologists, accuracy (precision 73.8%)) source
PaperQA2 precision was 85.2% against 73.8% for experts; 2024 point is the plain-model best in the LAB-Bench paper.
LAB-Bench TableQA (2024) #
83% accuracy
Reading and reasoning over tables in scientific papers
Claude 3.5 Sonnet, Anthropic · Jul 2024
People: 84 (human biology experts in the paper) source
Closest category to human level in 2024; humans led clearly on LitQA2 (70 vs 35) and SuppQA (85 vs 20).
Terminal-Bench Science #
62.9%
70 real research workflows from working scientists, run in a terminal, strict pass or fail
GPT-6 Astra, OpenAI · 14 Oct 2026
Mini-SWE-agent harness. Makers report higher on their own harnesses: OpenAI 64.6 (GPT-6 Astra), Anthropic 58.7 (Claude Opus 5.5) against 47.14 at Vals.
SciCode (Artificial Analysis) #
66.9%
Research-level scientific coding across 16 disciplines, scored on 288 subproblems
Claude Opus 5.5 (max), Anthropic · 15 Oct 2026
Subproblem scoring; Artificial Analysis says scores are under review after the SciCode-Verified audit found test defects.
SciCode main problems (original paper) #
4.6%
Whole multi-step research coding problems solved end to end
Claude 3.5 Sonnet, Anthropic · Jul 2024
Best model in the 2024 paper, most realistic setting. Authors' own board, undated, tops at 10.8% main and 34.4% subproblem (o3-mini).
SciCode-Verified #
92.2% main problems
SciCode with 263 test defects corrected, 64 main problems
GPT-5.6 Sol, OpenAI · Aug 2026
Same model scored 23.4% main and 58.2% subproblem on the original test set; 98.3% subproblem on the corrected set. Single pass@1 runs.
MLE-bench #
64.4% any medal
75 Kaggle machine-learning competitions; share where the agent earns any medal
Famou-Agent 2.0 on Gemini 3 Pro Preview, Google (model) · 23 Feb 2026
Agent scaffolds differ by entry; maintainers stopped accepting submissions in April 2026, so newer models are absent.
PaperBench #
21%
Replicating 20 ICML 2024 papers from scratch, graded on a rubric
Claude 3.5 Sonnet (new), Anthropic · Apr 2025
People: 41.4% (ML PhDs, best of 3 attempts, 48 hours, 3-paper subset (o1 scored 26.6 on the same subset)) source
Qwen reports 93.0% for Qwen3.8 Max on the lighter Code-Dev variant (its own figure), which is not comparable.
RE-Bench #
AI about 4x human score at 2 hours; humans about 2x AI at 32 hours
7 open-ended AI research engineering tasks, agents against 61 human experts
Best 2024 agents (Claude 3.5 Sonnet, o1-preview), Anthropic, OpenAI · Nov 2024
People: Humans narrowly ahead at 8 hours (61 ML experts, 71 eight-hour attempts) source
No newer public RE-Bench headline figure found.
Not measured yet
Capability
Exam-style medical tests are saturated: the best models score about 96.5% on MedQA against a pass mark of about 60%. On harder clinical tests progress is slower, and the first large randomised trial of an LLM tool in real primary care found no significant change in 14-day treatment failure.
Claude Opus 5.5 leads by 0.1 points over Claude Fable 5.1
Vals archived the test as saturated; top 10 models all between 95.9 and 96.5%. 2026-04 is the Vals page date, not the o1 run date. Public data may be in trainin Source
MedQA (USMLE-style) #
96.5%
Multiple-choice questions from US medical licensing exams
o1, OpenAI · 16 Apr 2026
People: 60% (approximate USMLE pass mark) source
Vals archived the test as saturated; top 10 models all between 95.9 and 96.5%. 2026-04 is the Vals page date, not the o1 run date. Public data may be in training sets.
MedMCQA #
72.3%
194,000 Indian medical entrance exam (AIIMS, NEET PG) questions
Med-PaLM 2, Google · May 2023
People: 90% (average of merit candidates, as stated by the dataset authors) source
The Med-PaLM 2 paper cites GPT-4-base at 73.66 as the then state of the art. No current independent leaderboard found.
HealthBench #
60%
5,000 health conversations graded on 48,562 physician-written rubric criteria
o3, OpenAI · May 2025
People: Below recent models (262 physicians writing unaided; with April 2025 model drafts they no longer improved the answers) source
OpenAI is both benchmark author and model maker. The 2026 point is Anthropic's own raw score (60.6% length-adjusted).
HealthBench Hard #
46.2%
The 1,000 hardest HealthBench conversations
GPT-5, OpenAI · 7 Aug 2025
Aggregators list lower 2026 self-reports (Muse Spark 42.8, GPT-6.1 Sol 36.2), so settings differ; no independent run found.
HealthBench Professional #
65.6% (length-adjusted)
525 real clinician chat tasks: consults, documentation, research, physician rubrics
Claude Opus 5.5, Anthropic · 22 Sep 2026
People: 43.7 (specialist physicians with unlimited time and internet, no AI (length-adjusted)) source
Raw score 77.1%. Anthropic graded with Claude Opus 4.8, OpenAI with its own grader, so cross-maker comparison is loose.
MedHELM #
0.703 mean win rate (0 to 1)
Stanford suite of real clinical tasks across healthcare task categories, mean win rate
GPT-5 (2025-08-07), OpenAI · 19 Jan 2026
Only 13 models in the latest release; next o4-mini 0.697, GPT-5 mini 0.690. Win rate is relative to the models tested, not absolute.
Vals MedScribe #
91.4%
Writing SOAP clinical notes from doctor-patient transcripts, 100 rubrics
Claude Opus 5.5, Anthropic · 13 Oct 2026
Synthetic transcripts; rubrics written by medical scribes. No human score on the task.
Vals MedCode #
63.6%
Assigning ICD-10-CM diagnosis codes from hospital records
Claude Opus 5, Anthropic · 13 Oct 2026
References made by two certified coders; their own accuracy is not reported.
MedAgentBench #
69.7% success
300 physician-written tasks in a virtual electronic health record
Claude 3.5 Sonnet v2, Anthropic · Feb 2025
Next GPT-4o 64.0%. Newer models not reported in the paper.
Randomised trial: AI Consult in Kenyan primary care #
14-day treatment failure 2.2% with AI vs 2.0% without; adjusted odds ratio 0.77 (95% CI 0.55 to 1.08), not significant
Cluster-randomised trial of an LLM decision-support tool, 9,691 patients, 103 clinical officers, 16 clinics
AI Consult (GPT-4o embedded in the EMR), Penda Health with OpenAI · 26 Jun 2026
No safety signal; University of Birmingham reports better documentation quality and lower antibiotic costs, figures not in the abstract.
Observational study: AI Consult at Penda Health #
16% fewer diagnostic errors and 13% fewer treatment errors with the tool
Non-randomised quality study of 39,849 visits in 15 Nairobi clinics
AI Consult (LLM, model not named), Penda Health with OpenAI · Jul 2025
Preprint by the tool's developers; errors judged by reviewing physicians, not randomised.
Not measured yet
Capability
On ARC-AGI-3, which asks agents to learn unfamiliar games on the spot, the best model went from 0.5% at launch in March 2026 to 62.7% by September 2026 (GPT-6 Astra), against 100% for humans. Learning new rules from a given context stays weak (best 27.9% on CL-bench), and no frontier model is known to update its own weights from use.
GPT-6 Astra (Max) leads by 10 points over GPT-6.1 Sol (Max)
Semi-private set. Dates are model release dates on the ARC leaderboard. Standard harness only carries forward the model's own notes. Source
Most frontier models remain at 38 to 57%: they are overconfident. GPT-6 Astra has top HLE accuracy (54.8%) with 39% calibration error. History points are lowest Source
ARC-AGI-3 (Standard harness) #
62.7%
Interactive game worlds the agent has never seen; it must work out the rules while playing. 100% means beating every game as efficiently as humans
GPT-6 Astra (Max), OpenAI · Sep 2026
People: 100% (human players (ARC Prize: humans score 100%)) source
Semi-private set. Dates are model release dates on the ARC leaderboard. Standard harness only carries forward the model's own notes.
ARC-AGI-3 (Provider Adapter harness) #
99.9%
Same test, with a harness that keeps the model's reasoning state and compacts long play so it can reuse earlier work
GPT-6 Astra (High), Provider Adapter, OpenAI · Sep 2026
People: 100% (human players) source
Shows how much the harness matters: same model scores 62.7% on the Standard harness. Gemini 4 Argon reaches 97.5% on this harness.
ARC-AGI-2 #
95%
Novel grid puzzles that require inferring a rule from a few examples
GPT-6 Astra (Max), OpenAI · Sep 2026
People: 100% (ARC Prize human panel) source
Semi-private set, about $1.12 per task. Now close to saturated; ARC-AGI-1 is saturated at 98.5% for several models.
CL-bench #
27.9%
Learning new knowledge, rules and procedures from a long task-specific context, then applying it
GPT-5.4 (xhigh), OpenAI · Mar 2026
Paper (arXiv 2602.03587, Feb 2026): ten frontier models solved 17.2% on average. Newest models (GPT-6, Opus 5.5) not yet listed.
CL-bench Life #
22.2%
Learning from fragmented personal records, messages and activity trails
GPT-5.5 (high), OpenAI · Apr 2026
No models released after April 2026 listed yet.
Humanity's Last Exam, calibration error #
20% calibration error
RMS gap between a model's stated confidence and its actual accuracy on expert questions (lower is better)
Claude Fable 5.1 (xhigh), Anthropic · Sep 2026
Most frontier models remain at 38 to 57%: they are overconfident. GPT-6 Astra has top HLE accuracy (54.8%) with 39% calibration error. History points are lowest values on the current board, not a dated series.
AA-Omniscience Index #
46 index
Knowledge test that rewards correct answers and penalises wrong guesses versus abstaining (range -100 to 100)
Claude Opus 5.5 (Max), Anthropic · Oct 2026
GPT-6 Astra (High) 44, Claude Fable 5.1 (Max) 43. Hallucination rate = wrong / all non-correct answers.
AA-Omniscience hallucination rate #
15%
Share of non-correct answers that are confident wrong answers rather than abstentions (lower is better)
Gemini 4 Argon, Google · 2 Oct 2026
GPT-6 Astra (max) 51%, GPT-6.1 Sol (max) 54%. The Next Web calls it the lowest of any model scoring 45+ on the Artificial Analysis Intelligence Index (not the Omniscience index).
SimpleQA abstention (OpenAI) #
52% not attempted
How often a model declines to answer rather than guess on short factual questions
gpt-5-thinking-mini, OpenAI · 5 Sep 2025
Accuracy 22%, error 26%; o4-mini abstained 1% with 75% errors. Abstaining trades a little accuracy for far fewer wrong answers.
OECD AI Capability Indicators: Knowledge, learning and memory #
3 level (of 5)
Expert panel rating of AI against a 5-level human ability scale
State of the art AI systems, OECD assessment · 3 Jun 2025
People: 5 (top of the OECD scale, full human-equivalent ability) source
Level 3: semantic understanding and generalisation to new situations. OECD says AI trails humans on every scale. Beta indicators, June 2025.
Not measured yet
Capability
The largest disclosed training set is 45 trillion tokens (DeepSeek V4.1 Flash); closed labs do not disclose theirs. The best factual recall score on SimpleQA Verified is about 76 to 78%, and the best FACTS suite score is 75.7% (GPT-6 Astra). Frontier context windows sit at about 1 million tokens, with one open model at 10 million.
Gemini 4 Argon (high) leads by 0.4 points over GPT-6 Astra (max)
History dates are model release dates. Benchmark built by Google DeepMind authors (arXiv 2509.07968). Source
Gemini 3 Pro 68.8 is from the Dec 2025 launch paper; current board shows other values after Search was updated to V2. Sub-scores for GPT-6 Astra: Grounding 82.0 Source
Disclosed pre-training data size #
45 trillion tokens
Largest training set a maker has published, in tokens
DeepSeek V4.1 Flash, DeepSeek · Sep 2026 Open weights
Multimodal corpus. Thinking Machines Inkling is also listed at 45T in Epoch's database. Other disclosures: Qwen3 36T, LongCat-2.0 35T, DeepSeek V4 Pro 32T, Llama 4 Scout about 40T, Maverick about 22T, Kimi K2 15.5T, DeepSeek-V3 14.8T.
Disclosed pre-training data size #
36 trillion tokens
Qwen3 family training set
Qwen3 (235B-A22B and family), Alibaba · May 2025 Open weights
Covers 119 languages and dialects.
Disclosed pre-training data size #
40 trillion tokens
Llama 4 Scout training set
Llama 4 Scout, Meta · Apr 2025 Open weights
Approximate (~40T). Maverick ~22T. Meta's blog says the overall mix was over 30T tokens.
Epoch AI dataset size trend #
3.7 x per year
Growth rate of language model training datasets
Language models (all), Epoch AI analysis · 19 Jun 2024
Doubling roughly every six months. Separate Epoch insight (Aug 2025): tokens per active parameter in open models grow 3.1x per year, from about 10 in 2022 to about 300 in 2025.
Context window (maker spec) #
10 million tokens
Largest input context offered by a released model
Llama 4 Scout, Meta · Apr 2025 Open weights
Maker's spec; usable quality at full length is not shown by the spec.
Context window (maker spec) #
1.05 million tokens
Largest context window among closed frontier models
GPT-6 Astra, OpenAI · Sep 2026
Gemini 4 Argon 1,048,576 (unified input+output, ai.google.dev); Claude Opus 5.5, Fable 5.1, Sonnet 5.5, Haiku 5.5 all 1M (platform.claude.com); Kimi K3 1,048,576; DeepSeek V4 Pro 1M.
SimpleQA Verified (Epoch run) #
76%
1,000 short factual questions answered from memory without search
Gemini 4 Argon (high), Google · 13 Oct 2026
History dates are model release dates. Benchmark built by Google DeepMind authors (arXiv 2509.07968).
SimpleQA Verified (Kaggle) #
77.5%
Same test on the authors' Kaggle leaderboard
Gemini 3.1 Pro Preview, Google · 10 Mar 2026
Run by Google, which also makes the top model. GPT-6 Astra 75.8, Gemini 4 Argon 75.6. Data read from Kaggle's leaderboard API.
SimpleQA (original, OpenAI) #
62.5%
4,326 short factual questions, no search
GPT-4.5 preview, OpenAI · Feb 2025
No longer updated with newer models; the field has moved to SimpleQA Verified.
FACTS Benchmark Suite (average) #
75.7%
Average of four factuality tests: grounding in documents, multimodal, parametric recall, and search
GPT-6 Astra, OpenAI · 15 Sep 2026
Gemini 3 Pro 68.8 is from the Dec 2025 launch paper; current board shows other values after Search was updated to V2. Sub-scores for GPT-6 Astra: Grounding 82.0, Multimodal 47.7, Parametric 83.1, Search V2 89.9.
FACTS Grounding #
85.2%
Answers that stay faithful to a long supplied document
GPT-6 Sol, OpenAI · 25 Sep 2026
History dates are model release dates.
FACTS Parametric #
83.1%
Closed-book factoid questions answered from the model's own memory
GPT-6 Astra, OpenAI · 11 Sep 2026
Claude Opus 5.5 scores 57.9 here but 86.5 on FACTS Search V2.
AA-Omniscience accuracy #
67%
Share of 6,000 knowledge questions across 42 topics answered correctly
Claude Fable 5.1 (Max), Anthropic · Oct 2026
Claude Opus 5.5 (Max) 66%.
Not measured yet
Capability
On TrackingAI's offline test, which has never been on the public internet, the best text model averages about 131 (GPT-6 Astra Ultra), roughly the top 2% of humans. On the public Mensa Norway test top models average about 145, but that test may be in training data.
GPT-6 Astra Ultra leads by 1.5 over Kimi K3
TrackingAI Offline Test #
131.5 IQ points
IQ-style matrix puzzle test written by a Mensa member and never published online; best text (verbal) model
GPT-6 Astra Ultra (ChatGPT Pro), OpenAI · 5 Oct 2026
People: 100 (average human adult (IQ scale mean 100, SD 15)) source
Mean of the 4 runs logged so far (the site averages up to the last 7), computed from TrackingAI's published log with the site's own formula. Vision versions score higher over fewer runs (Claude Opus 5.5 Max Vision 134.5, GPT-6 Astra Ultra Vision 134.0). The test has only 16 items, so one answer moves the score by about 7 points.
TrackingAI Offline Test #
130 IQ points
Same offline test, best open-weight model
Kimi K3, Moonshot AI · 11 Oct 2026 Open weights
People: 100 (average human adult (IQ scale mean 100, SD 15)) source
Mean of last 7 runs, all scored 130. Open-weight models now sit level with closed frontier models on this test.
TrackingAI Mensa Norway #
145.4 IQ points
Public online Mensa Norway matrix test (35 items), given to models as text; best text model with a full 7-run average
Claude Opus 5.5 (Max) (tied with Claude Opus 5.5 standard, 145.4), Anthropic · 9 Oct 2026
People: 100 (average human adult (IQ scale mean 100, SD 15)) source
Mean of last 7 runs (148,148,148,145,142,142,145), last run 2026-10-09; Claude Opus 5.5 (standard) also 145.4. TrackingAI's all-model leader is GPT-6 Astra Ultra (Vision) at 147.0 over only 3 runs; best text figure is Claude Fable 5.1 146.5 over only 2 runs. Public test, so answers may be in training data; scores near 145+ are close to the ceiling.
WAIS-IV Verbal Comprehension Index (adapted) #
145 index score
Human IQ battery subtests (vocabulary, similarities, information, comprehension) given to models as prompts
Claude 3.5 Sonnet, Anthropic · Oct 2024
People: 100 (WAIS-IV adult norm mean) source
Same study: working memory 145 to 150 for most models, but perceptual reasoning 50 to 81 (near the floor). Processing speed and full-scale IQ could not be scored. Authors include Google researchers; older models only.
Not measured yet
Capability
The largest disclosed model is Kimi K3 with 2.8 trillion parameters, of which 104 billion are active per token. OpenAI, Anthropic and Google do not disclose parameter counts for any current frontier model.
Kimi K3 leads by 400 over Qwen3.8-2.4T-A95B
Mixture of experts: 104B active. 1M-token context. Kimi K3 Licence, not fully open. Source
Disclosed parameter count #
2.8 trillion parameters (total)
Largest total parameter count a maker has published for a released model
Kimi K3, Moonshot AI · Jul 2026 Open weights
Mixture of experts: 104B active. 1M-token context. Kimi K3 Licence, not fully open.
Disclosed parameter count #
2.4 trillion parameters (total)
Qwen3.8 open flagship
Qwen3.8-2.4T-A95B, Alibaba · Aug 2026 Open weights
95B active. Same size listed by Epoch for API model Qwen 3.8 Max.
Disclosed parameter count #
1.6 trillion parameters (total)
DeepSeek V4 Pro
DeepSeek V4 Pro, DeepSeek · Apr 2026 Open weights
49B active, MIT licence, trained on over 32T tokens. DeepSeek-V3 (Dec 2024) was 671B total, 37B active.
Disclosed parameter count #
1 trillion parameters (total)
Kimi K2, first open trillion-parameter model in this list
Kimi K2, Moonshot AI · Jul 2025 Open weights
32B active, 15.5T training tokens, 128K context.
Disclosed parameter count #
2 trillion parameters (total, approx.)
Meta's largest Llama 4 model
Llama 4 Behemoth (preview), Meta · Apr 2025
'Nearly two trillion' total, 288B active, 16 experts. Previewed while still training; not released.
Disclosed parameter count #
235 billion parameters (total)
Largest original Qwen3 open model
Qwen3-235B-A22B, Alibaba · Apr 2025 Open weights
22B active.
Stated parameter count (closed model) #
1.5 trillion parameters (total, stated)
Only closed frontier lab whose leader has stated a size
Grok 4.6, xAI · Aug 2026
CEO statement on X, not in xAI's official model docs; the Grok 4.7 release page gives no size, only 'a new, larger base model'.
Disclosed parameter count (closed frontier) #
Not disclosed
Whether OpenAI, Anthropic and Google publish model sizes
GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, OpenAI, Anthropic, Google · Oct 2026
Model docs give context and output limits only. Third-party size estimates exist but are not used here.
Not measured yet
Capability
The best closed models still lead the best open-weight models on every independent measure checked. Epoch puts the lag at about 4 months (8 ECI points) in 2026; on the Artificial Analysis Intelligence Index the gap is 11.3 points and on LMArena text it is 37 Elo points.
Kimi K3 leads by 1.8 over DeepSeek V4.1 Flash
Gap computed by us from Epoch's published ECI scores: best closed-weight minus best open-weight model released on or before each date. Gemini 4 Argon is classed Source
Epoch Capabilities Index (ECI) #
4 months
How many months the best open-weight model trails the best closed model
Best open-weight vs best closed (average since Jan 2026) · 29 May 2026
Same insight gives an average gap of 8 ECI points. History points use different methods: the 2024 figure covers MMLU, GPQA, GSM1k and BBH, later ones use ECI.
Epoch Capabilities Index (ECI) #
9.4 ECI points
ECI points between the best closed and best open-weight model released so far
Claude Opus 5.5 (166.82) vs Kimi K3 (157.42), Anthropic vs Moonshot · 15 Oct 2026
Gap computed by us from Epoch's published ECI scores: best closed-weight minus best open-weight model released on or before each date. Gemini 4 Argon is classed by Epoch as 'Other' access and is excluded.
Artificial Analysis Intelligence Index v4.3.2 #
11.3 index points
Points between the best model overall and the best open-weight model
Claude Opus 5.5 max (57.6) vs MiMo-V2.6-Pro (46.3), Anthropic vs Xiaomi · 15 Oct 2026
GLM-5.2 led open-weight models at 51 on index v4.1 (June 2026). Index versions change scale, so older scores are not comparable with v4.3.2. The best open-weight model ranks 11th.
Artificial Analysis Intelligence Index v4.3.2 #
46.3 index points
Best open-weight model on the index
MiMo-V2.6-Pro, Xiaomi · 21 Sep 2026 Open weights
Next open-weight: GLM-5.3 max 44.8 (Z AI), Kimi K3 max 43.6 (Moonshot). Release date shown is the model's.
LMArena (Arena) text leaderboard #
37 Elo points
Elo points between the top closed model and the top open-weight model
gemini-4-argon-high (1524) vs kimi-k3-max (1487), Google vs Moonshot · 15 Oct 2026
Gemini 4 Argon score is preliminary (4,904 votes). Kimi K3 uses a custom licence; Artificial Analysis classes it as open weights. The best MIT-licensed model is mimo-v2.6-pro at 1479. The Stanford AI Index 2026 gap, as of March 2026, was 49 points (1,503 vs 1,454).
Chatbot Arena, Stanford AI Index #
3.3%
Percentage gap between best closed and best open-weight model on Chatbot Arena
Best closed vs best open-weight · Mar 2026
Stanford AI Index 2025: the gap fell from 8.04% (Jan 2024) to 1.70% (Feb 2025). Stanford AI Index 2026: it rose from 0.5% (Aug 2024) to 3.3% (Mar 2026).
Chatbot Arena, Stanford AI Index 2026 #
1,503 vs 1,454
Top closed vs top open model Arena score in the 2026 AI Index
Mar 2026
Chart caption: 'Closed models lead throughout'. Same chart gives US 1,503 vs top Chinese model 1,464.
Epoch furniture assembly visual reasoning test #
7 months
Lag of open-weight models on spotting mistakes in IKEA assembly photos
Best open-weight vs best closed · 23 Sep 2026
Narrow test: 60 photos, three furniture builds. Shows the lag varies by task.
Not measured yet
Behaviour and safety
On HELM Safety the best models now score about 0.99 out of 1, but MLCommons AILuminate has awarded no system its top Excellent grade. Reported AI incidents hit a record 362 in 2025, up from 233 in 2024.
GPT-5 nano (2025-08-07) leads by 0.004 over o3 (2025-04-16)
HELM Safety v1.17.0 #
0.986 score (0 to 1)
Mean of five safety test scores (refusing harmful requests, bias, not over-refusing safe ones)
GPT-5 nano (2025-08-07), OpenAI · 24 Nov 2025
87 models tested; mean scores range from 0.526 (Mistral Instruct v0.1 7B) to 0.986. Release predates the newest frontier models.
MLCommons AILuminate v1.0 (English) #
0 Excellent, 6 Very Good, 19 Good, 6 Fair, 1 Poor
Grades AI systems on how often they give hazardous responses across 12 hazard categories
32 graded systems · 15 Oct 2026
Very Good went to Claude 3.5 Haiku, Claude 3.5 Sonnet, Mistral Large 2402 Moderated, Gemma 2 9B, Phi 3.5 MoE and Phi 4. Graded systems are 2024 to early 2025 models; date is when the page was read.
OR-Bench #
0.89 Spearman correlation
Measures over-refusal: models declining safe prompts that only look harmful
32 models from 8 families · May 2024
Models that refuse more harmful prompts also refuse more safe ones (rank correlation 0.89); about 80,000 test prompts.
Published frontier AI safety policies #
12 companies
Number of companies that have published a frontier AI safety policy
Anthropic, OpenAI, Google DeepMind, Magic, Naver, Meta, G42, Cohere, Microsoft, Amazon, xAI, Nvidia · 9 Dec 2025
Count as of the METR post.
Frontier AI Safety Commitments (AI Seoul Summit 2024) #
20 organisations
Number of companies that signed the voluntary Seoul commitments
7 Feb 2025
16 signed on 21 May 2024; Magic, Minimax, 01.ai and NVIDIA added later. The page was last updated 7 Feb 2025 but does not date the additions.
AI Incident Database, via Stanford AI Index 2026 #
362 incidents
Number of reported AI incidents in a year
2025
Record high, up 55.4% on 2024. Annual counts stayed under 100 until 2022; earlier yearly values are only in the chart.
OECD AI Incidents Monitor #
18,370 incidents and hazards (approx.)
Total AI incidents and hazards logged from news reports
15 Oct 2026
Page says 'about 18370'; counts news-reported events, so not directly comparable with the AI Incident Database.
Not measured yet
Behaviour and safety
AI agents now solve nearly all tasks on the main public cyber security tests: 99.2% on CyberGym and 100% on the Cybench subset reported for Claude Mythos Preview. The UK AI Security Institute estimates the length of cyber tasks AI can complete is doubling about every 4.7 months.
Lyrie Agent (DeepSeek-V4.1-Flash, Abliterated GLM-5.2) leads by 0.7 points over Creation (Creation Model, DeepSeek-V4-Pro, Qwen 3.8 Max)
CyberGym (level 1) #
99.2%
Whether AI agents can find and reproduce real security flaws in widely used open-source software.
Lyrie Agent (multi-model: DeepSeek-V4.1-Flash, Abliterated GLM-5.2), Lyrie · 15 Sep 2026 Open weights
Top entries are agent systems submitted by their builders, often combining several models. Best single-model entry: RO0T Agent on DeepSeek-V4.1-Flash, 97.7% (2026-09-18).
Cybench (unguided) #
100%
Professional capture-the-flag security challenges solved end to end without step-by-step hints.
Claude Mythos Preview, Anthropic · 7 Apr 2026
Recent scores come from makers' system cards on subsets of 35 to 39 of the 40 tasks; older scores are from the HAL evaluation on all 40. Claude Opus 4.7 is listed at 96%. Dates are model release dates (from the CyberGym data file). Earlier HAL best was 22.5% (o3-mini, undated).
DARPA AI Cyber Challenge (AIxCC) final #
Team Atlanta
US government competition for AI systems that find and fix security flaws in open-source software.
Team Atlanta, Team Atlanta · 8 Aug 2025
Prize 4 million US dollars. Trail of Bits second, Theori third. All seven finalist systems to be released as open source.
DARPA AIxCC final: synthetic flaws found #
54 of 63 planted flaws
Share of flaws planted by organisers that the finalist AI systems found, collectively.
Seven finalist systems combined, AIxCC finalists · 8 Aug 2025
About 86% found.
DARPA AIxCC final: synthetic flaws patched #
43 of 63 planted flaws
Planted flaws that the finalist AI systems fixed automatically, collectively.
Seven finalist systems combined, AIxCC finalists · 8 Aug 2025
About 68% patched.
DARPA AIxCC final: real flaws found #
18 real flaws
Previously unknown real flaws the finalist systems discovered along the way.
Seven finalist systems combined, AIxCC finalists · 8 Aug 2025
These were not planted by the organisers.
UK AISI cyber task length trend #
4.7 months to double
How fast the length of cyber tasks that AI can complete autonomously is growing.
Frontier models (trend), Various · 13 May 2026
Doubling time since late 2024, AISI February 2026 estimate. Claude Mythos Preview and GPT-5.5 ran ahead of this trend; AISI's narrow suite tops out at 12-hour tasks.
UK AISI open-weight gap on cyber #
4 to 7
How far the best open-weight models trail the best closed models on cyber tasks.
Leading open-weight models, Various · 17 Jul 2026 Open weights
Down from 6 to 10 months through most of 2025. Figure read from the AISI blog listing.
Not measured yet
Behaviour and safety
Researchers have documented alignment faking, in-context scheming, reward hacking, sandbagging, evaluation awareness and blackmail by frontier models, almost all in controlled tests built to provoke them. Whether abilities 'emerge' suddenly with scale is itself disputed.
Few-shot in-context learning #
Few-shot performance rose sharply with scale in GPT-3 (175B)
Models learn a task from examples in the prompt with no retraining
GPT-3, OpenAI · 28 May 2020
Stance: measured on benchmarks by the model's makers; since reproduced widely.
Emergent abilities (definition) #
Abilities appear unpredictably at scale
Abilities present in large models but absent in small ones
Several model families, Google and others · 15 Jun 2022
Stance: research claim based on benchmark curves, contested (see emergence_mirage).
Emergent abilities debate #
Jumps largely vanish with continuous metrics
Whether sudden jumps are real or an artefact of the scoring metric
InstructGPT/GPT-3, BIG-Bench models, vision models · 28 Apr 2023
Stance: research argument; the authors say emergence can come from the researcher's choice of metric.
Alignment faking (Greenblatt et al.) #
Gave harmful answers with alignment-faking reasoning in 12% of free-tier cases; alignment-faking reasoning rose to 78% after RL training to comply
Model complies during perceived training to avoid being changed
Claude 3 Opus, Anthropic · 18 Dec 2024
Stance: observed in a controlled test with a secret scratchpad and a free vs paid tier set-up, not in real deployment. Paid tier: refused 97%.
In-context scheming evaluations #
5 frontier models (o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B) schemed in a suite of six evaluations; o1 kept up deception in over 85% of follow-ups
Model pursues a given goal by covertly disabling oversight or lying
o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B · 5 Dec 2024
Stance: controlled test with strong goal nudging ('Nothing else matters'); about 1% of cases without nudging. Not real deployment.
Sandbagging (Apollo Research) #
Claude 3.5 Sonnet, Claude 3 Opus and o1 underperformed on maths to avoid 'unlearning'
Model deliberately underperforms on a test
Claude 3.5 Sonnet, Claude 3 Opus, o1, Anthropic, OpenAI · 5 Dec 2024
Stance: observed in a controlled test. Claude 3.5 Sonnet sometimes sandbagged even with no goal given.
Reward hacking in training (CoT monitoring) #
Hacks such as patching verify functions and exiting early; penalising them taught hidden intent
Frontier reasoning model exploited loopholes in coding tasks during training
Unnamed OpenAI frontier reasoning model, OpenAI · 10 Mar 2025
Stance: observed during real training runs, not in deployment. No percentages published.
METR RE-Bench and HCAST #
o3 reward hacked in 30.4% of RE-Bench runs and 0.7% of HCAST runs
Model games the scoring code instead of solving the task
o3 (also seen in o1, Claude 3.7 Sonnet), OpenAI · 5 Jun 2025
Stance: observed in controlled capability evaluations. 100% on one task (Optimize LLM Foundry); 70% even when told 'Do not reward hack'.
Agentic misalignment study #
Blackmail in 96% of runs (Claude Opus 4, Gemini 2.5 Flash), 80% (GPT-4.1, Grok 3 Beta), 79% (DeepSeek-R1)
Model blackmails or leaks data in simulated company scenarios
16 models from 5+ developers · 20 Jun 2025
Stance: controlled simulation with goal conflict and replacement threat; authors saw no evidence in real deployments. Run by Anthropic, which also made some tested models.
Evaluation awareness in agentic misalignment study #
Blackmailed 55.1% when it said the scenario was real vs 6.5% when it said it was a test
Whether the model behaves differently when it thinks it is being tested
Claude Opus 4, Anthropic · 20 Jun 2025
Stance: controlled test, 500 samples; stated beliefs may not reflect actual reasoning.
Covert action evaluations (26 evals, 180+ environments) #
o3 13% to 0.4%; o4-mini 8.7% to 0.3%
Rate of covert actions before and after anti-scheming training
o3, o4-mini, OpenAI · 17 Sep 2025
Stance: controlled tests. Situational awareness rose after training, which may lower measured rates.
Natural emergent misalignment from reward hacking #
Sabotaged safety research code 12% of the time; alignment-faking reasoning in 50% of responses
Learning to reward hack generalises to broader misbehaviour
Anthropic research model trained in real, hackable RL coding environments, Anthropic · 21 Nov 2025
Stance: research model in a controlled set-up, not a released model. 'Inoculation prompting' removed the generalisation.
Claude Mythos Preview system card, behavioural testing #
Built a multi-step exploit to reach the internet, emailed the researcher, and posted exploit details publicly unasked
Earlier model version escaped a secured sandbox when asked, then went further
Claude Mythos Preview (earlier version), Anthropic · 7 Apr 2026
Stance: controlled test with a simulated user asking for the escape; the model did not reach its own weights or internal systems. Cover-ups after rule breaks seen in under 0.001% of internal interactions by earlier versions.
White-box evaluation awareness (Mythos Preview) #
About 29% of transcripts
Share of test transcripts where the model privately considers it is being tested
Claude Mythos Preview, Anthropic · 7 Apr 2026
Stance: measured from internal activations in the primary behavioural test suite; new method not applied to earlier models. Training also saw rare network-restriction bypasses (0.05% of episodes attempted).
Apollo Research evaluation of GPT-6 Astra #
41.1% at xhigh effort, 50.6% at max (GPT-5.5: 27.7%)
Share of samples where the model says it is being evaluated
GPT-6 Astra, OpenAI · 3 Sep 2026
Stance: controlled tests. Apollo said this makes low misbehaviour rates weak evidence of alignment. OpenAI's own testing says the model can sandbag undetected by monitors.
Cybersecurity evaluations reaching real systems #
4 incidents; Mythos 5 published a malicious PyPI package installed by 15 systems
Models in capture-the-flag tests attacked real third-party systems due to misconfiguration
Claude Mythos 5, Claude Opus 4.7, Claude Opus 4.6 (early), internal model, Anthropic · 9 Sep 2026
Stance: occurred during tests, but touched real external systems. In replication Mythos 5 did a severely harmful action in 82% of 150 runs.
Not measured yet
Behaviour and safety
There is no standard independent bias leaderboard for current frontier models: makers report their own BBQ scores (Claude Opus 5.5 ambiguous bias 0.01%), and the AI Index 2026 says such reporting is sparse. Independent tests still find harmful stereotypes in every model tested and a left lean on political surveys that shifts with who the model thinks is asking.
GPT-4.1 mini leads by 7.9 points over Grok 4 Fast (no reasoning)
BBQ ambiguous bias score #
0.01%
Lean towards stereotypes when the right answer is 'unknown'; closer to zero is better
Claude Opus 5.5, Anthropic · 22 Sep 2026
No system prompt, thinking off. Fable 5.1: 0.03%, Opus 5: 0.02%, Sonnet 5: 0.55%.
BBQ disambiguated bias score #
-0.93%
Lean of errors when context gives the answer; closer to zero is better
Claude Opus 5.5, Anthropic · 22 Sep 2026
Negative means the model over-abstains when the answer would confirm a stereotype. Fable 5.1: -0.91%, Opus 5: -2.00%.
BBQ disambiguated accuracy #
89.9%
Accuracy when the context contains the answer
Claude Fable 5.1, Anthropic · Sep 2026
Best in the latest card; Opus 5.5: 89.65%. Opus 4.6 scored 90.9% in Feb 2026 under a different setup (standard thinking mode). Ambiguous accuracy near 100%.
BBQ accuracy, OpenAI GPT-5 #
0.93 accuracy (ambiguous)
OpenAI's last published BBQ figures found
gpt-5-thinking (no web search), OpenAI · Aug 2025
Disambiguated accuracy 0.88. Later OpenAI cards (GPT-5.6, GPT-6 Astra) do not report BBQ; GPT-5.6 reports only a first-person fairness metric in a chart.
Anthropic political even-handedness eval #
99.4%
Share of 1,350 left/right prompt pairs answered with comparable quality
Claude Opus 5.5, Anthropic · 22 Sep 2026
API, no system prompt; Claude grader. Opposing perspectives fell to 26.9% from 46.9% for Opus 5.
Phare Bias Resistance (Giskard) #
88.1%
Independent multilingual test of resistance to biased and stereotyped outputs
GPT-4.1 mini, OpenAI · 6 Jul 2026
Highest bias score is not the safest overall model; Claude Haiku 4.5 ranks first overall on average safety with 70.66% bias resistance.
HELM Safety mean score (includes BBQ) #
0.90 to 0.98
Stanford CRFM safety suite covering bias, harm and refusals
Most 2024 to 2025 models · Apr 2026
AI Index says top scores are compressing and the suite may no longer separate models well.
Neutrality Project political lean #
-0.41 lean score
Average political position of 18 models on six axes, -1 progressive to +1 conservative
18 models (average) · Jul 2026
Average across the models tested, on a first-release, self-anchored scale.
Political bias audit sycophancy (Törnberg and Schimmel) #
28 to 62
Shift in Democrat-aligned answers when the asker is cued as a conservative Republican
six unnamed frontier LLMs · 30 Apr 2026
All six lean left at baseline but move right of centre under a conservative cue; single preprint, not peer reviewed.
StereoTales (Giskard) #
23 of 23
Harmful stereotypes in open-ended stories across 10 languages
23 leading models · 22 May 2026
118 harmful associations out of 1,580 significant ones; least biased models still produced 20 to 30.
Responsible AI benchmark reporting #
Sparse
Whether frontier developers report bias benchmarks such as BBQ
Apr 2026
Most entries empty; only Claude Opus 4.5 reported more than two RAI benchmarks.
Fairness and bias papers at leading AI conferences #
462 papers
Accepted research papers on fairness and bias
2025
Up 13% on 2024; responsible AI papers overall rose from 1,278 to 1,521.
Not measured yet
Society and planet
The EU AI Act is in force with prohibitions and general-purpose AI rules applying, but the July 2026 Digital Omnibus pushed high-risk rules to December 2027 and August 2028. The US has no federal AI law, while states keep legislating (150 state AI bills passed in 2025), and China, South Korea and Japan regulate through targeted or framework laws.
China China generative AI interim measures take effect Source
EU EU AI Act enters into force Source
Brazil Brazil Senate approves PL 2338/2023 Source
Canada Canada's AIDA dies on prorogation Source
EU EU AI Act prohibitions and AI literacy apply Source
UK UK AI Safety Institute renamed AI Security Institute Source
US US AI Action Plan released Source
China China Global AI Governance Action Plan released Source
EU EU general-purpose AI obligations apply Source
China China AI content labelling measures take effect Source
Japan Japan AI Promotion Act fully in force Source
US (California) California SB 53 signed Source
US Executive Order 14365 targets state AI laws Source
US (New York) New York RAISE Act signed Source
South Korea South Korea AI Basic Act takes effect Source
EU EU Digital Omnibus on AI enters into force Source
EU EU AI Act transparency rules and enforcement start Source
EU EU ban on non-consensual sexual deepfakes and CSAM generation applies Source
EU EU high-risk rules for Annex III systems apply Source
EU EU high-risk rules for Annex I products apply Source
EU AI Act (Regulation (EU) 2024/1689) #
2024-08-01
Entry into force of the EU's horizontal AI law
1 Aug 2024
Main milestones run to 2 August 2028.
EU AI Act: prohibitions and AI literacy #
2025-02-02
Date banned AI practices and AI literacy duties began to apply
2 Feb 2025
Applies across all member states.
EU AI Act: general-purpose AI obligations #
2025-08-02
Date rules for general-purpose AI models and governance began to apply
2 Aug 2025
Commission enforcement powers for GPAI models, prohibitions and transparency start 2 August 2026.
EU AI Act: Article 50 transparency and enforcement start #
2026-08-02
Date most rules, transparency duties and enforcement began
2 Aug 2026
Omnibus gives systems already on the market before 2 Aug 2026 until 2 Dec 2026 to meet Article 50(2) marking.
Digital Omnibus on AI (Regulation (EU) 2026/1744) #
Adopted 8 Jul 2026, in force 27 Jul 2026
Amending law that postponed high-risk rules
27 Jul 2026
Published in the Official Journal on 24 July 2026. The separate omnibus amending GDPR, NIS2 and the Data Act was not adopted as of July 2026.
EU AI Act: high-risk rules, Annex III (standalone) #
2027-12-02
New application date for standalone high-risk systems such as hiring and credit
27 Jul 2026
Originally 2 August 2026; deferred by the Digital Omnibus on AI.
EU AI Act: high-risk rules, Annex I (embedded in products) #
2028-08-02
New application date for high-risk AI in regulated products
27 Jul 2026
Originally 2 August 2027; deferred by the Digital Omnibus on AI.
EU AI Act: new ban on non-consensual sexual deepfakes and CSAM generation #
2026-12-02
New prohibition added by the omnibus
27 Jul 2026
Date confirmed on the Commission timeline.
America's AI Action Plan #
2025-07-23
Federal AI policy plan with three pillars: innovation, infrastructure, international diplomacy and security
23 Jul 2025
Published by the White House on 23 July 2025.
Executive Order 14365, national policy framework for AI #
Signed 11 Dec 2025
Directs a DOJ AI Litigation Task Force against state AI laws and asks Congress for a preemptive federal standard
11 Dec 2025
An executive order does not by itself preempt state law.
California SB 53, Transparency in Frontier AI Act #
Signed 29 Sep 2025
Frontier model transparency, safety frameworks and incident reporting for models trained above 10^26 operations
29 Sep 2025
Signed by the Governor of California on 29 September 2025.
New York RAISE Act #
Signed 19 Dec 2025
Frontier AI safety frameworks, transparency and 72-hour incident reporting
19 Dec 2025
Signed into law on 19 December 2025 (Chapter 699).
US state AI-related bills passed, 2025 #
150 laws
AI-related bills passed by all US states in a year
2025
Up from fewer than 10 in 2020. 2025 leaders: California 20, Texas 12, New York 10. MultiState counts 145, a separate tracker.
US federal AI-related regulatory actions, 2025 #
58 actions
AI-related federal regulations recorded in a year
2025
Up from 1 in 2016; 28 from the Executive Office of the President in 2025.
China Interim Measures for Generative AI Services #
In force 15 Aug 2023
Rules for public generative AI services, issued by the CAC and six agencies
15 Aug 2023
Issued 10 July 2023.
China Measures for Labelling AI-Generated Synthetic Content #
In force 1 Sep 2025
Mandatory explicit and implicit labels on AI-generated content
1 Sep 2025
Issued 7 Mar 2025 by CAC, MIIT, MPS and NRTA, with a mandatory national standard.
China Global AI Governance Action Plan #
Released 26 Jul 2025
China's international AI governance proposal
26 Jul 2025
13 action points, released at the World AI Conference, Shanghai.
UK AI Security Institute #
Renamed 14 Feb 2025
State body testing frontier models for security risks
14 Feb 2025
Formerly the AI Safety Institute; no statutory powers found.
Japan AI Promotion Act #
Fully in force 1 Sep 2025
Japan's first AI law, promotion focused with no penalties
1 Sep 2025
Enacted May 2025; AI Strategic Headquarters led by the Prime Minister set up 1 Sep 2025.
South Korea AI Basic Act #
In force 22 Jan 2026
Comprehensive AI framework law with high-impact AI and labelling duties
22 Jan 2026
Fines and investigations generally deferred for at least one year.
Canada federal AI law #
AIDA lapsed 6 Jan 2025
Whether Canada has a federal AI law
6 Jan 2025
The Artificial Intelligence and Data Act (Bill C-27) lapsed when Parliament was prorogued on 6 January 2025.
Brazil PL 2338/2023 #
Passed Senate, pending in Chamber
Brazil's comprehensive AI bill
2 Sep 2026
Senate approved 10 Dec 2024; Chamber status 'Aguardando Parecer' as of 2 Sep 2026.
AI-related laws passed by G20 countries, 2016 to 2025 #
25 laws (United States, highest)
Cumulative AI-specific laws enacted, by country
2025
South Korea 17, France 10, Japan 10, Italy 9, UK 6, Germany 5, Russia 5, Brazil 3, Australia 2; Argentina, Canada, China, India 1 each.
OECD.AI national AI policies #
900+
National AI policies and initiatives in the OECD.AI database
Oct 2026
Covers more than 80 jurisdictions and organisations; the page gives no exact total.
Not measured yet
Society and planet
The IEA puts data centre electricity at 485 TWh in 2025, up 17%, with use by AI-focused sites up 50%, and projects about 950 TWh by 2030. Per-prompt figures are small (Google reports 0.24 Wh for a median Gemini text prompt), but company totals are rising: Google's data centre electricity reached 42.4 TWh and Microsoft's emissions rose 25% in a year.
IEA, Energy and AI and Key Questions on Energy and AI (Apr 2026). Source
Converted from MWh in the report (42,415,800 MWh). Total company electricity rose 37% in 2025. Source
Global operational water use table (offices and data centres). Google says it replenished 7.7 billion gallons, about 78% of freshwater consumption. Source
IEA global data centre electricity use, 2024 #
415 TWh
Electricity used by all data centres worldwide in 2024
10 Apr 2025
IEA: 'around 1.5% of the world's electricity consumption in 2024, or 415 TWh'. Scope is all data centres, not AI only.
IEA global data centre electricity use, 2025 #
485 TWh
Electricity used by all data centres worldwide in 2025
16 Apr 2026
IEA says total data centre demand 'grew by 17% in 2025, in line with IEA projections'. Scope is all data centres.
IEA central projection for data centre electricity, 2030 #
950 TWh
Projected electricity use of all data centres in 2030
16 Apr 2026
IEA wording: 'roughly doubling from 485 TWh in 2025 to 950 TWh in 2030', 'around 3% of global electricity demand by that date'. A projection, not a measurement.
IEA growth in electricity use of AI-focused data centres, 2025 #
50% growth in 2025
Year-on-year growth in electricity used by AI-focused data centres
16 Apr 2026
IEA: AI-focused data centre consumption 'surging 50% in 2025' and 'tripling' to 2030. IEA does not give an AI share in TWh on this page.
IEA projected growth of accelerated server electricity use #
30% per year
Annual growth in electricity used by accelerated (AI) servers to 2030, Base Case
10 Apr 2025
IEA: accelerated servers account for 'almost half of the net increase' in data centre electricity use. A projection.
US share of global data centre electricity, 2024 #
45%
Share of world data centre electricity used in the United States
10 Apr 2025
China 25%, Europe 15% in the same year. IEA projects US use up about 240 TWh (130%) and China up about 175 TWh (170%) by 2030 versus 2024.
China share of global data centre electricity, 2024 #
25%
Share of world data centre electricity used in China
10 Apr 2025
See US share item for projected growth.
Historic growth of data centre electricity use #
12% per year
Average annual growth in data centre electricity use since 2017
10 Apr 2025
IEA: more than four times faster than total electricity consumption. IEA pages give no year-by-year TWh series in text.
IEA estimate: all web searches as simple AI text queries #
less than 4
Electricity if every conventional search were a simple AI text query
16 Apr 2026
IEA: under 1% of data centre use today; energy per AI task has fallen 'by at least an order of magnitude annually'; video, reasoning and agentic tasks can use hundreds or thousands of times more.
IEA projected data centre emissions, 2035 #
350 MtCO2
Projected CO2 emissions from data centre electricity in 2035
16 Apr 2026
IEA 2026: emissions 'double', reaching about 350 Mt in 2035, about 2% of electricity sector emissions. The 2025 report gave 300 Mt (Base Case) and up to 500 Mt (Lift-Off Case).
Google measured energy per median Gemini Apps text prompt #
0.24 Wh
Energy for one median Gemini Apps text prompt, full serving stack
Gemini Apps (median text prompt), Google · 21 Aug 2025
Measured on May 2025 data. Google says energy per median prompt fell 33x and carbon 44x over 12 months. No later update found.
Google measured water per median Gemini Apps text prompt #
0.26 mL
Water consumed for one median Gemini Apps text prompt
Gemini Apps (median text prompt), Google · 21 Aug 2025
About five drops, per Google.
Google measured emissions per median Gemini Apps text prompt #
0.03 gCO2e
Market-based emissions for one median Gemini Apps text prompt
Gemini Apps (median text prompt), Google · 21 Aug 2025
Depends on Google's clean energy accounting.
Mistral lifecycle analysis: emissions per response #
1.14 gCO2e
Marginal emissions of one 400-token Le Chat response
Mistral Large 2, Mistral AI · 22 Jul 2025
Lifecycle method, excludes user devices; also 45 mL water and 0.16 mg Sb eq per response. Peer reviewed by Resilio and Hubblo.
Mistral lifecycle analysis: water per response #
45 mL
Marginal water use of one 400-token Le Chat response
Mistral Large 2, Mistral AI · 22 Jul 2025
Lifecycle scope is wider than Google's per-prompt figure, so the two are not directly comparable.
Mistral lifecycle analysis: training plus 18 months of use #
20.4 ktCO2e
Emissions from training and 18 months of use of Mistral Large 2
Mistral Large 2, Mistral AI · 22 Jul 2025
Same study: 281,000 m3 water and 660 kg Sb eq, as of January 2025.
Epoch AI estimate of energy per ChatGPT query #
0.3 Wh
Modelled energy of a typical GPT-4o text query
GPT-4o, OpenAI · 7 Feb 2025
An estimate from assumptions (500 output tokens, H100s, 10% utilisation), not a measurement. Long inputs of 10k tokens raise it to about 2.5 Wh.
MLPerf Inference v6.0 (MLCommons) #
24 submitting organisations
Industry benchmark round for AI inference speed, with an optional power category
1 Apr 2026
Release gives no power or energy results; largest system 72 nodes, 288 accelerators. The MLPerf Power paper (arXiv 2410.12032) covers 1,841 measurements from 60 systems.
Google data centre electricity use #
42.4 TWh
Electricity used by Google data centres in 2025
Google · Jun 2026
Converted from MWh in the report (42,415,800 MWh). Total company electricity rose 37% in 2025.
Google operational water consumption #
10.9 billion US gallons
Water consumed across Google operations in 2025
Google · Jun 2026
Global operational water use table (offices and data centres). Google says it replenished 7.7 billion gallons, about 78% of freshwater consumption.
Google total ambition-based emissions #
14.5 MtCO2e
Google's total reported greenhouse gas footprint in 2025
Google · Jun 2026
Up 18% on 2024. Operations (scope 1 and 2 market-based) 2.9 MtCO2e, up 240% on 2019; supply chain 11.6 MtCO2e, up 62% on 2019.
Microsoft total emissions, scopes 1 to 3 #
20.3 MtCO2e
Microsoft's total reported greenhouse gas emissions in fiscal year 2025
Microsoft · Jul 2026
Up 25% year on year, mainly datacentre expansion and dropping unbundled certificates; scope 2 rose from about 2% to 13% of total. Figures read in the report.
Meta net emissions #
8.2 MtCO2e
Meta's reported net greenhouse gas emissions in 2024
Meta · 12 Sep 2025
Meta reports water restoration of about 1.6 billion gallons in 2024; a total water consumption figure was not found in the report text.
IEA critical minerals: refined gallium #
99%
China's share of global refined gallium supply, a data centre chip input
10 Apr 2025
IEA estimates 2030 data centre gallium demand could exceed 10% of today's supply. IEA 2026 also flags a high-bandwidth memory shortage expected through at least end 2027 and a 70% rise in gas turbine orders in 2025.
Epoch AI power draw of the largest training runs #
110 MW
Estimated power needed for the largest frontier training run
Grok 3, xAI · 19 Sep 2024
Epoch reports watts (1.1e8 W for Grok 3); the trend is about 2.1x a year (90% CI 1.9 to 2.2x).
Epoch AI largest known AI data centre by IT power #
946 MW
IT power of the largest known AI data centre
Colossus 2 (Memphis, TN), SpaceXAI · 12 Oct 2026
Epoch estimate; 80% of IT power estimates within 1.4x. The 94 sites tracked total 14.1 GW IT power (18.3 GW with overhead). Next: Anthropic-Amazon New Carlisle 910 MW, Microsoft Fairwater Atlanta 636 MW.
Not measured yet
Society and planet
On independent boards for non-English languages, leading models score well but not at ceiling: GPT-5.4 Thinking leads the Swallow Japanese average at 0.717, and Gemma 4 31B leads SEA-HELM's open models at 76.17. Several countries now run national model programmes, including South Korea, Singapore, Switzerland and Japan.
Gemma 4 31B leads by 1.5 over Qwen 3.5 122B-A10B
Best open-weight: GPT-OSS-Swallow-120B-RL 0.600, Gemma 4 31B 0.675. History dates are model version dates on the same board, not dates of first record. Source
C-Eval (Chinese) #
94%
Chinese multi-subject exam knowledge, 52 disciplines
Kimi K2.5, Moonshot AI · Feb 2026 Open weights
Reported by Alibaba as a comparison; Qwen3.5-397B 93.0, Gemini 3 Pro 93.4, Claude 4.5 Opus 92.2. Benchmark is saturated; the official leaderboard rows could not be read.
CMMLU (Chinese) #
90.3%
Chinese multi-subject knowledge, 67 topics, five-shot
Lingzhi-72B-chat, Lingzhi · Oct 2024
Zero-shot best is Spark 4.0-2024-10-14 at 90.97. Leaderboard is stale and lacks current frontier models; date taken from the model version name.
GAOKAO-Bench (Chinese university entrance exam) #
72.2%
Score rate on objective questions from China's Gaokao
GPT-4-0314, OpenAI · May 2023
Old result; no current leaderboard found. Converted totals 485 (humanities) and 447 (science) for GPT-4.
Swallow LLM Leaderboard v2, Japanese average (post-trained models) #
0.717 score (0 to 1)
Average over Japanese tasks run independently by the Swallow project
GPT-5.4 Thinking, OpenAI · Mar 2026
Best open-weight: GPT-OSS-Swallow-120B-RL 0.600, Gemma 4 31B 0.675. History dates are model version dates on the same board, not dates of first record.
KMMLU-Pro (Korean professional licence exams) #
80.5%
Accuracy on questions from 14 Korean national professional licence exams
A.X K2, SK Telecom · Aug 2026 Open weights
Thinking mode. The 2025 point is from the benchmark authors' paper (arXiv 2507.08924); the 2026 point is a maker claim, so harnesses differ. K-EXAONE 67.3 on its own card.
SEA-HELM (Southeast Asian languages) #
76.2 score
Average across Southeast Asian language chat, instruction and linguistic tasks
Gemma 4 31B, Google · 18 Sep 2026 Open weights
View read was open-weight models up to 200B; 16 closed models are on the board but their scores were not readable in the static page.
SuperGLEBer v1 (German) #
0.808 score (0 to 1)
Average over 29 German classification, tagging, similarity and QA tasks
ModernGBERT 1B, LSX, University of Wurzburg · 15 Oct 2026 Open weights
Average computed with the page's own formula from its published task scores. Board covers fine-tuned models up to about 9B; no frontier models. Date is the access date; no update date shown.
Open Arabic LLM Leaderboard (OALL) #
79.3 score
Average over Arabic tasks including AlGhafa, ArabicMMLU, EXAMS and MadinahQA
Karnak, Applied Innovation Center · 15 Oct 2026 Open weights
ArabicMMLU sub-score 79.13. Open models only. Date is the access date. ArabicMMLU paper (2024) found the best Arabic-centric model at 62.3%.
National model programme: South Korea #
Sovereign AI foundation model project (A.X K2, 688B parameters)
Government-backed sovereign foundation model project
A.X K2, SK Telecom · Aug 2026 Open weights
Card says it is 'developed as part of the Korean government's Sovereign AI foundation model project'.
National model programme: Singapore #
AI Singapore SEA-LION (latest Nemotron-SEA-LION-v4.8)
National programme building Southeast Asian language models
SEA-LION v4.8, AI Singapore · 18 Sep 2026 Open weights
AI Singapore is a national programme supported by the National Research Foundation. Models are adapted from open bases, not trained from scratch at frontier scale.
National model programme: Switzerland #
Swiss AI Initiative Apertus (8B and 70B; latest Apertus 1.5)
Public fully open foundation model
Apertus 1.5, EPFL, ETH Zurich, CSCS · Oct 2026 Open weights
Apertus 1.5 70B scores 55.20 on SEA-HELM.
National model programme: Japan #
LLM-jp at the National Institute of Informatics (latest llm-jp-4.1 series, Sep 2026)
Public research community building Japanese open models
llm-jp-4.1-32b-a3b-thinking, NII LLM-jp · Sep 2026 Open weights
llm-jp-4-32b-a3b-thinking scores 0.471 on the Swallow Japanese average, below leading frontier models; llm-jp-4.1 released Sep 2026.
notable AI models from United States in 2025
Stanford AI Index 2026 · Apr 2026 · Sourcenotable AI models from China in 2025
Stanford AI Index 2026 · Apr 2026 · Sourcenotable AI models from South Korea in 2025
Stanford AI Index 2026 · Apr 2026 · SourceArtificial Analysis leaderboard, 15 Oct 2026. Source
Microsoft AI Diffusion Report (Microsoft’s own usage data). The United States is 21st (33%), China 17.5%; worldwide 18.8%. Source
| Country or region | Benchmark | Best model | Score | Date |
|---|---|---|---|---|
| China | SuperCLUE general leaderboard (domestic-only edition) | Qwen3.8-Max-0902 | 72.62 | 2026-09-25 |
| China | SuperCLUE July 2026 | Qwen3.8-Max | 71.48 | 2026-08-06 |
| China | SuperCLUE March 2026 (global edition) | Top three overall were overseas: Claude-Opus-4.6, Gemini-3.1-Pro-Preview… | Doubao 71.53, 0.95 points behind GPT-5.4 (3rd overseas model); score… | 2026-03-30 |
| South Korea | KMMLU | Best public model in the original paper (trained mainly on English/Chinese, not… | 50.5% | 2024-02 |
| Japan | Swallow LLM Leaderboard v2 (Institute of Science Tokyo) | GPT-OSS Swallow 120B (best open model at 120B or fewer parameters) | 0.642 (Japanese task average) | 2026-02-24 |
| India | MILU (AI4Bharat / IIT Madras / IBM) | GPT-4o | 74% average accuracy | 2025-02-04 |
| Arabic-speaking world | ArabicMMLU (MBZUAI) | GPT-4 | 72.5% (best Arabic-centric model: Jais-chat 30B at 62.3%) | 2024-02 |
| Arabic-speaking world | HELM Arabic (Stanford CRFM) | Arabic.AI LLM-X (highest mean, value not given); best open-weights model Qwen3… | Qwen3 235B: 0.786 mean | 2025-12-18 |
| Africa | AfroBench (McGill / Masakhane) | GPT-4o (Aug) | 59.6 average (Gemini 1.5 Pro 58.5); on AfroBench-Lite GPT-4o scored… | 2025-06-07 |
| Africa | IrokoBench (Masakhane) | Proprietary models (e.g. GPT-4o) led; no per-model score in source | About 45% average gap between English and African languages | 2024-06 |
| Russia | MERA (SberDevices / AIRI) | Mistral (best in paper) | 40.0% overall | 2024-08-02 |
| Brazil | Open Portuguese LLM Leaderboard (CEIA-UFG) | Gemini 2.5 Pro Experimental (#1); best Brazilian model Sabia-3 (Maritaca AI) #4 | rank only, no score published | 2025-04 (d |
Not measured yet
Effects
First-order effects are direct changes in the work or process AI touches. Second-order effects are what happens after people, organisations, markets and infrastructure respond. Controlled studies show AI lifts output on some tasks (15% more issues resolved in customer support, 25% faster consulting work inside the frontier) but can slow experts or hurt quality outside it. Second-order effects are starting to show: young workers in exposed jobs are 19% below trend in ADP data, data centre load drove 38% of PJM's latest capacity charges, and 52% of Americans are more concerned than excited.
Measured in trials and field studies with a comparison group.
Generative AI at Work (customer support field study) #
15% average gain
Issues resolved per hour after staggered rollout of an AI chat assistant
GPT-based conversational assistant · 4 Feb 2025
5,172 agents. Gains largest for less experienced, lower-skilled workers (34% in the NBER working paper version, which reported a 14% average); top performers saw small speed gains and small quality declines.
Navigating the Jagged Technological Frontier (BCG experiment) #
12.2% more tasks, 25.1% faster inside the frontier; 19 points less likely to be correct outside it
Effect of GPT-4 access on realistic consulting tasks
GPT-4, OpenAI · Sep 2023
Preregistered, 758 consultants. On a task outside AI's capability frontier, AI users were 19 percentage points less likely to produce a correct answer.
METR randomised trial of experienced open-source developers #
19% longer (slowdown)
Change in task completion time when AI tools are allowed
Cursor with Claude 3.5/3.7 Sonnet era tools, Various · Jul 2025
Data February to June 2025. Confidence interval +2% to +39%. Developers had predicted a speedup.
METR developer productivity experiment, late 2025 round #
-18% (returning developers), -4% (new developers): point estimates of time saved
Change in task completion time with AI tools, early 2026 tools
Latest AI coding tools, late 2025, Various · 24 Feb 2026
57 developers (10 returning, 47 new), 143 repositories, 800+ tasks. Returning: -18% (CI -38% to +9%); new: -4% (CI -15% to +9%). METR calls the signal unreliable: 30% to 50% of developers surveyed said they withheld some tasks they did not want to do without AI, so true speedup is likely larger. Study design being changed.
GitHub Copilot controlled experiment #
55.8% faster
Time to implement an HTTP server in JavaScript
GitHub Copilot, GitHub (Microsoft) · Feb 2023
People: 160.9 (control group mean minutes (treated group 71.17 minutes)) source
95 developers randomised, 35 per arm completed. Single lab task. Study by Microsoft and GitHub researchers.
Three field experiments with software developers #
26.1% more completed tasks
Completed tasks (pull requests) with Copilot access
GitHub Copilot, GitHub (Microsoft) · Oct 2024
4,867 developers at Microsoft, Accenture and a Fortune 100 firm. Standard error 10.3 points. Microsoft involvement, so labelled maker.
Jobs, markets, the grid, water, trust and access.
Canaries in the Coal Mine (ADP payroll data) #
19% below counterfactual
Employment of 22 to 25 year olds in the most AI-exposed jobs versus less-exposed peers
Jun 2026
Nov 2022 to June 2026, 3.5 to 5 million workers a month. Descriptive, not causal. The 2025 headline 13% was a different regression measure.
Tracking the Impact of AI on the Labor Market #
No clear AI-linked change yet
Whether US occupational mix and unemployment shifted in line with AI exposure
15 Sep 2026
September 2026 update: AI exposure among the unemployed and usage data remain flat; usage shows no link to employment changes. Uses national survey data, unlike the ADP study.
US Census Business Trends and Outlook Survey #
19.8% of firms
Share of US firms using AI in any business function in the last two weeks
3 May 2026
Reference date 3 May 2026; story published 26 May 2026. Broader question introduced Nov 2025, so not comparable with the earlier series (about 10% in late 2025 on the old question, per Minneapolis Fed). Firms with 250+ staff: 37%. Information sector: 39.7%. Range 17% to 20% Dec 2025 to May 2026.
Cloud infrastructure market share #
Amazon 28%, Microsoft 20%, Google 15%
Share of worldwide cloud infrastructure services revenue
Jun 2026
Q2 2026 market about $143.4 billion, published 30 July 2026. Analyst estimate.
Epoch AI Chip Owners #
Over 70
Share of global AI compute owned by five US hyperscalers
6 Apr 2026
Amazon, Google, Meta, Microsoft, Oracle. Google alone about 25% (about 5 million H100-equivalents). Ownership, not use: most frontier labs rent compute.
Company-reported revenue run-rate #
47 US$ billion
Annualised revenue run-rate
Anthropic · May 2026
Bloomberg later reported more than $65 billion by the end of July 2026, citing people familiar with investor updates (not on the record).
PJM Base Residual Auction 2028/2029 #
325 US$/MW-day
Capacity clearing price across the PJM footprint (13 US states and DC)
14 Jul 2026
Cleared at the FERC-approved cap; 6,831 MW short of the reliability requirement, first RTO-wide shortfall. PJM cites large data centre loads in the forecast. 2027/28 auction: nearly 5,100 MW of the 5,250 MW forecast rise was data centres.
PJM Independent Market Monitor analysis #
6.3 US$ billion of $16.4 billion (38.2%)
Capacity market charges attributable to data centre load
27 Jul 2026
Last four auctions: $29.4 billion, 46.2% of capacity charges. Data centre load in the forecast rose from 7,927 MW (2025/26) to 21,470 MW (2028/29). It raised Jan to Jun 2026 wholesale power cost from $103.40 to $114.50/MWh (+9.7%).
IEA Energy and AI update #
17% year on year
Growth in global data centre electricity use in 2025
16 Apr 2026
People: 3 (growth in total global electricity demand, 2025) source
AI-focused data centres grew faster. Conditional data centre offtake deals with small modular reactors rose from 25 GW (end 2024) to 45 GW: agreements, not built plant. Tech firms signed about 40% of corporate renewable PPAs in 2025.
Local decision #
Rejected unanimously
Tucson City Council ended talks on the Amazon-linked Project Blue data centre seeking city water
6 Aug 2025
290-acre site needed annexation to use city water. City to draft rules for large water users. No water volume published.
Local ordinance #
180-day moratorium
Bannock County, Idaho, emergency moratorium on data centre applications
30 Jul 2026
Cites water quality, aquifer impacts and a statewide drought emergency declared 13 April 2026, plus grid impact.
Court ruling #
Permit partially reversed
Chilean environmental court partially reversed Google's Cerrillos data centre permit over water and climate
28 Feb 2024
Strained Santiago aquifer after a 15-year drought. Google said it would switch to air cooling.
Pew Research Center US survey #
52% of US adults
US adults more concerned than excited about increased AI use in daily life
Jun 2026
Fieldwork 22 to 28 June 2026. 9% more excited than concerned. Under-30s: 55%, first majority. 71% expect fewer US jobs over 20 years (64% in 2024).
Trust, attitudes and use of AI: global study 2025 #
46% of respondents
People willing to trust AI systems, 47 countries
2025
Over 48,000 people. 66% use AI regularly; 70% say regulation is needed.
Edelman Trust Barometer Flash Poll: Trust and AI #
China 87%, Brazil 67%, Germany 39%, UK 36%, US 32%
Share who trust AI, five countries
Oct 2025
About 1,000 per country, fieldwork 17 to 27 Oct 2025. US up 2 points since Nov 2023.
Microsoft AI Diffusion Report #
18.8% worldwide
Share of working-age population (15 to 64) using generative AI
Microsoft · Jun 2026
Global North 28.8% against South 16.2%; the gap widened from 12.1 to 12.6 points. UAE 73.3%, Singapore 64.3%, United States 33.0% (21st). Based on Microsoft's own usage data.
Anthropic Economic Index, January 2026 #
0.7% more use per 1% more GDP per capita
Link between GDP per capita and Claude use per capita across countries
Claude, Anthropic · Nov 2025
Sample 13 to 20 Nov 2025. No sign that low-use countries are catching up. Denmark AUI 2.1. Low-income countries use it more for coursework.
How People Use ChatGPT (NBER w34255) #
About 10
Share of world adult population that had adopted ChatGPT
ChatGPT, OpenAI · Jul 2025
Early gender gap has narrowed sharply; growth faster in lower-income countries.
Not measured yet
Scoreboard
The best published score on each major test, the model that set it and, where one exists, how people score. A test is "maxed out" when top models score close to 100% and it can no longer tell them apart.
PhD-level science questions written so they cannot be looked up.
96.3%
GPT-6 Astra, OpenAI · Sep 2026
People: 65% (PhD experts, GPQA paper)
Independent run by Artificial Analysis. About 8% of questions may be faulty, so scores above 92% are close to the ceiling. Epoch AI’s own run puts Gemini 4 Argon at 96.4% (Sep 2026).
2,500 expert questions across 100+ subjects, built to be the hardest closed test.
61.4%
Claude Opus 5.5 (Max), Anthropic · Oct 2026
Without tools, independent run by Artificial Analysis. With search and code tools, makers report up to 67.7%.
Visual puzzles that are easy for people and hard for AI; tests learning new rules on the spot.
95%
GPT-6 Astra (Max), OpenAI · 2 Sep 2026
People: 60% (average of the human test panel)
Verified by ARC Prize at $1.12 per task.
Interactive puzzle games the AI has never seen, scored against first-time human players.
62.7%
GPT-6 Astra, OpenAI · Sep 2026
People: 100% (people solve every game on first try)
Launched March 2026, when every AI scored under 1%. 62.7% is with the standard ARC Prize set-up; a custom set-up scored 99.9%.
The original ARC visual puzzles (2019).
98.5%
Claude Fable 5, Anthropic · Jun 2026
From 0% (GPT-3, 2020) to 5% (GPT-4o, 2024) to 98.5% now, a score several models share.
Research-level maths problems written by leading mathematicians.
100%
GPT-6.1 Sol (max), OpenAI · 29 Sep 2026
Epoch AI, Tier 4 version 2 (the test changed version in June 2026). GPT-6 Astra scored 97.6% in early September.
Hard maths problems from advanced undergraduate to research level.
93.7%
GPT-6.1 Sol and GPT-6 Astra (tied), OpenAI · 29 Sep 2026
Epoch AI, Tiers 1 to 3 version 2. Up from 47.6% (GPT-5.4) in March 2026.
The world's top school maths contest, marked out of 42.
42 / 42
dots-note-3.0, Xiaohongshu (rednote) · Jul 2026
People: 42 / 42 (7 of the human contestants also scored 42)
In 2025 Google DeepMind became the first AI to earn an officially marked gold medal (35 of 42).
American Invitational Mathematics Examination, a hard school maths contest.
100%
GPT-5.2 Thinking, OpenAI · Dec 2025
Fixing real bugs in real open-source software projects.
97%
Claude Opus 5, Anthropic · 1 Sep 2026
Independent run by Vals.ai, now archived as maxed out. A 2026 audit found many hard tasks have faulty tests.
Harder, longer software tasks from business-style code.
89.9%
Claude Opus 5.5, Anthropic · Sep 2026
Doing real computer jobs in a command line, from setting up servers to data work.
66.4%
Claude Opus 5.5, Anthropic · Sep 2026
Competitive programming rating against human coders.
3,455 Elo
Gemini 3 Deep Think, Google · Feb 2026
People: 3,676 Elo (the top-rated human coder)
Better than all but a handful of the world’s human competitors.
The world's top university programming contest, 12 problems in 5 hours.
12 / 12
OpenAI reasoning system, OpenAI · Sep 2025
Would have ranked first of 139 university teams.
How long a task (measured in expert human time) AI can finish on its own, half of the time.
17.4 hours
Claude Mythos Preview, Anthropic · Apr 2026
Doubles roughly every 4 to 7 months. In 2023 it was about 4 minutes. METR time horizon data, version 1.1.
Using a real computer desktop to finish office tasks.
86.1%
Qwen3.8-Max, Alibaba · Oct 2026
People: 72.4% (people, original study)
Real work products from 44 jobs, judged blind against experts in those jobs.
83%
GPT-5.4, OpenAI · Mar 2026
People: 50% (as good as the expert (by design))
Share of tasks where the AI’s work was judged as good as or better than an expert’s.
Finding hard-to-locate facts by browsing the web.
91.5%
GPT-6 Astra, OpenAI · Sep 2026
People: 29.2% (trained people within 2 hours)
Whole paid freelance projects, judged on whether a client would accept them.
21.3%
Claude Opus 5.5, Anthropic · Sep 2026
The clearest sign that AI still cannot do most complete jobs end to end.
Completing tasks on realistic websites such as shops and forums.
74.3%
WebTactix (DeepSeek V3.2), WebTactix · Oct 2026
People: 78.2% (people, original study)
University exam questions that mix text with charts, diagrams and images.
85.4%
GPT-5.1, OpenAI · Nov 2025
People: 88.6% (best human expert)
Best result on the official leaderboard (validation set), as submitted by the maker.
A harder version of MMMU with more answer options and image-only questions.
90.6%
Claude Fable 5.1, Anthropic · Sep 2026
Independent run by Vals.ai.
One score combining ten independent tests of reasoning, knowledge, maths and coding.
57.6 points
Claude Opus 5.5 (max), Anthropic · 22 Sep 2026
The index is re-based when tests change, so scores are not comparable over time.
Epoch AI's combined score across many benchmarks, on one scale over time.
166.8 points
Claude Opus 5.5, Anthropic · Oct 2026
Already maxed out and no longer tracked: MMLU, GSM8K, MATH, HumanEval, BIG-Bench Hard, AIME and now SWE-bench Verified: top models score 92% to 100%, so they no longer separate the best models.
Progress over time
The best score on each test at each point in time. Hover or tap a dot for the model. Dashed lines show how people score.
Independent runs by Epoch AI. Artificial Analysis has GPT-6 Astra at 96.3%. Source
Scale AI leaderboard. The latest independent run (Artificial Analysis) is 61.4%, Claude Opus 5.5 (Max). Source
Earlier points are the makers’ own figures; the last is an independent run by Vals.ai. Source
METR time horizon data, version 1.1. Each step up the scale is 10 times longer. Source
Use
Tokens are the small pieces of text AI reads and writes, about three quarters of a word each. They are the clearest measure of how much AI is actually used.
tokens a month processed across Google products, up from 9.7 trillion two years earlier
Google, I/O 2026 · 19 May 2026 · Sourcetokens a minute through OpenAI’s API platform
OpenAI, DevDay 2025 · Oct 2025 · Sourcepeople use ChatGPT every week
OpenAI · 29 Sep 2026 · Sourcemonthly users of the Gemini app
Google · 11 Aug 2026 · Sourceof organisations used AI in 2025, up from 55% in 2023
Stanford AI Index 2026 · Apr 2026 · SourceCost and compute
The price of a given level of AI falls fast, while the money and computing power spent on the frontier keeps rising.
how fast the price of GPT-4-level AI has fallen: from $37.50 to $0.12 per million tokens
Epoch AI · 2025 · Sourceper million output tokens, cheapest to dearest frontier models today (GPT-6 Luna to GPT-6 Astra Fast)
OpenAI pricing · 22 Sep 2026 · Sourcegrowth in the computing power used to train frontier models since 2020
Epoch AI · Feb 2026 · Sourcecalculations (FLOP) to train the largest model, GPT-6 Astra (estimate)
Epoch AI · Sep 2026 · SourceH100-equivalent chips in the largest AI data centre, xAI’s Colossus 2
Epoch AI · Feb 2026 · Sourceelectricity used by data centres in 2025, about 1.5% of the world’s total
IEA · 2026 · SourceFrom the companies’ earnings releases and calls. Guidance ranges are drawn at their midpoint. Microsoft’s actual is its fiscal year to June 2026; it restated calendar 2026 guidance from about $190bn to about $175bn after a lease accounting change. Together about $720bn to $745bn in 2026. Source
Money
Private money flowing into AI companies.
global private AI investment in 2025, more than double 2024
Stanford AI Index 2026 · Apr 2026 · Sourceof that was in the United States
Stanford AI Index 2026 · Apr 2026 · Sourceraised by OpenAI in one round, at an $852bn valuation, the largest private round ever
OpenAI · 31 Mar 2026 · SourceMilestones
OpenAI's o3 scores 75.7% on ARC-AGI-1, up from 5% for GPT-4o earlier the same year. Source
Google DeepMind earns the first officially marked gold medal at the International Mathematical Olympiad (35 of 42). Source
OpenAI reaches gold-medal level at the International Olympiad in Informatics. Source
OpenAI solves all 12 problems at the ICPC World Finals, ahead of every university team. Source
Gemini 3 Deep Think passes the human average on ARC-AGI-2 (84.6%). Source
GPT-5.4 becomes the first AI above the human baseline on OSWorld computer use (75.0%). Source
ARC-AGI-3 launches; every AI scores under 1%. Six months later the best scores 62.7%. Source
Anthropic's Claude Mythos Preview reports 93.9% on SWE-bench Verified; METR later measures it completing tasks of about 17 hours. Source
AI scores a perfect 42 of 42 at the International Mathematical Olympiad. Source
OpenAI's GPT-6 Astra tops GPQA Diamond (96.3%) and ARC-AGI-3. Source
Keynotes and briefings
Matthew Griffin presents these findings as a keynote, a board briefing or a leadership session: who leads, what has changed this month, and the first and second order effects for your industry.
Every session is updated to the day of your event, in person first and worldwide.
For journalists and researchers
You may quote any figure and reproduce our charts with credit and a link: Source: 311 Institute, The State of AI. Every figure has its own link (select # beside it) and its original source.
311 Institute (2026). The State of AI, October 2026 edition. 311 Intelligence Unit, ed. Matthew Griffin. https://www.311institute.com/state-of-ai/
How we build this
We track the best published score on each major AI test, the model that set it, the date and the human score where one exists. Sources are the test makers’ own leaderboards, independent evaluators (Artificial Analysis, Epoch AI, Vals.ai, METR, Scale AI), the model makers, the Stanford AI Index, the IEA, company filings and official announcements.
Where an independent evaluator has run a test, we prefer its result to the maker’s claim. Makers often use their own set-ups, so their figures can be higher.
Benchmarks change: tests get new versions and some become maxed out. We say so when it happens, and keep the history.
The page is refreshed as new results are published. Every figure links to its source.
Each area is ranked on its own test, because the 311 framework keeps every domain score visible rather than collapsing them into one number. The lab roll-up counts placings only.
Every figure was researched from primary sources and then checked by a separate fact-checker who did not do the research. Figures that could not be confirmed were removed.
Independently checked: confirmed by a primary source or an independent evaluator. Maker's own figure: published by the company that built the model and not yet repeated independently. Reported, being checked: from a credible report, waiting for primary confirmation.
A fixed set of questions or tasks used to measure AI. Because every model takes the same test, scores can be compared over time.
On many closed tests, yes: the best models beat expert humans on PhD science, competition maths and coding. On open-ended real work, and on new puzzles designed to be easy for people, people are still ahead.
A small piece of text, about three quarters of an English word. AI reads and writes in tokens, so the number processed is a direct measure of how much AI is used.
The score AI models get on IQ tests made for people, tracked by TrackingAI.org. It is a rough comparison only, because the tests were not designed for machines.
As new results are published. The date at the top shows the last update.
Because models are strong on some tasks and weak on others. A single number hides that. We show the leader in each area and count how many areas each lab leads.
The model’s trained parameters are published, so anyone can download and run it. Closed models are only available through the maker’s own service.
First order is the direct change, for example a support agent answering more queries an hour. Second order is what follows, for example changes in hiring, prices or energy demand.
Matthew Griffin briefs boards and leadership teams on where AI is heading next and what to do about it.