Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 7 min read

Gemini 4 Neon Benchmarks: Coding, Agents and Cost per Task

Gemini 4 Neon benchmarks put it #1 on the Artificial Analysis Coding Agent Index (58.6), at 65.2% on Terminal-Bench 4.1 and #2 on the Vals Index (70.9%). Google also says it costs 32% less per task than Gemini 4 Carbon a

Gemini 4 Neon Benchmarks: Coding, Agents and Cost per Task

Gemini 4 Neon benchmarks put it #1 on the Artificial Analysis Coding Agent Index (58.6), at 65.2% on Terminal-Bench 4.1 and #2 on the Vals Index (70.9%). Google also says it costs 32% less per task than Gemini 4 Carbon at its default high thinking level. Most rows were run at xhigh, though, and some on Google’s own harness.

Try Apidog today

That last detail is the point of this post. Our Gemini 4 Neon overview covers specs and pricing. Here you get every published score, who measured it, at which thinking level, and how to turn β€œcost per task” into a number you can check against your own bill.

Gemini 4 Neon benchmarks at a glance

Gemini 4 Neon is Google’s middle Gemini 4 model, released November 18, 2026, priced at $3 per million input tokens and $15 per million output tokens. Its strongest published results are in agentic coding and knowledge work, where it ranks #1 or #2 on four independent leaderboards.

These are the headline rows from Google’s launch post:

Gemini 4 Neon headline benchmark table

Google also reports #1 on Vals AI’s Legal Research Bench and Tax Agent Bench.

Gemini 4 Neon benchmark scores grouped by who measured them: third-party leaderboards (AA Coding Agent Index 58.6, Vals Index 70.9%, Agent Arena 1,336, Terminal-Bench 4.1 65.2%, AutomationBench 56.3%) and Google self-computed rows (SWE-sweep 39.0%, Agents' Last Exam 75.8%, Terminal-Bench Science 70.5%)

Gemini 4 Neon coding benchmarks

Coding is where Neon’s case is strongest, and also where the harness matters most. Google’s evaluation methodology PDF explains how each coding row was produced.

Artificial Analysis Coding Agent Index: 58.6, #1. This is an independent composite, so it’s the row least shaped by Google’s setup. You can check the current standings on Artificial Analysis.

Terminal-Bench 4.1: 65.2%. Artificial Analysis ran Neon using the public Antigravity-CLI harness, version 1.5.9. Other models’ scores come from the official leaderboard, and every result uses cheating-adjusted accuracy. Fair, but Neon ran inside Google’s own agent tooling.

FrontierCode 2.0: 46.2%. This one is from the public leaderboard and was run at high, the default you get in production. If you only trust one coding number for everyday traffic, trust this one.

SWE-sweep: 39.0% in Antigravity, 12.8% on mini-SWE. Both are self-computed by Google. The 3x jump comes from the harness: Google notes that Antigravity-CLI includes a default tool that fans sub-agents out across the codebase, which suits bug-finding tasks. Opus 5.5 ran in Claude Code and GPT-6.2 Sol in Codex for the same comparison. Read 12.8% as the model and 39.0% as the model plus its best tooling.

Same model, two harnesses: Gemini 4 Neon scores 12.8% on SWE-sweep with the mini-SWE agent and 39.0% in Google Antigravity, where sub-agents fan out across the codebase

Agent and knowledge-work benchmarks

These rows matter if you plan to run Neon unattended across many steps.

  • Agent Arena Index: 1,336, #2. Built from 4,200+ real-world agent sessions. Google also says Neon ranks #2 there on steerability, meaning it adapts cleanly when a user corrects it mid-task.
  • AutomationBench: 56.3%, #1. Zapier’s benchmark for business automation, from the public leaderboard.
  • Vending-Bench 2: $16,843, #1. Andon Labs’ long-horizon simulation of running a business. The score is money earned, not a percentage.
  • Agents’ Last Exam v1.1: 75.8%. Self-computed on the 122-task set inside Antigravity-CLI 1.4.8 with a 5-hour window per run. Opus 5.5’s score comes from the public leaderboard and GPT-6.2 Sol’s from its system card, so the setups differ.
  • Chartography: 73.5%. Reading dense professional charts, self-computed without tools.

For science and math, Neon posts 70.5% on Terminal-Bench Science (self-computed, averaged over 5 attempts), 84.0% on RiemannBench and 71.3% on HLE-Diamond (from the Artificial Analysis leaderboard).

How Google ran the numbers

Four rules from the methodology change how you should read the table:

  1. Highest thinking level by default. β€œAll of the results are run with the Gemini API with the highest thinking settings unless indicated otherwise.” That’s xhigh. Only FrontierCode 2.0 is explicitly marked high. Production calls default to high, so expect fewer thinking tokens and possibly lower scores than the table shows.
  2. Pass@1, single attempt. No majority voting or parallel test-time compute, except where noted (Terminal-Bench Science averages 5 attempts).
  3. Competitor numbers are self-reported. Non-Gemini scores come from each provider unless noted. Opus 5.5 and GPT-6.2 Sol use their maximum reasoning setting where available.
  4. Safety filters on, except cyber. The cybersecurity rows ran with filters off.

Our thinking levels guide covers when xhigh is worth its extra tokens.

Cost per task: what the 32% claim means

Cost per task is the total spend to finish one benchmark task, not the price per token. A model with a higher token price can still cost less per task if it reaches the answer in fewer steps and fewer thinking tokens.

Google makes three cost claims:

Claim Thinking level Compared with
32% lower cost per task, with a high AA Intelligence Index score high Gemini 4 Carbon
Roughly 50% to 80% lower cost per task on Vals Index and Agent Arena high Top competing frontier models
36%+ lower cost per task xhigh Competing models at their highest setting

Note the mismatch: the accuracy rows mostly use xhigh, while the headline 32% saving is at high. Both are true, but not at the same time. You get the cost saving or the top score, and the gap between them is yours to measure.

Here’s how to estimate your own cost per task. Thinking tokens bill as output.

$$\text{cost} = \frac{\text{input} \times 3 + \text{cached input} \times 0.15 + \text{output} \times 15}{1{,}000{,}000}$$

A coding task with 20,000 input tokens and 4,000 output tokens costs $0.06 + $0.06 = $0.12. If 15,000 of those input tokens hit the cache, it drops to $0.015 + $0.00225 + $0.06 = about $0.077. Run the same task at xhigh and the output count is what moves, so log tokens per level, not just pass or fail. Our Gemini 4 Neon pricing breakdown covers Batch, Flex and Priority rates.

Cyber and safety rows

The cyber scores come from β€œGemini 4 Neon for cyber,” which goes to trusted defenders through the Fairwind Program rather than the public API.

Benchmark Score Note
SEC-bench Pro 90.5% State of the art, roughly half the cost per challenge
WebExploitBench v1.2 51.26% State of the art, half the cost of large models
AutoBaxBench (hard tier) 76% State of the art, more secure code than Carbon on every tier
CWE-bench v2 70.7% 3.4% uplift over Carbon
ExploitGym Not quoted Frontier-level using half the tokens of Carbon

Safety numbers from the same post: Neon refused 1.98% of Terminal-Bench 4.0 tasks, scored 0.0% cheating on Terminal-Bench 4.0, took shortcuts on 2.45% of Google’s coding cheating evaluation, and passed 99.2% of its agent honesty evaluation. With real-time safeguards at its highest thinking level, it resisted every attack in Gray Swan’s indirect prompt injection benchmark.

Where Neon doesn’t lead

Neon isn’t first everywhere, and Google says so:

  • Artificial Analysis Intelligence Index: #3 overall, β€œjust behind larger, slower models.”
  • Vals Index: #2, not #1.
  • Agent Arena Index: #2.

Google also still points you to Carbon for its hardest long-horizon projects. If you’re choosing between the three Gemini 4 models, our Neon vs Carbon vs Hydrogen comparison maps the trade-offs, and the Gemini 4 Argon benchmarks post shows how the earlier model’s table read.

Benchmark Gemini 4 Neon yourself with Apidog CLI

Leaderboards describe Google’s harness, not your product. The fastest way to know whether Neon earns its price is to let your coding agent run the same prompts at high and xhigh and log tokens and latency for each.

There’s no free API tier for Neon, so you need a paid Gemini API key. This request follows Google’s documented Interactions API format; we haven’t live-run it for this post:

curl -s "https://generativelanguage.googleapis.com/v1beta/interactions" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gemini-4-neon", "input": "Find the race condition in this retry queue.", "generation_config": {"thinking_level": "high"}}'

Apidog CLI lets Claude Code, Codex, Cursor or OpenCode create and run API tests from the terminal. Set it up once:

npm install -g apidog-cli@latest
apidog login --with-token <YOUR_TOKEN>
npx -y skills add https://apidog.com   # pick your agent when prompted

Then give your agent one prompt:

In Apidog project <id>, create a Gemini API environment with GEMINI_API_KEY
and THINKING variables. Add a test case for POST /v1beta/interactions on
gemini-4-neon that sends my 10 eval prompts from prompts.csv, asserts HTTP 200
and a non-empty output, and records token usage and response time. Run it at
THINKING=high and THINKING=xhigh and compare cost per task at $3/$15 per 1M.

The agent creates the environment, checks the test case payload against its schema, then runs it once per thinking level:

apidog environment create gemini-api --project <id> --base-url https://generativelanguage.googleapis.com
apidog cli-schema get test-case-create
apidog cli-schema validate test-case-create --file case.json
apidog test-case create --project <id> --file case.json
apidog run --test-case <caseId> --project <id> -e <envId> -d prompts.csv --env-var "THINKING=high" -r cli,json
apidog run --test-case <caseId> --project <id> -e <envId> -d prompts.csv --env-var "THINKING=xhigh" -r cli,json

Add --timeout-request 900000 if your prompts are long, since hard xhigh runs can take minutes. To rerun the comparison on every prompt change, put the same apidog run ... -r junit into CI with the Apidog CLI CI/CD guide, or start from Apidog CLI in Claude Code. Apidog’s Free plan covers up to 4 users.

FAQ

What is Gemini 4 Neon’s best benchmark result? #1 on the Artificial Analysis Coding Agent Index at 58.6, plus #1 on AutomationBench (56.3%) and Vending-Bench 2 ($16,843).

What thinking level were the benchmarks run at? Mostly xhigh, the highest setting. Google’s methodology says results use the highest thinking settings unless noted; FrontierCode 2.0 (46.2%) is marked as the default high.

Is Gemini 4 Neon cheaper than Carbon per task? Google says 32% lower cost per task at high. That’s a per-task figure from benchmark runs, so confirm it on your own prompts before moving traffic.

Why does Neon score 39.0% and 12.8% on SWE-sweep? Different harnesses. 39.0% is inside Google Antigravity, which fans sub-agents out across the codebase; 12.8% is the simpler mini-SWE agent.

Are Gemini 4 Neon’s benchmarks independent? Partly. The Artificial Analysis, Vals, Arena, AutomationBench and FrontierCode rows come from third parties. SWE-sweep, Agents’ Last Exam and Terminal-Bench Science are self-computed by Google.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.