Thursday, November 19, 2026
  • About
  • Advertise
  • Privacy & Policy
  • Contact
Unlock Your Potential With AI | AI Mind Center
  • AI News & Trends

    Edge0-145B on a laptop? The 29GB claim, checked

    Gemini2.5 Pro (I/O edition) First Look: How to Get Access

    Gemini2.5 Pro (I/O edition) First Look: How to Get Access

    May the Fourth be with you: Launching AI MIND CENTER

    May the Fourth be with you: Launching AI MIND CENTER

  • AI Productivity Tools

    Seedream 5.5 blind test: FLUX 3 quality, 15x cheaper

    AI influencer character reference and cloned voice waveform on the left, three phone videos of the character on the right

    How to Create an AI Influencer: The Full Pipeline, the Failures, and What It Actually Costs

    The 17 AI Tools Actually Worth Using in 2026

    ⚡ Best AI Writing Tools 2025: Transform Your Content Creation Game

    ⚡ Best AI Writing Tools 2025: Transform Your Content Creation Game

    Top 15 AI Tools Every Developer Should Be Using in 2025

    Top 15 AI Tools Every Developer Should Be Using in 2025

  • AI for Developers

    Same model, up to 5x the cost? What Runta’s test says

    MAI Code 2: 96% of Opus at 5% of the cost?

    Getting Started with OpenAI’s API: A Practical Guide for Developers (2026 Edition)

    Getting Started with OpenAI’s API: A Practical Guide for Developers (2026 Edition)

    How to Use GitHub Copilot: From Beginner to Advanced (Complete 2026 Guide)

    Streamlining Workflows with Semantic Kernel

    Streamlining Workflows with Semantic Kernel

    Step-by-Step: Setting Up Microsoft.Extensions.AI

    Step-by-Step: Setting Up Microsoft.Extensions.AI

  • AI in Game Development

    The Ultimate Guide to AI-Generated Assets in Unreal Engine 5.3

    7 AI Game Development Tools that Are Changing the Industry

    7 AI Game Development Tools that Are Changing the Industry

  • AI Agents & Automation
No Result
View All Result
  • AI News & Trends

    Edge0-145B on a laptop? The 29GB claim, checked

    Gemini2.5 Pro (I/O edition) First Look: How to Get Access

    Gemini2.5 Pro (I/O edition) First Look: How to Get Access

    May the Fourth be with you: Launching AI MIND CENTER

    May the Fourth be with you: Launching AI MIND CENTER

  • AI Productivity Tools

    Seedream 5.5 blind test: FLUX 3 quality, 15x cheaper

    AI influencer character reference and cloned voice waveform on the left, three phone videos of the character on the right

    How to Create an AI Influencer: The Full Pipeline, the Failures, and What It Actually Costs

    The 17 AI Tools Actually Worth Using in 2026

    ⚡ Best AI Writing Tools 2025: Transform Your Content Creation Game

    ⚡ Best AI Writing Tools 2025: Transform Your Content Creation Game

    Top 15 AI Tools Every Developer Should Be Using in 2025

    Top 15 AI Tools Every Developer Should Be Using in 2025

  • AI for Developers

    Same model, up to 5x the cost? What Runta’s test says

    MAI Code 2: 96% of Opus at 5% of the cost?

    Getting Started with OpenAI’s API: A Practical Guide for Developers (2026 Edition)

    Getting Started with OpenAI’s API: A Practical Guide for Developers (2026 Edition)

    How to Use GitHub Copilot: From Beginner to Advanced (Complete 2026 Guide)

    Streamlining Workflows with Semantic Kernel

    Streamlining Workflows with Semantic Kernel

    Step-by-Step: Setting Up Microsoft.Extensions.AI

    Step-by-Step: Setting Up Microsoft.Extensions.AI

  • AI in Game Development

    The Ultimate Guide to AI-Generated Assets in Unreal Engine 5.3

    7 AI Game Development Tools that Are Changing the Industry

    7 AI Game Development Tools that Are Changing the Industry

  • AI Agents & Automation
No Result
View All Result
Unlock Your Potential With AI | AI Mind Center
No Result
View All Result
Home AI for Developers

Same model, up to 5x the cost? What Runta’s test says

Felipe Machado by Felipe Machado
November 19, 2026
in AI for Developers
9
0
16
SHARES
130
VIEWS
Share on FacebookShare on XShare on Linkedin

According to Runta, with the same model, the most expensive coding-agent harness cost 1.8 to 5.0x as much per task as the cheapest. The same study says 64.5% of the list-price cost went to runs that did not pass. The thread had 418 points and 133 comments on Hacker News.

Title and summary of Runta's article introducing FrontierHarness Eval v1.1
Title and summary of Runta's article introducing FrontierHarness Eval v1.1

A harness is the program around the model. The model is the brain. The harness is the tool that reads your files, runs commands, feeds results back and decides when to stop. Codex CLI, OpenCode, Cline CLI and Pi are harnesses.

The facts

  • The test. 22 harness configurations, 5 open models, 62 terminal tasks, 4 attempts each: 27,280 agent runs and 83B tokens. Source: Runta.
  • The gap. With the model fixed, the costliest harness was 1.8 to 5.0x the cheapest per task (1.9 to 4.9x per pass). Biggest case: with DeepSeek, Cline CLI cost 5.0x OpenCode v1 for 2.0 more points of pass rate. With Muse, Runta’s Figure 4 shows Claude Code as the most expensive harness, $1.98 per task against $0.702 for the cheapest (2.8x). Same source.
  • The model still sets most of the bill. According to Runta, the model explains 92.8% of the variance in cost per task, the harness alone 2.3%. The 1.8 to 5.0x gap is what is left once you have already picked the model. Same source.
  • Cost does not buy success. Rank correlation between cost and pass rate was -0.39 to +0.22 per model. Same source.
  • Failures. 64.5% of list-price cost went to trials that did not pass: 52.7% failures, 11.8% timeouts. Same source.
  • Overall. OpenCode v2 had the lowest cost per task ($0.561). ZCode had the highest pass rate (61.9%). Pi and Codex CLI are also on the cost-versus-pass-rate frontier. Same source.
Figure 1: scatter plot of cost versus pass rate for harnesses across all models
Figure 1: scatter plot of cost versus pass rate for harnesses across all models

Best value per model, according to Runta

Model Best-value harness
GLM-5.3 Codex CLI ($0.803 per pass)
GLM-5.3 Flash Pi
DeepSeek V4.1 Flash ZCode
Muse Oh My Pi
Kimi-K3 Alma

The best harness depends on the model. Codex CLI ranked 1st on cost per task with GLM-5.3 and 16th with Kimi-K3. The median harness moved 11 of 22 places between models.

Figure 4: cheapest versus most expensive harness for each model, log scale
Figure 4: cheapest versus most expensive harness for each model, log scale

Why steps, not prompt size, set the cost

Each step sends the whole context again, so a trial used 2.0 to 4.8M input tokens on average. According to Runta, steps per pass predict cost per pass (+0.51 rank correlation, p=0.015). Prompt size, tool count and cache hit rate do not (-0.03 to +0.02). Example: mini-SWE-agent has the smallest first request (882 tokens) but needs 38 steps per pass and ranks 17th of 22 on cost per pass.

What it changes

Developers building apps and solo founders: if you pay for agent tokens, picking only the model may leave up to 5x on the table. Method, which is ours:

  1. Pick one model and two harnesses.
  2. Run both on 10 to 20 of your own real tasks.
  3. Divide cost per task by pass rate. That is the cost per completed task.

Illustrative numbers, invented by us, not Runta data: harness A costs $0.50 per task and passes 50%, so $1.00 per completed task. Harness B costs $0.70 and passes 80%, so $0.875. The pricier one wins. Runta’s $0.803 per pass for Codex CLI with GLM-5.3 is this same metric. Runta warns the pass-rate 95% interval is about ±10 points, so use enough tasks.

Anyone delegating to agents: Runta says 68% (Pi) to 82% (Codex CLI) of failed runs in the five best-value harnesses ended with a confident final answer, not a time limit. The agent did not run out of time. It said “done” and was wrong. Our suggestion: make “run the checks, then stop” part of the task, and count a run as passed only after it.

Game developers and gamers: nothing direct here. Ignore it unless you run coding agents.

Read this before trusting it

  • It is Runta’s own benchmark, and Runta sells the runtime where it ran. The founder says so on Hacker News.
  • Open models only, from one provider (Baseten). Runta plans closed models such as Claude Opus 5.5, but they are not tested yet. We do not know which closed-model tools will rank where.
  • The Hacker News thread raised method criticism, for example that steps per pass explaining cost is close to circular.
  • We did not run this test ourselves.

Our take

The direction is credible and checkable: the harness is a cost lever, and a failed run still bills you. The exact multipliers belong to Runta’s setup. Test on your own tasks before changing tools.

Runta's paragraph stating that 64.5% of list-price cost went to trials that did not pass
Runta's paragraph stating that 64.5% of list-price cost went to trials that did not pass
Hacker News thread 'What is the best harness for each model?' with 418 points
Hacker News thread 'What is the best harness for each model?' with 418 points

Sources

  • Runta, “Introducing FrontierHarness Eval v1.1” (Shilin Zhu, with Baseten, 17 Nov 2026). Primary source.
  • Hacker News discussion (418 points, 133 comments, 19 Nov).
Felipe Machado

Felipe Machado

I'm truly excited about what AI has to offer—especially in how it can unlock our potential across every area of life. I dive deep into a variety of AI-related topics and share the most valuable insights and tools I've discovered that are helping me every day. That way, you don’t have to spend hours researching—I’ve done the hard work for you, so you can save time and focus on what matters most.

Related Posts

AI for Developers

MAI Code 2: 96% of Opus at 5% of the cost?

November 18, 2026
Getting Started with OpenAI’s API: A Practical Guide for Developers (2026 Edition)
AI for Developers

Getting Started with OpenAI’s API: A Practical Guide for Developers (2026 Edition)

August 9, 2026
AI for Developers

How to Use GitHub Copilot: From Beginner to Advanced (Complete 2026 Guide)

August 9, 2026

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

I agree to the Terms & Conditions and Privacy Policy.

Unlock Your Potential With AI | AI Mind Center

Unlock your potential with AI Mind Center! Explore AI tools, tutorials, and inspiration to grow, create, and geek out.

Follow Us

Subscribe to our newsletter!

Browse by Category

  • AI Agents & Automation
  • AI for Developers
  • AI in Game Development
  • AI News & Trends
  • AI Productivity Tools

Recent News

Same model, up to 5x the cost? What Runta’s test says

November 19, 2026

MAI Code 2: 96% of Opus at 5% of the cost?

November 18, 2026

Windows 11 may log which coding agents you use. Check yours

November 17, 2026
  • About
  • Advertise
  • Privacy & Policy
  • Contact

© 2026 | Website Made By aimindcenter.com.

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • AI News & Trends
  • AI Productivity Tools
  • AI in Game Development
  • AI for Developers
  • AI Agents & Automation

© 2026 | Website Made By aimindcenter.com.

This website uses cookies. By continuing to use this website you are giving consent to cookies being used. Visit our Privacy and Cookie Policy.