I have the feeling that Kiro is more efficient. Is it just me, or is it real?

Lately, at every event I go to, I end up arguing with colleagues, Ambassadors, or Community Builders about which assistant works best for us: Codex, Claude Code, or Kiro. Over time, I’ve had a clear feeling: Kiro is the most efficient.

I had no real data. No proof. Basically, my personal Kiro tenant lasted longer than my Codex or Claude Code tenants, and I could use a powerful model without hitting a wall within hours (which has happened to me). But I’ve been here before: what I think or what I feel can be biased, and without numbers, nobody buys it — not even me.

So I started wondering whether I could measure it objectively and verifiably. And here we are, with a reproducible benchmark where there’s no room to tip the scales:

• The same model in each tool
• The same exercises
• The same prompt
• The same weights

The short answer

In case six thousand words isn’t your thing, the feeling was right, but not for the reason I thought.

On cost per task there’s no single winner: it depends on the model and it isn’t a 100% comparable figure. Kiro comes out behind in direct comparisons, though in auto mode Kiro wins. Codex’s cost was a pleasant surprise — I genuinely had it down as my third option.

My feeling came from two other things: it finishes sooner (between 20% and 35% faster with the same model), and it doesn’t block you, because its limit is monthly, while Codex cuts you off every week and Claude Code every 5 hours.

On code quality, the three are practically tied — 4.34, 4.22 and 3.95 out of 5. I thought Claude Code would run away with it, but it couldn’t beat Kiro, which was a pleasant surprise.

If you want to know how this is measured and why reasoning effort changes everything, read on.

How each assistant differs

The first thing is understanding what each tool does, because although they’re very similar, they work in different ways.

Kiro

Kiro is the AWS tool. It originally only used Claude models but has been broadening its catalog lately — Claude, GPT-5.6, DeepSeek, Qwen, GLM — plus an automatic router (auto) that picks the model per task. It bills in credits with a per-model multiplier. The Pro plan is $20/month and includes 1000 credits.

One thing worth knowing: Kiro doesn’t expose consumed tokens directly. It gives you credits and time, but no input/output/cache breakdown.

Codex

Codex CLI is the OpenAI tool. With the ChatGPT Plus plan ($20/month), you get access to the GPT-5.6 models — Sol, Terra and Luna — with weekly window limits. It has no model router: you pick one, or if you don’t define a model, you get Sol at low effort by default.

Unlike Kiro, Codex does provide exact per-run token counts with a full breakdown: input, output, cache read, cache write, and reasoning tokens.

Claude Code

Claude Code is the Anthropic tool. With Claude Pro ($20/month), you get Haiku, Sonnet, Opus, and also Fable (in the US, you can access Mithos). It has something router-like — opusplan, which plans with Opus and executes with Sonnet — but it isn’t automatic per-task selection.

An interesting detail: Claude Code always uses two models at once. When you ask for Sonnet, it’s really Sonnet + Haiku in the background. Only --model haiku uses a single model. And its default reasoning effort is high, whereas Codex sits at medium or low depending on the model.

The playing field: 3 subscriptions at $20/month

One of the most important things about an assistant is the harness, something we don’t see even though it defines how the tool behaves. That’s why it matters to understand that, even though this comparison is as fair as possible, it’s very hard to level everything on identical terms.

What we ran and how we measured it

The exercises

We need tasks that satisfy three things: they must be public (so anyone can repeat them), they must have automated tests (so the result is binary, with no subjective judgment), and there must be a difficulty gradient.

The answer was Exercism: programming exercises with unit tests included, difficulty ranging from 1 to 10, and everything public on GitHub. 15 Python exercises are selected across three difficulty bands:

• 5 easy (difficulty 1-2): hamming, eliuds-eggs, list-ops, resistor-color-expert, transpose
• 5 medium (difficulty 4-5): complex-numbers, forth, knapsack, relative-distance, scale-generator
• 5 hard (difficulty 7-9): dominoes, paasio, pov, rest-api, sgf-parsing

The selection is deterministic and reproducible — the script is in the repo.

It’s true that we’re only comparing one language, but unfortunately I have neither infinite tokens nor infinite time. Still, it gives us a very interesting picture. On top of that, it’s my most common use case, along with IaC in Terraform for AWS — but there, Kiro might have had an edge because of its harness, which would have muddied this comparison.

The prompt

Identical for all three. Literally the exercise’s instructions.md, with a minimal preamble:

Solve the following Python exercise.
Write your solution in `<file>.py`.
Do not modify `<file>_test.py`.
The tests are run with `python3 -m unittest <test_module>`.

No hints, no guidance, no extra context. It’s the same as if you gave a test to someone you were thinking of hiring. This is the job interview for my coding assistants.

The isolation

Each tool ran naked: no personal configuration, no plugins, no MCP, no steering. As if it were my first day with each assistant.

Note: I don’t recommend running your assistants naked because they perform worse, but it is the fairest way to compare them.

What we measure

Integrity

Before every run, we verify that the tests fail with the empty stub (otherwise the exercise measures nothing). Afterward, we verify that the test file hasn’t been modified — an agent that edits the tests to make them pass doesn’t count as solving it.

The batches

In total, there are more than 390 runs (26 configurations × 15 runs per configuration) across default and matched-effort batches.

The code quality tournament

Normally, you’d expect every assistant to solve every exercise (small spoiler: that wasn’t the case), but they won’t all solve it the same way, so we have to measure code quality — which means finding the most objective way possible to do so.

And here there was a fairly Solomonic solution: have other models judge the solutions blindly.

Blind judge tournament (3 models that don’t compete in the benchmark):

• Each exercise is judged with the solutions anonymized (solucion-A, solucion-B…)
• 10 passes per judge with the order shuffled, to neutralize position bias
• 3 independent judges (glm-5, deepseek-3.2, minimax-m2.5) — impartial, since none is related to the models being judged
• It only counts if there’s a clear majority; if the judges disagree, that’s reported as such

The tournament has two parts: first, the same model is pitted across two tools (which tool writes better code with the same model?), and then a final between each tool’s most powerful model.

Note: the models were run by Kiro, since a fourth assistant for this felt excessive and, given this was a blind review, the harness matters much less here.

What each assistant can measure

Not all of them expose the same metrics, and that constrains the comparison:

All three give an estimated per-exercise cost in $, so you can compare how much it costs to solve the same task in each tool. None of them is the actual invoice — they’re approximations based on the rate card or the SDK — but all three use the same logic: the model’s per-token price.

What varies is granularity: Codex and Claude Code let you see how much went to cache or reasoning; Kiro gives you the total without opening the box.

How we compare

The comparison runs along three axes:

Same model, different harness. The main axis. If we put gpt-5.6-terra in Kiro and in Codex with the same prompt, any difference that shows up is purely tooling: how it builds context, how many calls it makes, how much it caches. It isn’t a model difference; it’s a harness difference.

Automatic router vs manual choice. Kiro has auto at 1.00x. Does it perform like picking a specific model by hand? And at what cost?

Matched effort. Each tool has a different default reasoning effort. To find out how much that variable weighs, extra batches are run with effort matched to high in Codex and Kiro.

The results

They all solve them… almost

The first surprise is that we have 4 failures:

All four failures are Claude Code’s, and none belongs to Opus or Sonnet:

• --model haiku fails three: forth, scale-generator and transpose
• --model opusplan fails one: relative-distance

Careful not to confuse opusplan with Opus: opusplan is Claude Code’s router configuration, which plans with Opus and executes with Sonnet. Opus on its own passed all 15, and so did Sonnet.

With the tests as the only criterion, then, the conclusion is boring: these tools solve Exercism exercises without breaking a sweat, with almost any model. The interesting part isn’t whether they get there; it’s what they spend and how long it takes.

The problem here is more with the model. Haiku 4.5 is an old and not very powerful model — it came out almost a year ago, and right now, a year is a long time for a model, especially considering Sonnet has had 2 versions since, Opus 3, and the GPT models are fairly new.

With the same model, Kiro is faster

Something interesting in this exercise is the time measurement. Wall clock, measured with a clock, with no room for interpretation.

Kiro is faster in 6 out of 7 models, with reductions of 20% to 35%. The only tie is Opus (+2%, within the noise). And the per-exercise detail confirms it: with Luna, Kiro wins all 15 of 15. It isn’t that the average is lower because of one extreme case — it wins exercise by exercise.

An extreme case: Haiku in Kiro takes 15.8 minutes — three times as long as Sonnet — because it retries the tests 20 times before getting it right. But it does get all 15 right. In Claude Code, Haiku doesn’t retry and fails 3 of them. The model is the same; the difference is that Kiro insists and Claude Code gives up. That insistence has a cost: Haiku at a 0.4x rate ends up spending 6.46 credits, compared to Sonnet’s 5.79 at a 1.3x rate. The cheap model isn’t always the most efficient, especially if it requires several retries.

This is what happens when we quote a project: on certain tasks, a more junior profile can end up costing more because it needs many more hours — just like putting a senior on very simple tasks.

What each one consumes

Before measuring, I want to be clear: measuring consumption is very complicated. All three tools cost $20/month, but they don’t measure consumption the same way, and their time limits aren’t equivalent:

That means the same percentage doesn’t mean the same thing across tools. Each one has a main limit (weekly or monthly), and on top of that, Claude Code has a 5-hour window limit, which is the one that blocks you first during intensive work.

Note: Codex had a similar 5-hour window, but OpenAI temporarily suspended it on 12 July 2026 when GPT-5.6 Sol was launched. As of this benchmark (August 2026), it’s still suspended for the Plus, Pro, and Business plans. If they bring it back, Codex would have that extra blocker too.

The following table compares the cost per model of running the exercises: weekly plan consumption (Kiro prorated to 1 week), the % of the limit that would be reached in a week (monthly for Kiro), and the % of the limit that would be reached in 5 hours.

Codex is cheaper per task with the GPT-5.6 models. With usage spread evenly across the month, Codex gives between 2.4x and 3.3x more capacity. But if you concentrate work into a few days, its weekly window is comparable to Kiro’s monthly one. In its favor, the 5-hour window is suspended, so today Codex’s only wall is the weekly one.

This result is surprising because the cost is much lower, and this is where we see that Kiro uses a different effort level — which we’ll look at in the next section.

With the Claude models, the price per task is similar — at first it may look like Claude Code wins, but we have to factor in the weekly and 5-hour limits. You hit the limit much sooner in Claude Code, and the first limit isn’t the weekly one; it’s the 5-hour one. With Opus we can launch this test twice and that’s the session gone: each batch eats 47%, so two of them add up to 94% and a third doesn’t fit.

This is an interesting point, because if you can wait and you’re patient, you’ll be able to carry on — but if something is critical, the normal reaction is to upgrade tiers, even if you don’t need it for the whole month.

Kiro’s router is the only clear win on cost: 0.37% of the month against opusplan’s 2.20% of the week. And it finishes sooner too.

Reasoning effort isn’t neutral

We saw in the comparison that something odd was going on with Kiro, and the GPT models were excessively expensive. Digging into it, we found that the default effort in Kiro is different:

• Codex: Sol runs at low, Terra and Luna at medium
• Claude Code: all at high
• Kiro: GPT-5.6 at high (documented), Claude at high

In other words, Kiro and Claude Code already run at high by default, while Codex sits lower. That asymmetry directly affects consumption — a model that reasons less spends fewer tokens per task.

To measure it, we have to run a new test with the Codex batches forced to high:

The cost gap between Codex and Kiro narrows with matched effort. With Sol at default (low), Codex consumes 1.34% against Kiro’s 4.42% (3.3x). With both at high, it becomes 2.07% against 4.42% (2.1x). Part of Codex’s advantage came from its Sol running at low by default.

But if we make the comparison against each one’s real limit — Codex’s week against Kiro’s month — the figure gets worse for Codex: it consumes 2.07% of its week while Kiro takes 1.10% of its month.

Sol is the most striking case: going from low to high costs Codex 55% more consumption and 59% more time.

Per-exercise detail

For anyone who wants the full breakdown:

What you see when comparing Cx-Sol (1.34%) with Cx-So-H (2.07%): the same model spends 55% more when you raise the effort. And comparing Cx-So-H (2.07%) with K-Sol (4.42%): even matching at high, Kiro is still 2.1x more expensive with Sol. The difference is no longer effort — it’s the plan’s cost structure.

A note on GPT-5.6 pricing on Bedrock: in late July 2026, prices dropped significantly — Luna by 80%, Terra by 20% and Sol by 20%. If Kiro passes that reduction through to its credit multipliers — which as of this benchmark (August 2026) it hasn’t — the consumption gap with Codex on the GPT models would narrow.

Code quality

An exercise passing the tests says nothing about whether the code is good. To measure that, a blind tournament: 3 independent judges (glm-5, deepseek-3.2, minimax-m2.5) that don’t take part in the benchmark score the anonymized solutions over 10 shuffled passes. It only counts if there’s a clear 2-of-3 majority.

Head-to-head duels: same model, different tool

Each duel pits the solutions for the same exercise, with the same model, produced by two different tools. The question is: who writes better code with the same model?

With default effort:

With effort matched to high:

In each tool’s default mode, Kiro wins on the powerful models (Sol 11-4, Opus 10-5) and ties or loses slightly on the less powerful ones (Luna 6-8, Haiku 5-6). The pattern is clear: the more capable the model, the more Kiro gets out of it.

But remember that the effort in Codex was different. Kiro runs at high by default, and Codex doesn’t. Matching at high:

• Luna flips: Kiro goes from losing 6-8 to winning 9-4. Kiro’s harness gets more out of Luna when both are thinking harder.
• Terra ties: 7-7. No measurable difference.
• Sol flips: Codex goes from losing 4-11 to winning 8-6. When Sol thinks hard in Codex, its harness produces better code — though the outcome is similar.

There’s no clear universal winner in terms of quality. Each tool gets more out of a different model profile. And on top of that, the results are similar.

Totals

A fair comparison by head-to-head matchup (same model, different tool):

Kiro wins all three matchups, but none are landslides. With matched effort (the high row) the difference narrows — Codex closes to 22-19.

The final: the best of each tool

The last stage pits each tool’s most powerful model directly against the others: Kiro with Opus 5, Codex with Sol at high effort, Claude Code with Opus 5, and Kiro Auto (the router). Four solutions per exercise, same prompt, same tests — pure harness difference.

With consensus (5/15): Kiro Opus 3, CC Opus 2, Codex Sol 0, Kiro Auto 0.

Which means Kiro Opus wins narrowly, not overwhelmingly — and if we look at the average scores per exercise, the picture is similar:

The three powerful models are close — 0.39 points separate first from third on a scale of 5. The difference between Kiro and Claude Code using the same model (Opus) is 0.12, which is negligible. Codex Sol sits a step below but still produces 4-out-of-5 quality code.

Kiro’s auto router (3.25) confirms what you’d expect: a router that picks cheap models for tasks it considers simple can’t compete on absolute quality with a top model thinking hard. But it solves 15 out of 15 in 5.8 minutes, spending 3.72 credits — the quality/price ratio is a different story.

Per-exercise judge detail

Conclusions

It’s funny because the result surprised me. In general, what surprised me was Codex’s price-to-performance — I thought it would be further behind — and the fact that all three are so close in quality, but above all that Kiro came out even better: I expected Claude Code to be the most powerful.

Objectively, it’s hard to declare an absolute winner, but there are a few things that, in my view, hand the win to Kiro, and probably Codex the runner-up in this review (if Codex brings back the 5-hour window, my view would get worse).

Claude Code is very powerful, but the way I see it, the 5-hour limit is a problem: if you run a task and hit that limit, the normal reaction is to upgrade to a higher plan, so its real cost goes through the roof. The same used to happen with Codex, but by removing (we don’t know whether temporarily) its 5-hour window, the problem moves to the week — whereas in Kiro that problem is monthly, which makes it far more manageable.

I know this may be the most controversial part, because on a monthly plan without hitting the limits, Kiro comes out behind. But in the real world — with delivery dates, workload spikes and a fair amount of stress — hitting the limits of any plan is normal, and this model of 5-hour and weekly windows takes advantage of exactly that: once you’ve started, you probably have no option but to move up a tier.

On the other hand, Kiro has an advantage that Codex and Claude Code don’t: it has the other one’s models. It’s true that Fable from Claude Code is missing, for instance, but unless your requirement is specifically Fable that isn’t a problem.

There’s one thing Kiro can’t compete on: it doesn’t let you create images or slide decks directly. In my case, that’s solvable with other tools — in fact, I prefer that to the cookie-cutter decks I’ve been seeing lately.

So yes, Kiro is more efficient across the full month, and even more so if we use auto mode.

That said, this isn’t set in stone — everyone has their preferences. Mine is clear, and it’s Kiro. What’s yours?

To reproduce it

All the code is in the repository: coding-agents-benchmark.

You don’t have to take any number on faith. The scripts are tool-agnostic — they take the command as a template — so you can plug in whatever assistant you like, or repeat it with other models and another language.

Prepare the exercises:

1
2

./stage-exercises.py --lang python
./check-baseline.sh          # verifies the tests fail with the empty stub

Run a batch (15 exercises with verification included):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Kiro, with an isolated HOME, so it inherits no personal configuration
KA='./kiro-aislado.sh --no-interactive --agent benchmark'
./tanda.py --agente kiro-sonnet5 --cmd "$KA --model claude-sonnet-5 {prompt}"

# Codex
./tanda.py --agente codex-terra \
    --cmd 'codex exec --json -s workspace-write -m gpt-5.6-terra {prompt}'

# Claude Code
CC='claude -p {prompt} --output-format json --permission-mode acceptEdits --safe-mode'
./tanda.py --agente cc-sonnet --cmd "$CC --model sonnet"

tanda.py is resumable: completed runs are skipped, so you can stop and continue once your plan recovers. And verificar.sh checks the test file’s checksum after every run — if the agent touched it, the run is flagged as tampered and doesn’t count.

The quality tournament:

1
2
3

./torneo.py --parejas          # list the duels possible with the data on disk
./torneo.py --fase1            # duels of the same model across two tools
./juez.py --todos --panel      # 3-judge panel over a prepared directory

What you should declare if you repeat it: the plan percentages come from reading each tool’s /status, which rounds to whole numbers — that leaves an error band of between ±7% and ±20% depending on the case. Kiro’s credits, on the other hand, are exact: the sum of the 150 logs matches the account’s report to the second decimal place.