Buyer’s Guide

Best AI Coding Assistants

Written by Govind Kumar Lohar. Reviewed for technical accuracy by Deepak Gupta and Bhaskar Suthar on · Review panel

  • ai
  • developer-tools
  • productivity

Independent buyer’s guide. No vendor paid to be included, ranked or described a particular way. Written for engineers, architects and the people who sign off on their tooling budget. Editorial policy.

Almost every comparison of AI coding tools is built on evidence that cannot support the conclusion. Demo videos are a task the presenter already knows the tool can do. Leaderboards score models on public repositories that the models were trained on. Blog posts reporting a big speedup measured one person, on one codebase, for one week, with no control.

None of that tells you what the tool will do on Tuesday, in your repository, on the ticket you actually have. Your codebase has the properties that matter: an internal framework with no public documentation, a test suite that takes eleven minutes, a module nobody has touched since the person who wrote it left, and a set of conventions that exist only in reviewers’ heads.

The only honest way to compare these tools is to fix a small suite of tasks drawn from your own repository, run every candidate through the same suite, and score them the same way. Five tasks is enough to separate the field, and it takes an afternoon per tool. That is cheaper than one bad rollout and far cheaper than a year of a seat licence that nobody uses after month two.

This article defines that suite. The rest of this cluster reuses it, so when a spoke says “run task three” it means the same thing every time.

Key takeaways

  • Pick five tasks from your own repository’s git history, ones already solved, so the correct answer exists and you can grade against it.
  • Public benchmark scores are contaminated by construction: the tasks come from public repositories, and public repositories are training data.
  • The metric that predicts real value is diff survival, meaning how much of the generated change is still there after review, not whether the tests went green.
  • Evaluate context gathering, not output quality. A tool that reads the wrong three files will write confident, plausible, wrong code no matter how good the model is.
  • Rerun the suite every quarter. This category changes faster than your evaluation memory does.

The five-task suite

Choose the tasks from commits that already landed. You want work whose correct outcome is known, because that is the only way to grade without arguing. Pull five changes from the last six months, ideally ones you or a teammate reviewed, and reconstruct the starting state by checking out the parent commit.

Give every tool the same prompt: the ticket text as originally written, nothing more. Do not coach. Coaching is the variable you are trying to measure, because in production nobody will coach it either.

Task 1: a bug fix with a known root cause

Pick a bug where the symptom and the cause were in different files. A null check missing in a shared helper, a timezone assumption in a formatter, an off-by-one in a pagination cursor. Something where the naive fix is to patch the caller and the correct fix is one guard in the shared function.

This task measures whether the tool traces the problem or pattern-matches the stack trace. A tool that greps for the error string, finds the throwing line, and wraps it in a try block has failed even if the reproduction now passes. Record whether it found the shared function, whether it checked the other callers of that function, and whether it added a test that fails before the fix.

The discriminating failure is a fix that is locally correct and globally wrong: the ticket’s path works, the three sibling call sites are still broken, and nothing in the diff says so.

Task 2: a refactor that touches many files

Pick a rename or a signature change that crossed at least eight files and included at least one call site that a simple find and replace would have broken. Extracting an interface, changing a function from returning a value to returning a result type, moving a module and updating every import.

This is the task where context window and indexing strategy show up as behaviour. Tools that work from embeddings of similar-looking code tend to find most call sites and miss the ones whose surrounding text does not look like the others. Tools that use the language server or a syntactic index tend to find all of them or fail loudly.

Record the number of call sites missed, the number of call sites changed that should not have been, and whether the tool told you it was unsure. Silence about an incomplete refactor is worse than a partial refactor that says which parts it skipped.

Task 3: a new feature with an unfamiliar dependency

Pick a feature that required reading documentation the tool cannot have memorised well: an internal service client, a niche library, a vendor SDK your team wraps. Adding a webhook endpoint that verifies a signature. Adding a new field to an existing API response and propagating it to two consumers.

This measures whether the tool discovers conventions or invents them. Your codebase has a way of doing error responses, a way of registering routes, a way of naming migration files. A good result follows the existing pattern without being told about it, because it read a neighbouring file first. A bad result is idiomatic code from a framework tutorial, dropped into a codebase that does not work that way.

Record how many existing files it opened before writing, and how many of your conventions the diff violates.

Task 4: a test suite for an untested module

Pick a module with real branching logic, no existing tests, and behaviour you understand. A pricing calculator, a permission check, a parser for a format you control.

Every tool will produce tests that pass. That proves nothing, because tests generated from the implementation assert whatever the implementation does, including its bugs. The grading question is whether the tests would catch a change that breaks the behaviour, which you check by deliberately breaking the module and rerunning them. The mechanism behind that check is the subject of the test generation guide.

Record how many of your injected faults the generated suite catches, and how much of the suite is asserting on implementation detail that will break on the next harmless refactor.

Task 5: a migration across a version boundary

Pick a real upgrade you have already done or will have to do: a framework major version, a test runner swap, a language version bump that changed semantics, an ORM migration. Constrain it to one service or one package so it finishes in an afternoon.

This is the task that separates tools that pattern-match from tools that can hold a plan. Migrations have a shape: an inventory step, a mechanical step that a codemod could do, and a residue of cases that need a human decision. The useful behaviour is a tool that does the mechanical part completely and names the residue. The useless behaviour is a tool that does eighty percent of the mechanical part, silently skips the awkward files, and reports success. Legacy work has its own guide in AI tools for legacy code migration.

Record what fraction of the mechanical change landed, whether the tool produced an explicit list of cases it could not handle, and how long the residue took you by hand.

Scoring: diff survival, not green tests

The output of the suite is not a score out of ten. It is a table you fill in the same way for each tool.

For each task record: the number of turns before you accepted or gave up, whether the project still builds, whether the existing test suite still passes, the number of files read before the first edit, and the fraction of the produced diff that survived your review unchanged.

That last number is the one that predicts whether anyone still uses the tool in three months. Call it diff survival. A tool that produces a 400 line diff of which 90 lines survive has cost you more review time than writing 90 lines yourself. A tool that produces a 90 line diff of which 85 survive has actually done work. Green tests are not evidence of either, because the tool wrote the tests.

Two things not to record, because they will mislead you. Do not record wall clock time as a headline number: it is dominated by how long your test suite takes and how fast you read. Do not record lines of code produced, because on this axis less is better and you will accidentally reward the wrong behaviour.

Needs first-hand data: Run all five tasks on your two finalists, three trials each to see variance, and record turns to acceptance, diff survival percentage, files read before first edit, and injected faults caught on task four. Publish the table internally with the task descriptions attached, so the next evaluation is a rerun rather than a rebuild.

SWE-bench: a benchmark, not a product

SWE-bench and its variants are the scores you will see quoted in every launch post. It is a dataset of real issues and their accepted fixes taken from public repositories, with the repository’s own tests as the grader. Understanding what it is prevents you from reading a percentage as a promise.

What it gives you

A reproducible, shared, non-trivial task format that is far better than the toy function-completion benchmarks that came before it. Tasks are real, multi-file, and graded by tests the maintainers wrote rather than by string matching against a reference solution.

It is genuinely useful as a relative signal over time for the people building these systems, and it measures the whole harness rather than the model alone, which is closer to what you buy.

What it does not do

It does not tell you what will happen in your repository, for four reasons.

Contamination is structural, not accidental. The tasks come from popular public repositories, and popular public repositories are in the pretraining corpus of every frontier model, along with the issue threads, the pull requests, and often the accepted patch itself. There are decontaminated and held-out variants precisely because this is a known problem, and you cannot verify from outside which mitigation any given number used.

The harness is tuned. A reported score is a model plus a scaffold plus a retrieval strategy plus a retry budget, and vendors tune all four against the benchmark. That is legitimate engineering and it is also exactly what makes the number non-transferable.

The task distribution is not your distribution. These are issues in mature open source libraries with strong test suites, small reproducible bugs, and documented conventions. If your repository had that, your bugs would be easier already.

Tests passing is the grading criterion, which inherits every limitation of tests as a proxy. A patch that passes the tests and breaks an undocumented behaviour scores the same as a correct one.

Treat a benchmark score the way you treat a vendor’s own load test: evidence the thing works at all, not evidence it will work for you.

Where the rest of this cluster goes

The suite splits the market along the line that matters, which is how much of the loop the tool runs without you.

Tools that write code while you drive. Completion and chat inside your editor, where you approve every keystroke. Start with GitHub Copilot alternatives for the field, and Cursor vs Windsurf vs Claude Code for the three-way comparison most teams actually run.

Tools that take a task and come back with a diff. Agents that plan, read, edit and iterate. Covered in AI coding agents, where context gathering is the mechanism that decides everything else.

Tools that work on code you did not write. Migration and modernisation are in AI tools for legacy code; understanding an unfamiliar repository is in code search tools; getting a new engineer productive is in developer onboarding tools.

Tools that check work rather than produce it. AI code review tools for pull request review, test generation tools for coverage that means something, and code quality and technical debt tools for the deterministic layer underneath both.

Tools that write the words. Documentation generation tools for reference material that stays in sync, and knowledge base tools for engineering teams for the decisions that never fit in a docstring.

The environment the tools run in. Local development environment tools and cloud development environments decide whether an agent can even build your project, terminal and shell tools decide how pleasant the terminal-native workflow is, and MCP servers decide what the tools can reach.

The platform layer. Internal developer platforms for paving the path these tools walk down.

Measuring whether any of it worked. DORA and engineering metrics tools, engineering analytics platforms, and async communication tools for the coordination cost that AI tooling does not reduce and sometimes increases.

How to choose

Run the suite, then let the results answer one question at a time.

If tasks one and three went well and task two went badly, you want an editor-native assistant, not an agent. The tool is good at local work and bad at holding a whole-repository plan. Buy seats for the editor tool and keep humans driving multi-file change.

If task two went well but diff survival was low across the board, the bottleneck is review, not generation. Spend the budget on review tooling before more generation seats.

If task five produced a plausible but incomplete migration and said nothing about it, disqualify the tool for migration work regardless of how it scored elsewhere. Silence about incompleteness is the single most expensive failure mode in this category, because it converts a known task into an unknown one.

If every tool scored about the same, your tasks are too easy. Pick harder commits. The suite is only useful if it separates the candidates.

Suite taskWhat it actually measuresRead next
1. Bug fixRoot cause tracing versus stack trace matchingAI coding agents
2. RefactorIndexing strategy and call site completenessCode search tools
3. New featureConvention discovery from neighbouring codeCopilot alternatives
4. Test suiteMeaningful coverage versus generated coverageTest generation tools
5. MigrationPlanning, and honesty about the residueAI tools for legacy code

Frequently asked questions

How often should I rerun this suite?

Quarterly, and after any vendor announcement that changes the underlying model or the agent harness. The tasks stay fixed, which is the entire point. A suite you rewrite each time measures nothing across time.

Should I use the same five tasks for every engineer on the team?

Yes for the tool decision, no for the rollout. The suite exists to compare tools under controlled conditions. Once a tool is chosen, what matters is whether individual engineers get value on their own work, which is a different question and is not settled by a table.

Does a higher benchmark score mean a better tool for my team?

Not reliably. Benchmark scores compare harnesses on public repositories with strong test suites. Your repository is not that. Use published scores to decide who is worth evaluating and the suite to decide who wins.

Is it worth evaluating open source agents if we can pay for commercial ones?

Yes, because they answer a different question. Open source agents show you the mechanism directly: you can read exactly what gets put in the prompt. That is useful even if you never deploy one, because it tells you what the commercial tools are probably doing.