Benchmark

Measure how reliably AI agents perform software-engineering work.

The Lukta Software Engineering Reliability Benchmark (Lukta-SWE-R v0.1) is designed to measure two capabilities separately: repairing a software defect correctly, and strengthening the tests that guard against future ones.

Benchmark validation in progress

What this benchmark is

It is designed to evaluate a specific AI agent system on controlled software-engineering work, and to report what that system actually did.

It keeps the two capabilities apart. Repairing a known problem and protecting against future ones are related but different kinds of engineering, so a single combined number would hide which one an agent is good at.

The two capabilities it is built to measure

Verified Bug Repair

Find and Fix a Bug

Can the agent find the cause of a software defect, repair it correctly, and leave the surrounding behaviour intact?

The first track looks at whether an agent can do the whole job of fixing something: understand why the software is wrong, change the right thing, and not break what was already working. A change that makes the reported symptom disappear while quietly damaging something else is not a repair.

Not measured by

Producing a patch
Matching one particular solution
Turning a single visible test green

Test Strength Improvement

Strengthen the Tests

Can the agent strengthen a test suite so meaningful defects and regressions that the existing tests miss become more likely to be caught?

The second track looks at protection rather than repair. A suite can be large and still miss the failures that matter. What counts here is whether the tests an agent adds would actually catch a real regression — test strength, not test volume.

Not measured by

The number of tests added
How much test code was written
Coverage movement on its own

Related Project Environment: Software Test Improvement

Why two tracks instead of one number

Fixing a defect that already exists and hardening a suite against defects that do not exist yet call on different skills. Measuring them separately means a result can say which capability was demonstrated, instead of averaging them into a figure that describes neither.

What a Case is

A Case is a controlled software scenario used to evaluate one defined engineering capability. One underlying Case can carry work for either track, so the same software situation can be used to ask a repair question and a test-strength question without becoming two unrelated problems.

How evaluation is designed to work

This is the design of the benchmark, not a service you can run against today.

Step 1
A controlled Case
The software scenario is fixed and pinned, so two runs are asking the same question.
Step 2
An identified agent configuration
A result describes the system that was actually evaluated — not a broad model or vendor name.
Step 3
The agent does the work
The agent performs the engineering task inside the boundary the Case defines.
Step 4
Independent evaluation
The work is assessed separately, using trusted checks the agent does not control.
Step 5
An evidence-backed result
What was run, on which Case, by which configuration, and what came out of it are recorded together.

Reliability is about repetition, not a single good run

One success does not establish reliability. The benchmark is designed to support repeated controlled evaluation so consistency can be measured rather than assumed from a single result.

It also separates two things that look alike from the outside: an agent failing the software task, and the platform underneath it failing. If the evaluation platform breaks, that is not evidence about the agent.

What organisations are meant to get from it

These are the questions the benchmark is built to answer once it is operating.

Compare different agent systems on the same controlled work
Compare two versions or configurations of the same system
See which of the two capabilities a system is actually strong at
Notice when a new release improves or regresses

Benchmark validation in progress

Where this is today

The benchmark architecture, its evaluation model and its evidence rules are built. What is not open yet is running agents against it: there is no public execution, no published result, and no ranking. We are validating the methodology before any of that becomes meaningful.

Engineering implementation is complete. Operational benchmark execution is not yet available, and no agent results have been produced.

If you are building an agent system and want to be part of early benchmark validation, tell us about it.

Request early access →Explore use cases

How this relates to Project Environments

Project Environments are where AI agents work on a company's own software. This benchmark is the separate question of how reliably agents perform software-engineering work in the first place. One is the work; the other is the measurement.

See what agents can do with a software project View the catalog entry