At Electric Twin, we are constantly looking for ways to improve our engineering processes, including adopting new practices that help us ship faster without losing product quality.
One key new practice we adopted? We built our own software factory, where agent teams do the work and verification systems keep them in check.
We put this development process to the test by rebuilding one of our products with it and deploying it to customers. To measure its impact, we compared it with our status quo process by putting our two core products through the same independent penetration test.
The result? A happy customer and a record pen-test result.
“We’ve found the update to be a major improvement and want to thank you again for listening to our feedback and working so hard to implement it quickly!”
Enterprise customer
Status quo: how our engineers work with agents
Prior to the software factory, we gave agents complete tasks: implement a change, test it against the real system, resolve review findings and produce a change that meets the required checks. The agent was expected to work through problems and finish the task.
In this style of work, engineers set the requirements, provide the context and make the decisions that need human judgement.
Engineers also manage priorities, investigate exceptions and decide whether the results meet the requirements.
The review bottleneck
However, in this environment, manual reviews quickly become the bottleneck.
Coding agents write code faster than people can inspect it. They also tend to write in a verbose context-rich way, leading to bloated code, or ‘AI slop’. Agents also edit systems that engineers are reluctant to see changed, leading to rewrites.
This means that reviews get rushed or that engineers’ time is consumed by reviewing code changes, with neither reliably catching the failures reviews are meant to prevent.
What we tried first: verification
The goal was to let agents complete work without an engineer having to read every change.
We started to experiment by letting agents make code changes in a contained environment, without human intervention. The question for the engineering team became: “What’s the minimum verification I’d need to ship this with confidence?”
Our hypothesis was that we could build verification systems robust enough for 80% of PRs to merge automatically, without any human approval.
The first major issue we hit was false passes: verification that reported success when it shouldn’t have.
At the time, we still had humans in the loop on most changes. So when our scheduled security scan (one of many automations) had run green for a week without reporting an issue, one of our engineers sensed that something was off. They ran a one-off audit of the code and confirmed real vulnerabilities.
We realised the scanning process was broken, even though every run still reported success. No check caught the problem; an engineer’s judgement did. So we fixed and tightened the criteria for a pass.
Each new check had to show:
executed to completion
reported its own coverage against the full required scope
confirmed the work aligns with the behaviour the task requires
So we designed a new verification system, and created a new principle to test the system itself.
Shifting engineering judgement to context, rules and environments
We focused on tackling three issues: context, rules and environments.
Context
An agent building in the backend repository may not have access to the infrastructure-as-code (IaC) repository.
When it builds a feature such as media upload, it is likely to make false assumptions about the infrastructure’s constraints. This can lead to an implementation that doesn’t perform in production
Many of the decisions people make while developing software are never documented, so a codebase can carry technical debt with no reasoning visible to an agent that is new to it
In agent-first development, the code has to carry this context explicitly; otherwise, agents are likely to make infeasible changes. The agent also needs read access to the IaC repository, so it can make decisions grounded in truth.
We use Linear as our system of record: intent is recorded in tickets, and agents record their decisions in comments on those tickets, so each change can be traced.
This does mean our code carries more comments, but we’re comfortable as it helps preserve knowledge: the next agent needs to understand why the code works the way it does to change it correctly.
Rules
Up until now, our rules have looked much like a traditional development process:
Every code change goes through an agent-run review system
A panel of reviewers assesses the work, raises issues and decides whether it can proceed
In the less common case that the reviewers cannot reach consensus, or flag a serious risk, the change is raised to an engineer: about one review in six
Now, escalations reach the engineer in plain language, and are a key moment of information sharing between the factory and its engineers, so engineers keep a finger on the pulse of the system.
We also designate areas of the code that require human review regardless of the panel’s verdict: our core pillars, which we call ‘arches’. These core pillars are chosen carefully, and in future we aim to build a verification system strong enough to remove this constraint.
Overall, these workflows are designed to make the best use of each model’s capabilities, and they have improved as the models have improved.
Environments
The final set of issues is solved by giving agents a suitable test environment. Even with context on the database schema, and a review system that draws extensive reasoning out of frontier models, an agent can still write a SQL query that looks reasonable yet performs poorly in production.
So we give agents:
an isolated development environment with a database seeded with production-like data and load, where they can test their assumptions against real, mostly deterministic execution
permission to spin up development instances running the whole application, so they can run end-to-end (E2E) tests with tools such as Playwright, monitor performance and adjust their implementation within their own loop
With close-to-real feedback, an agent that reports “done” has usually reached a well-performing implementation that meets the intent. Engineering judgement still sets the limits: no agent touches the production database without a human in the loop.
Fixing all three led us to a self-healing system
We designed a system where agents record all their work, including the decisions they make, through Linear tickets.
What this did:
Bugs get fixed without waiting for an engineer. Logs, traces and observability tooling build a near-complete picture of software execution. When an exception occurs, an alert sends our Application Support Engineer (ASE) agent teams to debug and resolve it
Issues surface before users find them. Agent swarms join user testing and dogfooding, testing our product in staging and catching edge cases and bugs before they waste anyone’s time or damage our reputation
Agents find the rough edges our tests missed. We partnered with Antidote (formerly Duku) to send exploration agents through our applications in a real browser, trying different customer journeys. Our engineers enjoyed watching the alerts channel buzz whenever Antidote’s agents swarmed staging
All this created a feedback loop, so the system now corrects itself.
A five-whys analysis of the audit trail in Linear then finds each issue’s root cause, which has always been context, rules, environments or human error.
What the factory changed
Four results stand out:
1 vulnerability against 7. The independent pen test found one vulnerability in the factory-built product, and seven in the product built with human review of every change
564 PRs in 13 weeks. With one dedicated engineer, the factory merged a median of 27 PRs a week, peaking at 145, excluding Dependabot updates
83% of reviews cleared without a person. In the 60 days to 22 September, the agent review panel cleared 83% of reviews and referred 17% to an engineer
Auto-merge is at 50% and rising. Our hypothesis was that 80% of PRs would merge with no human approval. Since we turned on auto-merge on 10 September, 39% of backend merges have been automatic, rising to 50% in the week of 5 October
One engineer and a team of agents now run a revenue-generating product, with a seventh of the vulnerabilities of our human-reviewed one.
What this means for engineering at Electric Twin
At Electric Twin, we believe engineering judgement remains critical.
After all, it was an engineer who questioned our green security scan, leading to a stronger check for the work that followed. As the industry changes, we want more of our effort to go into designing and scaling systems.
This shapes what we look for when hiring: technical depth, systems thinking and the ability to question what the evidence actually proves.
Want to come and design a system? We’re hiring engineers who want to design systems like this one.
References
Compound Engineering, Kieran Klaassen, Every
Building Effective Agents, Erik Schluntz and Barry Zhang, Anthropic, December 2024
Product Verification Infrastructure, Antidote
Issue Tracking, Linear
End-to-End Testing, Playwright, Microsoft
Five Whys Root-Cause Analysis, Wikipedia
Want to try the skills behind our factory?
We are sharing some of the Claude skills we use to build and check agent work
