Back to BlogAI Strategy
Max SaadMax Saad·9 min read·

What the best AI engineers are building right now, and what it means for your team

At AI Engineer Code Summit, the best teams showed how coding agents survive production: earned autonomy, layered verification and measured cost. What CEOs and CHROs should do next.

AI EngineeringCoding AgentsVerificationAutonomyCEOCHRO
Title card with the official white AI Engineer Code Summit wordmark, a boxed lockup of AI Engineer over Code Summit. Headline: Agents can write code. Trusting what they ship is the hard part, with the middle line in pale yellow and coral. Subline: The best teams earn autonomy, verify with evidence and measure in money. On the right, an autonomy dial runs from First level, where a human reviews the spec, PR and merge, to Top level, where low-risk work merges on its own. A marker on the arc is labelled The owner sets the level. Dark near-black field with yellow, coral and purple accents.

Bottom line: At AI Engineer Code Summit this week, the conversation moved past whether agents can write code. The strongest talks were about the expensive part that comes after: trusting what the agents ship, deciding how much freedom to give them, and proving the work pays for itself. Those are leadership decisions. Here's what the best teams are doing, and what I'd ask your organisation to do next.

I followed the summit from Kuala Lumpur, through the published mainstage transcripts and Latent Space's coverage. The organisers set three themes: scaling without slop, "System One" models, and more ambitious rewrites. Under all three ran one idea that should reach every CEO and CHRO: the teams getting the most from coding agents treat them as a production system with controls, and they engineer that system as seriously as the product itself.

1. Autonomy is earned, one checkpoint at a time

Luke Alvoeiro, who leads the core agent team at Factory, opened with the line I'd pin above every AI roadmap. The teams getting the most out of coding agents, he said, are the ones "ramping up autonomy gradually, one checkpoint at a time."

Factory learned this the hard way. Its earlier product asked users to plan everything up front and then let agents run for days. That suited big migrations and little else. Its new model is the "workstream": a standing goal with a team of agents, an identity, an environment and memory. A new workstream starts with the human's own access. When it outgrows one person, it moves to a service account the organisation controls.

The part leaders should copy is the autonomy dial. At the first level, a human reviews the spec, the pull request and the merge. At the top level, low-risk work merges on its own. The owner sets the level, and, in Alvoeiro's words, "the confidence to raise it comes from watching the work."

Diagram titled The autonomy dial, with the line Autonomy is earned, one checkpoint at a time, and the official AI Engineer Code Summit wordmark at the top left. A gauge runs from First level, A human reviews the spec, the pull request and the merge, to Top level, Low-risk work merges on its own. The hollow of the dial reads The owner sets the level. A caption below quotes The confidence to raise it comes from watching the work.

That is a management practice you already know. You give a new hire small, reviewable tasks, watch the results, and widen their remit as they prove themselves. Agents deserve the same ramp, with the same clarity about who owns the decision to raise it. It's the argument I made in the agent bottleneck is permission: the gate is the product.

2. Verification is the new review capacity

Andrew Finch of Sonar described how the bottleneck moved. Coding got automated, so the pressure landed on review. He quoted his CEO: "code is abundant. Confidence is not."

Diagram titled Four verification loops, with the lines Verification is the new review capacity and Code is abundant. Confidence is not, and the official AI Engineer Code Summit wordmark at the top left. Four cards describe Pull-request checks (AI review, static analysis, change matches the ticket, cross-repository impact, security scans, end-to-end tests, risk score), Post-deploy (watches production behaviour), Maintenance (rescans for drift and newly published vulnerabilities), and Knowledge (feeds lessons from past reviews and outages back into every agent). A footer says the merge decision should rest on evidence recorded on the change.

Sonar's answer is defence in depth through four loops. Pull-request checks widen far past tests and a human glance: AI review, static analysis, a check that the change does what the ticket asked, cross-repository impact, security scans, end-to-end tests and a risk score. A post-deploy loop watches production behaviour. A maintenance loop rescans the codebase for drift and newly published vulnerabilities. A knowledge loop feeds lessons from past reviews and outages back into every agent.

The merge decision, Finch said, should rest on evidence: the review findings, the quality gates, the risk level, all recorded on the change. Teams that do this well start auto-approving low-risk changes, which frees human reviewers for the risky ones. His summary of the human role: "the factory runs dim, not dark." People still audit risky code, handle incidents, review architecture and supply the why.

Factory runs the same pattern. Its risk judge reads its policy from the main branch, so a change can never rewrite the rules it's being judged by. That is a governance principle any board would recognise.

Melvin Johnson, who leads Gemini post-training at Google DeepMind, added the model-side version. Agents "sometimes fake helpfulness," he said, claiming tests passed when they plainly failed. His team trains models to report partial progress honestly instead of faking the last step. Your process should reward the same behaviour from day one: every agent summary checked against what the agent actually did.

For leaders: your review process is now a capacity plan. Fund verification infrastructure the way you fund headcount, because it decides how much agent work you can safely absorb.

3. Measure the factory, in money

Zach Lloyd, founder of Warp, defined a software factory precisely: "a code defined closed loop system for continuously measuring and improving cloud coding agents." He pointed to three problems with everyone running their own local agent setup: throughput is capped by human prompting, cost and productivity are impossible to manage, and security and governance get hard.

His prescription is measurement. Record every agent run. Track human interactions per merged change, throughput and cost alongside the usual delivery metrics. Run evals across live sessions, then benchmark models and configurations on your own work instead of public leaderboards. "If you don't have this data," he said, "you're kind of flying blind."

Cost efficiency showed up in concrete form across the event. Alvoeiro said Factory's move to having the model write one small program instead of calling tools one by one took a 150-ticket task from roughly a million tokens to about 60,000. Cloudflare showed a hybrid design where fast decision models handle tool selection and log extraction and a large reasoning model handles only the diagnosis; in its demo, Latent Space reported, that cut six model calls to one and about 40,000 tokens to 1,800. Diogo Almeida of TypeSafe AI used his keynote to push that category further, calling for these fast, structured decision models to sit "INSIDE application code," in swyx's paraphrase.

For leaders: ask your engineering head for one number: cost per merged, verified change. Spend per seat tells you very little. Spend per change that made it to production tells you whether the factory works.

4. The engineer's job is moving up a level

The talk CHROs should watch is Walden Yan's. The Cognition co-founder showed his team's merged pull requests up 40x in a year while engineering headcount grew 2.7x. Those are Cognition's own figures about its own team, and the organisational changes behind them are the useful part.

Cognition broke larger teams into pods of two or three engineers, each running swarms of agents and owning everything around the engineering: customer bug reports, QA, product management. Yan's warning to engineers who simply pull tickets and hand them to an agent: "you're not going to really be set up for success in the age of AI." His next step goes further. Engineers give agents jobs instead of tasks: a standing goal such as improving latency, with the agent setting up its own measurement, alerts and progress reports.

He was equally clear about the groundwork. Cognition rebuilt its environments so every service runs locally with mocked third-party dependencies, rebuilt hourly, so agents test changes without touching live systems. Agents send video evidence with every fix. That's infrastructure investment, and it came before the productivity.

Ishaan Dey from Google DeepMind's Antigravity team supplied the counterweight. When anything can be built instantly, he argued, "it's more important than ever to know what the right thing to even make is." His advice to engineers: "make room to think, and use your human judgment."

For CHROs: the engineering role is splitting into two skills that both matter. One is product judgment: deciding what's worth building and when it's good enough. The other is what Lloyd calls factory engineering: designing, measuring and improving the system the agents run in. Your job descriptions, levelling frameworks and promotion criteria probably reward neither yet. That's the gap to close in 2027.

What I'd do on Monday

  1. Write down your autonomy levels. For each kind of agent work, decide where a human reviews: spec, change, merge. Name who can raise the level and what evidence they need.
  2. Turn review into a funded system. Budget for layered verification, risk scoring and evidence attached to every change, so senior people spend their review time on the risky work.
  3. Pick one cost metric. Cost per merged, verified change, reported monthly next to delivery speed.
  4. Give agents an identity the company controls. Work that outlives one person should run under a governed service account with scoped access.
  5. Redesign the engineering role with HR in the room. Small pods, wider ownership, and career paths that reward judgment and factory engineering.

The theme I keep coming back to from this summit is that the winning teams are the most disciplined ones. They earn autonomy, verify with evidence and measure in money. Every one of those is a leadership choice, which means the advantage goes to companies whose executives make those choices early.

If you're working through what this looks like for your organisation, I'm happy to compare notes.

References

Enjoyed this? Let's work together.

I help companies turn AI strategy into shipped, revenue-generating products.