I set out to build a SaaS solution, what I didn’t realize in the beginning was that the customer facing SaaS was just the tip of the infrastructure iceberg I would find myself navigating. Looking back it seemed like every week a new microservice was needed to get the company up and running, it got to the point where, if I didn’t start using AI to accelerate the design, we would never launch. So I once again found myself in need of a new platform service, this time though, it was built on the same night I was executing a production migration. All my hands were full, so the service got built by a coding agent while I ran the migration, and I reviewed its output the way I review a human PR, which is to say I assumed competence, verified anyway, granted nothing on the word of a thing that had never been exercised, and read the report before I read the code. The service is real and it is in production, it works and is integrated into the stack: EventBridge ingest, snapshot pipeline, IAM boundaries, tests, CI.
“AI built a service” is the headline everybody wants, and it is not where the work originally went. Before I let the agent type a character, three artifacts had to be sitting on the table, and assembling them took longer than the build did.
The table stakes first
A ratified specification. Numbered sections (which AI loves…), explicitly listed open decisions (So I remember in weeks to come), a hand-off note addressed to a build model, and one standing order about the open decisions: flag each one with a recommendation, never silently decide. It ended up flagging five. Two of the five carried engineering I had not asked for and kept anyway, it had noticed that the parameter store I specified has no compare-and-set, moved the version authority into a conditional database update, and mirrored the result back out to the parameter. And by far the most important thing it did (because it was told to) was document why, the whole process through.
I have a list when I use AI, things it is never allowed to do, things I make concessions for… A hard constraint list. Zero AWS calls. Zero terraform against real backends. No invented account ids, no invented profile names, no quiet guesses at either, unknowns get marked pending and surface in the report. Exact security-tier rules. And a required report format covering every file, every judgment call, every local validation output. I am sure the models are livid with my micromanaging management style, but trust is an earned trait and when dealing with probability based generated output, almost never granted…
Review catch, and justified misses
There was one real defect. A crash-window hole. If the process died between advancing the version and writing the mirrored copy, the retry path came back, saw content that had not changed, did nothing about it, and left the mirror stale forever. The happy path was correct. The path where the process dies between two writes said nothing at all, which matches everything I have seen from these agents so far, strongest on structure and weakest on the half-second after a crash. We patched it, tested it, and moved on.
Then the live apply found two more that I could not have caught, because they only exist at the API boundary. The Terraform provider’s organization data source turns out to make a call that member accounts cannot perform by default, so configuration that validated offline died at plan time against the real cloud. And AWS rejects a retry policy on cross-account event bus targets, documented nowhere I looked, discovered by ValidationException at 1am.
The division of labor
Every architecture decision, every merge, every apply, every credentialed command stayed with me, and what came back from the agent was a branch and a report. Two of the three defects only surfaced when the real cloud said no, so whoever holds the apply has to be someone who reads a plan and stops mid-scroll, and I would normally call that caution theater if the theater had not been where the bugs were.
Cost accounting, since I try to keep myself honest: This time we got roughly two thousand lines of Terraform, Python, and tests in under forty minutes of wall clock, produced while I did other work, against about thirty minutes of my review and two real fixes. The same week we were not so lucky, an agent stalled mid-build on a feature job and had to be resumed with its context carried across intact, so the reliability is not clean either yet.
The open question I have not settled is which defect classes mean “tighten the constraints” and which mean “review is the permanent control.” The crash-window class feels permanent, all things considered, I figure I will grant it grace, assuming a model that is strong on structure is going to be weak on the process timing edge cases. The API-boundary class belongs to neither of those, because some things you just have to learn through the experience of failure with a giant “DENIED” in your cloud response.