In this long journey… at least it feels like it’s been long enough to comment on… I have questioned myself on a lot, what am I willing to concede on, what am I not, where can I make efficiencies, what has to be enforced, etc. Some things that I never thought I would question or change I even force changed in a fit of rage mid-development session. This is the story of one of those changes. It started out with me asking a very simple question to Claude, “How do we move forward from here?” When you are building and iterating with hyper focus on a single target, sometimes it is hard to see the larger picture, and you don’t build to the scope you actually require. In this case, I was so focused on getting the microservice in question working that I didn’t realize the way it was being built was incompatible with environment based deployments, I had essentially built it as a singleton, one deployment with the 1:1 assumption baked into its logic, expecting dev to talk to dev and prod to prod. Claude handed me a very distinct and detailed path for re-architecting everything and exerting a significant amount of work simply to get the other stages to be able to deploy, and the ripple effects were significant.
That was when an idea planted itself in my head… A control plane that serves multiple environments is ONE deployment, and the environments it serves simply become rows of data in that single deployment. This might seem like a super simple idea, but you have to understand I have been fighting singleton designs most of my career, because when used for an application design they breed absolute chaos, but when used for a control plane service, it makes a lot of sense. That thought invaded my mind like a weed and took root, spreading and growing at a significant pace, suddenly it wasn’t “How do we move forward from here?” it was “HOW DO WE APPLY THIS TO THE OTHER CONTROL PLANE SERVICES!?”. My enthusiasm cost me one very long night, as Claude helped me build the plan to convert four services under it in a single, highly motivated, extremely caffeinated night.
The per-env pattern is the default everybody reaches for, and I reached for it the same as everybody. My DNS service had a dev lane, my mail service had a dev bus and a dev table, and the plan was always “prod stack later.” Then I actually sat down and read what the prod copies would contain: the same Lambdas, the same rules, the same tables, the same handlers, differing in a name token and a handful of pointers. A mail sender does not hold a dev opinion, and a DNS provisioner writes to whatever zone the data hands it. So why was I planning to pay twice for things that behave identically? I had no answer.
The audit that settled it embarrassed me. My “dev” DNS lane was ALREADY serving production rows, because prod domains had accreted into its data over weeks along the path of least resistance, and the whole arrangement worked fine. The environment dimension had collapsed in practice weeks earlier, the infrastructure had just not admitted it yet.
The conversion
Three of the four services were mechanical changes:
- Names dropping the env token. Table keys keep it: rows are
ENV#devorENV#prodin one table, so a dev operation cannot touch a prod row without literally spelling prod. - Per-env pointers become per-row values in one committed, reviewable file. The response bus for dev, the parent domain for prod, all of it data now, with a PR diff standing in as the audit trail.
- Cross-account ARNs are derived in Terraform from account id plus naming convention. Hand-copied ARN strings caused two incidents before I banned them. A derived value cannot be typo’d into pointing at the wrong account.
The fourth service was the reason this took design work instead of an afternoon. Its handlers used ENVIRONMENT for actual logic, partition selection and event validation both. Ruminate on that long enough and what you have is a thing I started calling a masquerading env capable service, singleton-shaped on the outside and still holding every per-env assumption underneath. I was not going to ship that, not even as an interim.
So the environment gets resolved per EVENT. Cross-account EventBridge stamps every event with the sender’s account id and the sender cannot forge the stamp, so “which environment is this?” turns into “which account sent it, and where does that account sit in the org?” A small metadata service (ANOTHER SINGLETON BUILT JUST FOR THIS EFFORT!) publishes the account-to-OU topology as a shared parameter, the consumer maps OU to environment through a committed table, and accounts that deliberately host multiple environments get an explicit override row. Unknown account, unmapped OU, unreadable snapshot, mismatched claim: every one of those drops the event and emits a metric. Nothing guesses.
Move an account to a different OU and resolution updates itself on the next topology publish, so there is no account-to-environment map sitting anywhere waiting for me to forget to edit it. That was the whole argument for doing it this way, and an argument it was, just internally, I had to think long and hard around the security implications and logistics of this design. I originally designed a committed map, AI slapped the silliness out of me on that one… the objection was that hand-synced maps WILL drift, and drift here surfaces as silently dropped events at the worst hour you can imagine, so that was a resounding NO.
Details that saved the night
DynamoDB tables are pretty much my data paradise, my services live and die by them, except they don’t die… It makes it incredibly easy to dump and restore with item bytes UNCHANGED, and since every service is built on the single principle that zero PII is stored, it made the process incredibly easy. Rows were already env-scoped by key, so they moved verbatim. One table was keyed by fully-qualified domain names with no env segment at all, and I left its keys alone on purpose, the domain string already carries the environment and rewriting the keys would have broken logic that recomputes them from inputs. The ruling that had just cost me four services does not reach that table, and it was my job to notice where it stopped.
Route53 health checks are found by caller reference, and caller references are immutable. The new code keeps generating the exact strings the old lane created, so existing checks kept being found instead of duplicated. And the runtime-created DNS records were never Terraform-managed, so destroying the old stacks touched zero customer-facing records. I verified that before the terraform destroy instead of assuming it.
The result
It was a rough experience, juggling migrations, services, designs, breaks, incompatibilities, data schema shapes and decisions, what stays and goes, what changes and persists. In the moment it was mental overload and hell on earth banging my head against the wall at 4AM to try and get the final deploys working as expected, but the end was worth it, and it is so much more clean, efficient and maintainable so the effort was 100% worth it in my eyes. Will a customer see it or experience it? No, (And not just because I don’t have any due to an unlaunched product) this part of the company is a plane that the customer never sees, never understands, it runs invisible to them and is the glue that allows everything to flow smoothly behind the scenes. Just like a solid foundation of bricks, if the mortar is too thick, it actually weakens the structure, too thin and it can’t hold, you need the right size fit for the job, and for control plane services singletons were our right size fit.