In my ever-growing quest to tighten the ship around security, I added an explicit Deny on kms:* to an IAM permissions boundary because the roles behind it manage no key material on paper. Lock the door, defense in depth, easy win, ten-line diff, or so I thought; these innocent choices always find a way to haunt you. After the terraform apply, every function behind that boundary went dark, and dark is the right word for it rather than broken: zero log streams, zero handler output, invocation errors climbing steadily in the metrics, and events vanishing and then re-arriving minutes later as duplicates.
I spent the first hour on event delivery, working through bus rules, resource policies, and DLQs, all of which were fine, because none of them were ever the problem. What finally registered was the error metric sitting nonzero while the log group held NO streams at all. Not sparse logs. None. My code logs something even while it is dying, so my code had never run.
What was actually happening
Lambda encrypts environment variables at rest with a KMS key, and it decrypts them at cold start using your execution role’s credentials. That decrypt happens before the first line of your handler. Deny kms:* on the role and the runtime cannot decrypt its own env vars. The function fails initialization with a KMSAccessDeniedException you will never see in a log group, because logging starts after the point where it died.
Then the async retry machinery re-drives the event twice across a few minutes, which is where my ghost duplicates were coming from all along and the haunting got real. The whole arrangement looks exactly like a delivery problem, and it was an IAM problem the entire time, wearing a delivery costume and enjoying itself.
Why would a Deny I wrote for key management break a decryption I never wrote? Because we all know (and I should have known) kms:* does not care which of us wrote the call. The runtime makes KMS calls on the role’s behalf, and my application code was never the complete inventory of what that role does. The inventory was the bug, the policy just carried it out.
The fix
The fix was to deny the management verbs and leave the data verbs alone.
- Deny:
CreateKey,ScheduleKeyDeletion,PutKeyPolicy,CreateGrant,DisableKey,DisableKeyRotation, the alias operations - Do not touch:
Decrypt,GenerateDataKey, anything the runtime itself depends on
The guardrail survives all of that intact. A compromised role still cannot create a key, delete a key, re-policy a key, or quietly hand itself a grant. It can decrypt env vars, which it was doing every cold start of its life anyway, with or without my permission slip.
That corrected list now lives as a comment block in every boundary document I own, because this failure walks through every static check you own. The policy validates, the plan is clean, the terraform apply succeeds, every check I own reports a good day, and nothing anywhere complains until the next COLD START, which on a warm fleet can arrive hours later, long after you have stopped suspecting IAM of anything. The ghost in the machine was excised, no need for the G-Busters…
What I keep from this
As a human it is hard to keep all context in scope of a change we make to a complex system. The only real way to account for that natural weakness is to audit every boundary Deny against the calls the runtime makes on its own, not only the ones your application makes on purpose. Lambda decrypts env vars. CloudTrail is your friend. Other services have their own invisible uses for the role you handed them. Ask what the platform does with a role before your code ever gets control of it.
And “no logs at all” turned out to be a diagnosis rather than a dead end. Errors with logs are mine. Errors with no logs mean the platform never reached my code, and from there the suspect list runs three deep: runtime permissions, VPC networking, broken artifact. The metrics handed me the answer inside the first five minutes and I spent an hour refusing to take it. That hour is the reason this post exists.