While migrating my Terraform state backend to S3-native locking, I retired the dedicated state-encryption CMK as part of the cleanup, standard key hygiene, schedule the deletion with the 30-day window and move on. Within seconds every CI pipeline in the org that touched state was failing init with KMSInvalidStateException. The 30-day window I thought I had was for RECOVERING the key. The key itself stopped working the instant I scheduled it.
Those are the semantics almost nobody reads until they bite. ScheduleKeyDeletion puts the key into PendingDeletion IMMEDIATELY, and a key in that state refuses every cryptographic operation you ask of it. The waiting period exists so you can get the key material back, and it does nothing at all for the workloads still using it. Four live state objects were encrypted under that CMK when I scheduled it, so what I had actually done with one console click was turn their encryption key off, org-wide, while calling it cleanup.
Why there were still objects under the key
The migration plan was fine on paper, all things considered. New state writes were already landing under the AWS-managed aws/s3 key and the CMK was only supposed to be a historical footnote. What I had missed is that S3 encryption is per-OBJECT, set at write time. Flipping the bucket’s default encryption changes what NEW writes get, and every object already sitting in the bucket keeps whatever key encrypted it the last time somebody wrote it. Four state files across the org had not been rewritten since the flip, so they were still CMK-encrypted, and nothing shows you that unless you go ask each object directly:
aws s3api head-object --bucket <state-bucket> --key <path>/terraform.tfstate \
--query 'SSEKMSKeyId'I trusted the bucket default instead. A bucket default only ever governs the next write, and it will sit there looking authoritative about objects it has never touched.
The recovery
The fix is total and takes minutes once you understand what actually happened:
aws kms cancel-key-deletion --key-id <id>, which moves the key from PendingDeletion back to Disabled.aws kms enable-key --key-id <id>. State reads work again, CI unblocks. This pair is the whole emergency response, everything after is cleanup at leisure.- Re-encrypt each straggler in place with a self-copy onto the target key:
aws s3 cp s3://<bucket>/<key> s3://<bucket>/<key> \
--sse aws:kms --sse-kms-key-id alias/aws/s3- Verify with
head-objectthat every state object now reports the AWS-managed key, not the CMK. - Re-schedule the deletion, now against a key that encrypts nothing.
One catch on step 3. An S3 self-copy is the sanctioned way to rewrite an object’s encryption without downloading it, and it creates a new version every time. On a versioned state bucket that is exactly what you want, because the old CMK-encrypted versions stay recoverable right up until the key actually dies, and THAT is your real rollback window. I wrote the expiry date into the incident note, since “we can still recover old versions” turns false on the day the deletion completes and nothing anywhere will bring it up.
The checklist I’d hand anyone retiring a state CMK
- Enumerate every object under the key FIRST:
head-objectper state path, or S3 Inventory with the encryption field on big buckets. Bucket defaults tell you nothing about existing objects. - Re-encrypt via self-copy, verify per object, THEN schedule deletion. In that order, with the verification actually run and not assumed.
- Treat
ScheduleKeyDeletionasDisableKeyNowin your planning, because that’s what it is. If any consumer might still touch the key, you wantdisable-keyfirst as a deliberate canary step anyway, same blast radius, instantly reversible, and it tells you about stragglers without the adrenaline. - Write the version-recovery expiry into the incident note.
The root cause was not exotic. I read a recovery window as a grace period, I read a bucket’s default encryption as a statement about the objects inside it, and both misreadings feel completely fine right up to the second four repos’ CI goes red together. The whole incident cost me under an hour, which is the best thing I can say for shared state infrastructure and its blast radius. It broke loudly, it broke in one place, and one fix reached all of it.