REST API Gateway has behaviors that are documented, defensible, reasonable in the abstract, and still can seem like lies by the time you meet them mid-incident. I collected three of them over a few weeks. Each one cost real time, because in each one the symptom stood pointing confidently at a place the cause had never been.
The policy applied, never deployed
This catches me every time. No matter how long I work with API Gateway I will always inevitably run into this issue… A resource policy change on a REST API takes effect at the NEXT stage deployment, not at apply time. Terraform updates it, reports success, every describe call you run shows you the new policy, and the gateway goes right on enforcing the old one until something creates a fresh deployment. Probably one of the most frustrating issues to run into…
It took its pound of flesh, I paid three consecutive workflow failures before I realized it was happening. A cross-account call kept getting denied by a policy I had already fixed, and every inspection tool I own agreed with me that the fix was live, it was applied, it just had never been deployed. The runbook process modification those three failures encouraged was to check the stage’s deploymentId age after any policy-touching apply, and to force a create-deployment the moment it predates the change. Because who knows when the next person will run into this…
The same lie has a mirror image, and I have been hit by both sides now, like being squeezed between a rock and a hard place until I erupt in a development-induced frenzy. Months earlier, a resource policy that existed OUTSIDE Terraform state entirely, leftover from a manual experiment, sat there silently 403ing production requests while every piece of managed configuration looked immaculate. When config inspection and observed behavior disagree, one of the layers between them has a deployment boundary, and API Gateway turned out to have more of them than I had ever bothered to count.
The one-line diff apply, that couldn’t
Flipping a method response’s success status from 200 to 202 looks like the most trivial diff you can write in Terraform. It plans clean as an in-place update. The apply then dies against the deployed method, because that status code is create-time-only in the actual API, and in my case the failure sat inside a SHARED stack, so one immutable attribute took down the entire terraform apply process for everything standing behind it.
The escape is telling Terraform what AWS was going to do anyway, targeting the method response with -replace so that the plan models a destroy-and-create. When a trivial attribute flip fails mysteriously, go ask whether that attribute was ever mutable in the API at all, whatever the provider schema is telling you, because the provider cannot always warn you. Why does the API treat a status code as identity rather than configuration? I have a guess involving deployment snapshots and no evidence for it. It just does.
The 403 wearing a homepage suit
My CloudFront distribution fronts both the SPA and the API, and it remaps API 403 and 404 responses to the SPA index, which is the standard move and the only way deep links work in client-side routing. The side effect is that an edge-denied API request does not LOOK denied at all. It looks like the site took you gently by the elbow and walked you back to the front page.
During a production smoke test I spent a while investigating redirect behavior, when the actual event was an authorizer rejection wearing a homepage the CDN had lent it. The verdict from that evening has not moved since, through-the-CDN status codes are unusable for API smoke tests. My checks hit the execute endpoint directly now, where a 401 arrives undisguised and proves the auth layer is awake, and a 500 means real trouble. The CDN path gets tested for the one thing it actually owns, which is serving the shell.
All three lies cemented the same change in habit. I no longer (for the most part…) let a debugging session run long while configuration inspection and observed behavior disagree with each other, because that disagreement was never the noise I kept treating it as, it was the finding sitting in plain sight all three times. Name every layer sitting between what you configured and what you observed, then check each one for a cache, a deployment boundary, or a rewrite rule. API Gateway handed me one of each inside a month and I continue to dread what it will hand me in the future.