Skip to content

Day 51 of 100

The Cloud Permission Failure

Cloud SecurityIAMAdvanced20–25 min

πŸ”₯ Problem Statement

An application that's accessed a particular cloud resource reliably for months suddenly starts getting permission-denied errors on every attempt. Nothing in the application changed. Around the same time, a different team performed an unrelated IAM cleanup elsewhere in the organization β€” unrelated, as far as anyone initially assumed.

πŸ—οΈ Environment

  • Application: uses a cloud service identity to access a specific resource
  • Access: worked reliably for months
  • Recent activity: unrelated IAM cleanup performed by a different team
  • Symptom: application suddenly receives permission-denied errors

πŸ€” Your Challenge

How would you investigate an access failure like this through identity, policy, and audit evidence rather than guessing?

  • If the application's own code and credentials haven't changed, what else determines whether an access attempt succeeds?
  • Could a policy change made for an unrelated purpose still affect this application's access, even if nobody intended that?
  • What would cloud audit logs show you about exactly what changed and when?
  • How would you distinguish a role permission change from a policy inheritance change from a resource-level policy change?

Solution Hidden

Think through the problem yourself before looking at the answer.

πŸ’‘Solution

Step 1 β€” Understand the symptoms

Access to a cloud resource depends on the current state of IAM policy at the moment of the request β€” not on anything fixed or cached from months ago. If the application's code and credentials are genuinely unchanged, the explanation almost certainly lives in policy: a role's permissions changed, a policy this role inherited from was modified, or the target resource's own resource-level policy changed to no longer allow this identity. Any of these can happen without anyone touching the application at all.

Step 2 β€” Identify the likely bottleneck

The timing correlation with the "unrelated" IAM cleanup is worth taking seriously rather than dismissing β€” IAM cleanups often involve removing what looks like unused or overly broad permissions, and "unused" is sometimes a misjudgment based on incomplete visibility into what's actually depending on a given role or policy. A permission this application relied on, sitting in a shared or inherited policy, could easily have looked safe to remove without anyone realizing this specific application was quietly depending on it.

Step 3 β€” Investigation

  • Check cloud audit logs (e.g., CloudTrail, Cloud Audit Logs) for IAM changes around the time this started β€” most cloud providers log every policy modification with a timestamp and the identity that made it, which directly confirms or rules out a connection to the cleanup.
  • Compare the application's service identity's current effective permissions against what it had before β€” if historical policy versions are available, a direct diff is the fastest way to see exactly what was removed.
  • Check for policy inheritance β€” does this identity's access come from a directly attached policy, or from a group/role it inherits from that the cleanup might have touched without anyone connecting it to this specific application?
  • Check the target resource's own resource-level policy, separate from the identity's permissions β€” some cloud resources have their own access policies that must also allow the request, and a cleanup could have touched either side independently.

Once audit logs confirm the specific change, restore the necessary permission β€” ideally scoped precisely to what this application actually needs, rather than blanket-reverting the entire cleanup, which might have correctly removed other genuinely unused access. If the original permission was broader than necessary, this is a good opportunity to grant a more precisely scoped replacement rather than simply restoring the old, possibly-overbroad one.

Beyond the immediate fix, this points at a process gap: IAM cleanups need a way to verify a permission is genuinely unused before removal β€” checking actual usage via audit logs over a meaningful window, not just assuming based on how a policy looks β€” so a future cleanup doesn't silently break a dependency nobody remembered to check for.

Step 5 β€” Engineering lesson

Access failures are rarely explained by the application itself when its code and credentials haven't changed β€” they're explained by the current state of identity and policy, which can change from entirely unrelated activity elsewhere in the organization. Audit logs are the authoritative source for "what actually changed and when" in exactly this kind of investigation, turning "an unrelated cleanup happened around the same time" from a coincidence into a confirmed cause.

🧠 Today’s Takeaway

Access problems should be investigated through identity, policy and audit evidence.

πŸ”” Don’t Miss Tomorrow’s Challenge

A new real-world engineering challenge is released every day.
100 Days β†’ 100 Challenges β†’ 100 New Things Learned.

Get the useful stuff, not the noise.

Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

No spam. Unsubscribe anytime.