Day 25 of 100
The Wrong Environment Variable
🔥 Problem Statement
A production service starts behaving strangely — not crashing, not erroring loudly, just quietly connecting to the wrong place. Requests that should hit the production database, or a production downstream service, are landing somewhere else entirely — a staging endpoint, it turns out, once someone traces it far enough.
The application code hasn't changed. The last deployed version was verified working correctly against the right endpoints. Nothing in the codebase explains this.
🏗️ Environment
- Deployment: Kubernetes manifests + Helm values per environment
- Configuration: environment variables injected at deploy time
- Application code: unchanged, last deployed version verified working
- Symptom: production service connecting to an unexpected endpoint
🤔 Your Challenge
If the code is confirmed correct and unchanged, where else could 'which endpoint the app talks to' actually be determined?
- If a URL or endpoint isn't hardcoded in the application, where does the application get it from at runtime?
- What's the difference between application logic being wrong and configuration being wrong, in terms of where you'd look?
- Could two different deployment paths (a manual change, a template default, an overridden value) disagree about what a variable should be?
- What tool or command would show you the actual, effective configuration a running instance received — not what it was supposed to receive?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
With the code confirmed correct and unchanged, the endpoint the application connects to almost certainly comes from configuration, not from anything in the codebase itself — an environment variable, a config map, a Helm value, something injected at deploy time rather than compiled into the application. That reframes the investigation entirely: this isn't a code bug to trace through application logic, it's a configuration mismatch to trace through however configuration actually gets assembled and delivered for this environment.
Step 2 — Identify the likely bottleneck
A handful of common causes produce exactly this symptom: a Helm values file or environment-specific override that has the wrong value for this environment (a copy-paste from a staging config that never got corrected), a default value baked into a template that's supposed to be overridden per environment but wasn't, in this specific deployment; a manual change made directly against the running environment that was never reflected back into the source-controlled configuration and has since been silently relied upon; or a deployment pipeline that pulled configuration from the wrong environment's source by mistake.
Step 3 — Investigation
- Check the actual, effective configuration on the running instance — not what the configuration files say should be there, but what the running pod or container actually received.
kubectl execinto the pod and check environment variables directly, or check the resolved Helm values for this specific release. - Compare that against the source-controlled configuration for this environment — if they match, the source itself has the wrong value; if they don't match, something in the deployment pipeline or a manual change is overriding it.
- Check the deployment history for this environment — was there a recent deploy, a manual
kubectl set env, or a pipeline run that touched configuration around the time this started? - Check for configuration precedence issues — many systems layer defaults, environment-specific overrides, and manual overrides, and it's easy for a lower-precedence value to be silently ignored, or a higher-precedence one to persist longer than expected.
- Check whether this environment's configuration is even fully defined, or whether it's silently falling back to a default meant for a different environment because an expected override was never actually set here.
Step 4 — Recommended action
Correct the specific misconfigured value and redeploy through the normal pipeline — not with a manual, undocumented fix that would leave the source of truth still wrong and create exactly the same risk of drift that likely caused this in the first place. If the cause was a manual change made directly against the environment, that's worth addressing structurally: restrict who can make direct changes outside the deployment pipeline, or at minimum make such changes visible and reconciled back into source control quickly.
Add a validation step — even a simple one — that checks critical configuration values (database URLs, key endpoints) against expected patterns per environment before a deployment completes, so a staging URL landing in a production config gets caught before it goes live, not after.
Step 5 — Engineering lesson
Configuration errors are invisible to code review and invisible to the application's own logic — the code did exactly what it was told, correctly, with a value that happened to be wrong. Treating configuration with the same discipline as code — version controlled, reviewed, validated, with a clear single source of truth per environment — is what prevents "the code is right but something is still wrong" from becoming a recurring category of incident.
🧠 Today’s Takeaway
Configuration is part of your application. Treat it like code.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

