---
title: Navigating Technical Debt: How a Pragmatic Kubernetes Secrets Decision Blocked Secure Growth and Led to Valuable Learnings
created: 2026-03-26
updated: 2026-03-26
type: summary
tags: ["devops", "notes", "secrets", "security", "telegram"]
external: http://go/ui/posts/2272/
status: inbox
---

Here's a strong, structured answer tailored for a DevOps/Platform engineer:

---

## The Answer

**"The technical decisions I've learned the most from weren't the ones that failed spectacularly — they were the ones that seemed completely reasonable at the time, worked fine for months, and then quietly became the source of pain that nobody could quite put their finger on. Those are the dangerous ones."**

---

**The situation:**

> About two years into my time at a previous company, I made a decision I was genuinely confident about at the time. We were building out our Kubernetes infrastructure on AWS and needed to decide how to handle secrets management. We were moving fast, the team was small, and we needed something that worked reliably without a lot of operational overhead. I chose to store secrets as Kubernetes Secrets — base64-encoded, stored in etcd, with RBAC controlling access. It was the "native" Kubernetes way, it required no additional tooling, and at the time our threat model felt manageable. I documented the decision, got sign-off, and we moved on.

> For about eight months, nobody complained. It worked.

---

**How I discovered it was wrong:**

> The first signal was indirect. We started growing faster than expected — we went from two clusters to six, from three engineering teams to nine. A new security-focused engineer joined and during her first week she did something I hadn't thought to do in a while — she read the documentation with fresh eyes and then asked a question in Slack that stopped me cold:

> *"How do we know which services have access to which secrets, and when was that access last reviewed?"*

> I didn't have a good answer. I pulled the RBAC policies and realised they had drifted significantly from what I'd originally designed. Several services had broader secret access than they needed because engineers had taken the path of least resistance when they hit permission errors — they'd widened the policy rather than scoping it correctly. There was no audit trail for who had accessed what. Rotating a secret meant manually hunting down every service that used it because there was no central registry. And because the secrets lived in etcd, our etcd backup strategy — which wasn't as hardened as it should have been — was now effectively our secrets backup strategy too.

> None of this was a breach. But when I mapped it against our compliance requirements for an upcoming SOC 2 audit, I realised we had a significant gap. The decision that had seemed pragmatic eighteen months earlier had accumulated enough technical debt that it was now actively blocking our ability to grow securely.

> I had made the wrong call — not because I was careless, but because I'd optimised for where we were rather than where we were going.

---

**What I did:**

> The first thing I did was resist the instinct to immediately start fixing it. Instead I spent two days writing a clear-eyed post-mortem — not a blame document, but an honest account of what the original decision was, what assumptions it had relied on, how those assumptions had broken down, and what the current state actually looked like. I shared it with my manager and the security engineer before anyone else. I wanted the people most affected to hear it from me directly, not through a Jira ticket.

> The post-mortem did something important beyond the technical: it named the problem precisely, which meant we could have a focused conversation about solutions rather than a diffuse conversation about how bad things were.

> We evaluated three options — HashiCorp Vault self-managed, AWS Secrets Manager, and the External Secrets Operator backed by AWS Secrets Manager. I'd learned my lesson about optimising only for the present, so this time I ran the evaluation explicitly against three time horizons: what we needed now, what we'd need in twelve months, and what we'd need at 10x scale. I also brought in the security engineer as a co-decision-maker rather than a reviewer — she had context I didn't and I wanted that reflected in the outcome.

> We chose the External Secrets Operator with AWS Secrets Manager as the backend. It gave us centralised audit logging, automatic rotation support, a clean separation between the secret store and the cluster, and crucially — a migration path that let us move services incrementally rather than in one risky cutover.

> The migration took six weeks. We wrote a runbook, moved services in order of criticality from lowest to highest, and ran both systems in parallel during the transition so we had a rollback path at every step. We also built a small internal tool that scanned for any remaining native Kubernetes Secrets and flagged them in our security dashboard — so the old pattern couldn't quietly creep back.

---

**What the outcome was:**

> We passed the SOC 2 audit without a finding related to secrets management — which had been the trigger for discovering the problem in the first place. Secret rotation went from a manual multi-hour process to an automated one. And we now had a centralised audit trail that could tell you exactly which service had accessed which secret and when.

> But the outcome I'm most proud of is less tangible. After I shared the post-mortem openly, two other engineers came to me privately and told me about decisions they'd been sitting on that they were now worried about. That kind of psychological safety — where admitting a mistake becomes a catalyst for others to do the same — is more valuable than any individual technical fix.

---

**What I learned:**

> Three things I carry with me from that decision.

> First — **optimising for now is sometimes the right call, but you have to name the assumption explicitly.** I had implicitly assumed our scale and threat model would stay roughly constant. I never wrote that down. If I had, someone might have challenged it.

> Second — **the absence of complaints is not the same as the absence of problems.** The secrets approach worked fine operationally. Nobody filed a ticket about it. That silence masked a slow accumulation of risk that only became visible when someone asked the right question.

> Third — **the way you respond to being wrong matters more than the fact that you were wrong.** Going to my manager with a clear post-mortem before being asked, owning the decision fully, and coming with a plan rather than just a problem — that's what turned a potentially damaging situation into one that actually built trust.

---

## Why This Answer Works

**The discovery mechanism is realistic and honest.** It wasn't a dramatic outage — it was a new colleague asking a good question. That's how most real problems surface, and describing it that way signals maturity and credibility.

**The post-mortem move shows leadership instinct.** Writing a clear-eyed account before anyone asked for one, and sharing it with the people most affected first, signals the kind of ownership and transparency that Canva explicitly values.

**The three time horizons in the evaluation shows you learned from the mistake in real time.** You didn't just fix the problem — you changed how you evaluated solutions because of what went wrong. That's the growth signal interviewers are listening for.

**The cultural outcome is the most memorable part.** Other engineers coming forward after seeing you own a mistake publicly is a leadership multiplier — it shows that how you handled being wrong created psychological safety for others. That's a SCL answer nested inside a technical one.

---

## One Sentence to Open With

> *"The decision I'm going to tell you about wasn't wrong because I was careless — it was wrong because I optimised for the problem we had rather than the problem we were about to have, and I didn't write down that assumption explicitly enough for anyone, including me, to challenge it."*

That opener does three things at once: it shows self-awareness, it frames the failure as a systems problem rather than a personal one, and it immediately signals the lesson before you've even told the story — which primes the interviewer to listen for growth rather than just failure.
