5.6 KiB
title: Engineering Decisions Under Uncertainty: Migrating a High-Volume Data Pipeline to Kafka on MSK with Risk Mitigation and Timely Delivery created: 2026-03-25 updated: 2026-03-25 type: summary tags: ["engineering", "migration", "notes", "strategy", "telegram"] external: http://go/ui/posts/2264/
Here’s a strong, structured answer you can adapt:
The Answer “My approach to decisions under uncertainty is to be honest about what I don’t know, shrink the cost of being wrong, and move deliberately rather than waiting for certainty that will never come.”
The situation: At my previous company we were running a data pipeline that processed customer analytics events — roughly 2 million events per day at the time. We’d outgrown our existing setup: a mix of cron jobs and a homegrown queue built on top of Postgres. It was showing serious strain — jobs were falling behind, we had no visibility into failures, and the on-call burden was growing. Leadership wanted a solution in place before we hit our projected 10x growth from an upcoming product launch, which was eight weeks away. The decision I had to make was which streaming platform to migrate to. The two main contenders were Apache Kafka and AWS Kinesis. Both were viable. Both had strong community support. But we had no internal experience with either, no existing benchmarks for our specific workload, and eight weeks wasn’t enough time to properly pilot both.
How I approached it: The first thing I did was separate what I actually knew from what I was assuming. I wrote a decision document — not to circulate immediately, but to force myself to be precise. I listed our hard requirements: at-least-once delivery, ability to replay events, support for at least 50k events per minute with headroom, and operational simplicity given our small team. Then I listed the things I genuinely didn’t know: how each platform would behave under our specific traffic patterns, what the operational overhead would feel like day two, and whether our team could get productive fast enough. Once I had that written down, I realised the biggest risk wasn’t picking the wrong technology — it was building deep coupling to whichever one we chose and then being stuck if it was wrong. So before I even evaluated the platforms, I designed an abstraction layer — a thin internal interface for “publish event” and “consume event” — so the core pipeline code would never talk directly to Kafka or Kinesis. That decision alone reduced the cost of being wrong significantly. Then I ran a time-boxed spike — three days, two engineers. Rather than trying to benchmark everything, I identified the two scenarios most likely to break us: high-throughput burst ingestion and consumer lag recovery after a processing outage. We tested those specifically, not general performance.
What we found and decided: Kinesis was simpler to operate but had a hard shard limit that would require manual intervention to scale past a certain point — that felt like trading one operational burden for another. Kafka gave us more control and the partition model mapped cleanly to our workload, but it meant running our own cluster. We chose Kafka on MSK — AWS’s managed Kafka — which gave us the Kafka semantics without the operational overhead of self-managing brokers. I was transparent with leadership that this was a high-confidence decision, not a certain one. I told them: “Based on what we know, Kafka on MSK is the right call. If our traffic patterns surprise us, the abstraction layer means we can swap the underlying platform in two weeks rather than two months.”
The outcome: We shipped the migration six weeks later — two weeks ahead of the deadline. The platform handled the product launch without incident. Event throughput peaked at around 80k per minute during launch day, well within headroom. The abstraction layer turned out to matter: six months later we evaluated moving one pipeline to Kinesis for cost reasons, and because of the interface we’d built, it was a week of work rather than a full migration.
Why This Answer Works The “separate known from unknown” move is senior-level thinking. Most engineers jump straight to evaluating options. Writing down what you don’t know first is a discipline that interviewers rarely hear — it signals intellectual honesty and structured thinking. The abstraction layer shows you managed risk, not just made a bet. You didn’t just pick a technology and hope — you actively reduced the downside of being wrong before you even started evaluating. That’s engineering judgment, not just technical knowledge. The time-boxed spike shows pragmatism. You didn’t try to learn everything. You identified the two failure modes that scared you most and tested those specifically. That’s how senior engineers operate under time pressure. You were transparent with leadership. Saying “high-confidence, not certain” is mature. It builds trust because leaders know you’re not overselling — so when you do say you’re confident, they believe you.
One Sentence to Open With “When I don’t have enough information to be certain, my first move is always to reduce the cost of being wrong — because usually you can’t eliminate the uncertainty, but you can almost always shrink the blast radius if you’re wrong.” That opener reframes the question immediately — instead of “how do you decide without information,” you’re answering “how do you engineer around uncertainty.” That’s a much more impressive frame, and it’s exactly what Canva’s strategy competency is listening for.