Working Notes

Effective site reliability engineering demands two distinct skills that rarely get named together: the ability to act decisively under pressure with incomplete information, and the ability to extract meaning from data that wasn't designed with you in mind. The first is the ER physician's skill. Triage doesn't wait for complete information. You're reading vitals, interpreting presenting symptoms, and making consequential decisions in compressed time. Speed matters, but so does pattern recognition. The wrong call isn't just inefficient — it can make things worse. The second is the data scientist's skill. The raw data exists. The challenge is knowing which of it means something. Signal extraction requires understanding not just what the instruments are reporting, but why they were placed there, what assumptions they encode, and where their blind spots are. A metric you don't understand is noise with a label on it. Effective SRE work requires both modes — someti...

A green dashboard and a healthy database are not the same thing. That distinction sounds obvious until you're forty minutes into an incident where every panel looks nominal and your storage layer is silently throttling writes for a specific tenant on a specific shard. That scenario is not hypothetical. It's the shape of the failures dashboards are structurally unable to surface — not because the tooling is bad, but because dashboards encode what you already know to look for. Observability is the capability to answer questions you haven't thought to ask yet. Those are different things, and treating one as the other leaves you blind at exactly the moment it matters most. Here's the structural problem: dashboards encode past hypotheses. That replication lag panel exists because you once had a replication lag problem. That connection pool chart was built after a connection exhaustion incident. The query throughput tile reflects the questions your team had during whatever po...

The frustration is real. LinkedIn is full of it — entry-level roles requiring three to five years of experience, candidates rejected before they get a conversation, a generation convinced the door is locked from the inside. That frustration is worth taking seriously. But most of the conversation stops at the complaint. What's less discussed is the mechanism: why the mismatch exists, who's responsible for it, and what can actually be done. I've been on the hiring side enough times to know we're not blameless. What candidates can do with this If you're early in your career, the job description problem cuts in your favor more than you might think. Most postings describe an ideal candidate, not a threshold. If you meet the core requirements — the actual must-haves — apply. The worst outcome is silence, which you were already getting by not applying. The experience you have is probably more credible than you're presenting it. Internships, co-ops, serious side project...

Last week, I spent a morning at Snowflake's Data for Breakfast event, which is exactly what it sounds like: an early session where practitioners walk through what's new in the platform and where customers are actually getting value. The through-line across most of the presentations wasn't a specific feature. It was a concept that kept surfacing as a prerequisite for making AI work reliably against real business data. They called it the semantic layer. The framing was direct: without it, AI tools querying your warehouse will produce answers that are fast, fluent, and inconsistently grounded in what your business actually measures. With it, you give the AI something reliable to reason against. That framing stuck with me — partly because it applies well beyond Snowflake, and partly because it names something that data teams have wrestled with long before AI entered the picture. What a Semantic Layer Actually Is The term sounds more technical than it needs to be. At its core, a...

There's a specific moment most on-call engineers recognize. You're in a post-mortem, walking through a memory exhaustion failure that took longer to diagnose than it should have. Someone mentions they've seen this pattern before. Someone else says they filed a ticket after the last one. You go looking for the ticket. It's closed — marked resolved after the immediate fix, never connected to anything in the roadmap. The underlying failure mode is still there. You're just meeting it again. That moment is worth examining, because it reveals something about how reliability work actually moves — or doesn't — through most technology organizations. The on-call engineers usually have detailed, accurate knowledge of where the system is fragile. They know which failure modes the runbooks don't fully cover, which write patterns cause CPU plateaus the monitoring doesn't catch, which edge cases will eventually surface as customer escalations. That knowledge gets expre...

Nobody files a support ticket to say the database was up all day.  Users don't praise reliability. They don't notice it at all — until it's gone. And when it goes, they don't think "the SLO was breached." They think "I lost data" or "my app was down." They think it twice, and then they start looking for alternatives. That asymmetry is especially sharp in database products, where reliability isn't one thing — it's several, and they can fail independently. Not a Single Dial When teams talk about database reliability, availability usually dominates the conversation. Is the endpoint reachable? Is the cluster healthy? Can we fail over? Those questions matter. But availability failures have a particular character: they're visible, they're shared, and recovery brings relief. An outage is a trauma event. Everyone knows it's happening, everyone mobilizes, and when the cluster comes back, there's a collective exhale. It gets a...