Working Notes

A green dashboard and a healthy database are not the same thing. That distinction sounds obvious until you're forty minutes into an incident where every panel looks nominal and your storage layer is silently throttling writes for a specific tenant on a specific shard. That scenario is not hypothetical. It's the shape of the failures dashboards are structurally unable to surface — not because the tooling is bad, but because dashboards encode what you already know to look for. Observability is the capability to answer questions you haven't thought to ask yet. Those are different things, and treating one as the other leaves you blind at exactly the moment it matters most. Here's the structural problem: dashboards encode past hypotheses. That replication lag panel exists because you once had a replication lag problem. That connection pool chart was built after a connection exhaustion incident. The query throughput tile reflects the questions your team had during whatever po...

The frustration is real. LinkedIn is full of it — entry-level roles requiring three to five years of experience, candidates rejected before they get a conversation, a generation convinced the door is locked from the inside. That frustration is worth taking seriously. But most of the conversation stops at the complaint. What's less discussed is the mechanism: why the mismatch exists, who's responsible for it, and what can actually be done. I've been on the hiring side enough times to know we're not blameless. What candidates can do with this If you're early in your career, the job description problem cuts in your favor more than you might think. Most postings describe an ideal candidate, not a threshold. If you meet the core requirements — the actual must-haves — apply. The worst outcome is silence, which you were already getting by not applying. The experience you have is probably more credible than you're presenting it. Internships, co-ops, serious side project...

Last week, I spent a morning at Snowflake's Data for Breakfast event, which is exactly what it sounds like: an early session where practitioners walk through what's new in the platform and where customers are actually getting value. The through-line across most of the presentations wasn't a specific feature. It was a concept that kept surfacing as a prerequisite for making AI work reliably against real business data. They called it the semantic layer. The framing was direct: without it, AI tools querying your warehouse will produce answers that are fast, fluent, and inconsistently grounded in what your business actually measures. With it, you give the AI something reliable to reason against. That framing stuck with me — partly because it applies well beyond Snowflake, and partly because it names something that data teams have wrestled with long before AI entered the picture. What a Semantic Layer Actually Is The term sounds more technical than it needs to be. At its core, a...

There's a specific moment most on-call engineers recognize. You're in a post-mortem, walking through a memory exhaustion failure that took longer to diagnose than it should have. Someone mentions they've seen this pattern before. Someone else says they filed a ticket after the last one. You go looking for the ticket. It's closed — marked resolved after the immediate fix, never connected to anything in the roadmap. The underlying failure mode is still there. You're just meeting it again. That moment is worth examining, because it reveals something about how reliability work actually moves — or doesn't — through most technology organizations. The on-call engineers usually have detailed, accurate knowledge of where the system is fragile. They know which failure modes the runbooks don't fully cover, which write patterns cause CPU plateaus the monitoring doesn't catch, which edge cases will eventually surface as customer escalations. That knowledge gets expre...

Nobody files a support ticket to say the database was up all day.  Users don't praise reliability. They don't notice it at all — until it's gone. And when it goes, they don't think "the SLO was breached." They think "I lost data" or "my app was down." They think it twice, and then they start looking for alternatives. That asymmetry is especially sharp in database products, where reliability isn't one thing — it's several, and they can fail independently. Not a Single Dial When teams talk about database reliability, availability usually dominates the conversation. Is the endpoint reachable? Is the cluster healthy? Can we fail over? Those questions matter. But availability failures have a particular character: they're visible, they're shared, and recovery brings relief. An outage is a trauma event. Everyone knows it's happening, everyone mobilizes, and when the cluster comes back, there's a collective exhale. It gets a...

Most OLTP databases are predominantly read workloads. SELECT statements outnumber INSERT, UPDATE, and DELETE by a wide margin — sometimes by an order of magnitude. Users browse and display far more often than they create or modify. Read scaling gets attention first because read pressure arrives first, and the solutions are well-understood: replicas, caches, connection pooling. It works. Writes arrive later and are more disruptive when they do. In the  previous post on schema as interface , the argument was that your schema has clients who depend on certain access patterns being cheap. Writes are a new kind of client the original design often didn't account for. Indexes that accelerate reads add overhead to every write. Constraints that protect integrity can increase contention. The schema doesn't change. The workload does. And suddenly the interface is expensive in new ways. Writes expose the actual shape of your system. Not the architecture diagram shape — the real shape. Wher...