Working Notes

There's a specific moment most on-call engineers recognize. You're in a post-mortem, walking through a memory exhaustion failure that took longer to diagnose than it should have. Someone mentions they've seen this pattern before. Someone else says they filed a ticket after the last one. You go looking for the ticket. It's closed — marked resolved after the immediate fix, never connected to anything in the roadmap. The underlying failure mode is still there. You're just meeting it again. That moment is worth examining, because it reveals something about how reliability work actually moves — or doesn't — through most technology organizations. The on-call engineers usually have detailed, accurate knowledge of where the system is fragile. They know which failure modes the runbooks don't fully cover, which write patterns cause CPU plateaus the monitoring doesn't catch, which edge cases will eventually surface as customer escalations. That knowledge gets expre...

Nobody files a support ticket to say the database was up all day.  Users don't praise reliability. They don't notice it at all — until it's gone. And when it goes, they don't think "the SLO was breached." They think "I lost data" or "my app was down." They think it twice, and then they start looking for alternatives. That asymmetry is especially sharp in database products, where reliability isn't one thing — it's several, and they can fail independently. Not a Single Dial When teams talk about database reliability, availability usually dominates the conversation. Is the endpoint reachable? Is the cluster healthy? Can we fail over? Those questions matter. But availability failures have a particular character: they're visible, they're shared, and recovery brings relief. An outage is a trauma event. Everyone knows it's happening, everyone mobilizes, and when the cluster comes back, there's a collective exhale. It gets a...

Most OLTP databases are predominantly read workloads. SELECT statements outnumber INSERT, UPDATE, and DELETE by a wide margin — sometimes by an order of magnitude. Users browse and display far more often than they create or modify. Read scaling gets attention first because read pressure arrives first, and the solutions are well-understood: replicas, caches, connection pooling. It works. Writes arrive later and are more disruptive when they do. In the  previous post on schema as interface , the argument was that your schema has clients who depend on certain access patterns being cheap. Writes are a new kind of client the original design often didn't account for. Indexes that accelerate reads add overhead to every write. Constraints that protect integrity can increase contention. The schema doesn't change. The workload does. And suddenly the interface is expensive in new ways. Writes expose the actual shape of your system. Not the architecture diagram shape — the real shape. Wher...

Most teams adopt an ORM for good reasons. In the early sprints, having the database feel like a natural extension of the application — models mapping to tables, relationships expressed in code — lets teams move fast without needing deep SQL fluency on every story. The ORM earns its place well before the first release. What I've observed is what happens after. The product ships, sprint priorities shift toward feature work and production issues, and the ORM quietly becomes tribal knowledge. The engineers who set it up carry a mental model of how it works and why certain decisions were made, but that understanding rarely survives team rotation — much like documentation, it gets treated as something someone will formalize eventually. New engineers learn just enough to be productive, the ORM fades into the background, and by the time something goes seriously wrong with the database, it's effectively a museum piece. This is the Day 2 problem with ORMs — not that they're the wr...

When people say “schema as interface,” they usually mean naming and semantics. That’s part of it. But in a production system, the real interface is the set of access paths the query planner can execute cheaply. Every `SELECT`, join, background job, and dashboard query depends not just on tables and columns, but on index shape, cardinality, constraints, and data distribution. The schema is not only a contract of meaning; it is a contract of performance. I’ve seen more systems struggle from schema drift than from application bugs. Not because the code was wrong, but because the data model made correct code slow. A table starts simple and well-indexed. Features accumulate. Nullable columns get added “just in case.” Enums become free text. Indexes are added reactively. At 50k rows, everything works. At 50 million, joins that used to be cheap become dominant costs. Nothing is technically broken—but the interface has changed. We would consider an API broken if latency jumped from 40ms to 4 s...

There’s a growing cottage industry on LinkedIn devoted to spotting AI-written content. People trade lists of tells—emojis, bullet points, excessive em dashes—as if the problem is detection. It isn’t. Those tells are just symptoms. The real issue is that a lot of what’s being published has no point of view, regardless of how it was produced. I came to this topic the same way many posts start: with an observation and a vague sense that something felt off. Using ChatGPT didn’t give me the idea. It helped me sharpen it. Through iteration, the problem became clearer—not that AI is writing content, but that it’s being used as a substitute for thinking. When that happens, the output is predictable: fluent, tidy, and empty. This failure mode isn’t new. Long before AI, we were already summarizing conference talks, management books, and other people’s blog posts. The same ideas get rephrased, lightly personalized, and pushed back into the feed. What’s usually missing isn’t information—it’s i...