Systems Distributed Conference Recap: The Growing Gap Between What We Build and What We Understand
It had been a VERY long time since I attended an engineering conference—2019, to be exact. I spoke at an internal conference during my time at Apple, but between work and home, attending external events was difficult. Now that my youngest is three, I decided it was time to venture out.
I learned about the Systems Distributed Conference, hosted by TigerBeetle, through Ray Myers. We’re both part of a coaching cohort led by Andrea Goulet.
Given my work on embedded systems across three lines of business at Apple—Watch, Mac, and iPhone—and my earlier experience building SaaS products such as Mint.com, I’m always drawn to distributed systems.
Across two days of talks on systems languages, databases, and distributed systems, one idea kept resurfacing: the tools to build reliable systems have existed for decades. The gap now is comprehension and craft, not capability.
The conference wasn’t centered on one shiny new breakthrough. Instead, it felt like a rediscovery of discipline: understand your system, assume it is flawed, and design accordingly. Having shipped embedded hardware at Apple and SaaS at Mint, I’ve seen what happens when we lose that discipline at scale and why it’s worth recovering.
These are the four ideas I’m still thinking about—and, at the end, a book that ties them together. I’d love to hear what stood out to others who attended.
1. The Comprehension Gap
James Cowling, CTO of Convex.dev, spoke about the importance of abstractions and how they help us develop software and ship products faster. An application developer, for example, doesn’t need to understand all the inner workings of a CPU to build something valuable.
But as the number of abstractions increases, that same convenience creates what Cowling called the comprehension gap: a widening distance between what a system does and how well the people working on it understand why it behaves that way.
Cowling then sharpened the point in a way I keep thinking about. Developers, he noted, never really cared how filesystems work, how a CPU is architected, how a language is compiled, or how a third-party service is built—and that was fine.
What’s new is that they increasingly don’t care how their own systems work. He called this the caring gap.
A comprehension gap is a natural consequence of abstraction; a caring gap is a choice. The distance only becomes dangerous when no one feels responsible for closing it.
I’ve heard versions of this from people who vibe-coded an application over the past year. Having AI produce software that delivers the desired result is valuable—but the moment something breaks, the person who built it may be stuck because they don’t understand how the underlying system works.
AI didn’t create this problem. It accelerates a tension that has always existed in software: abstractions help us move faster, but they can also hide the details we need when something goes wrong. From what I’ve seen, the comprehension gap widens for two primary reasons.
Complexity fragments ownership and understanding
Modern systems can be too complex for one person—or even one team—to understand in their entirety. I’ve experienced this first hand while working on products like the iPhone. Each directly responsible individual might be focused on a particular module: the compass, accelerometer, gyroscope, camera, firmware, or software layer. Each team necessarily designs and tests its component according to its own understanding of the customer use cases.
Then everything comes together in burn-in testing before mass production—and suddenly the system lights up with issues that no single module produced on its own. A defective part, an unusual thermal ramp, or a software bug introduces noise elsewhere in the system, and now several teams are pulled in to unravel it.
SaaS systems have the same problem in a different form. A slow page load could originate in the user’s device or browser, the front-end code, the network, an application server, or the database. The visible symptom tells you very little about the underlying cause.
Testing helps—unit, acceptance, integration, load—but I have yet to see a company, large or small, that can test every possible interaction and condition.
What I have seen teams use to close the gap is observability: telemetry, performance measurement, and profiling to find the conditions that produce a failure. From there it often takes cross-functional teams and vendors to trace the issue across organizational and technical boundaries. That process can take weeks, if not months.
The issue isn’t that people don’t care or aren’t doing their jobs. It’s that specialization gives each person a detailed view of one part of the system while making the interactions between those parts harder to see.
Time-to-market forces tradeoffs
The second reason is time-to-market. When something has to ship on schedule—and frankly, what doesn’t?—teams have to prioritize the problems they can solve. That means bugs get shipped.
The practical question isn’t whether a system has bugs. It’s which bugs are acceptable to ship—based on frequency, severity, detectability, and effect on the customer.
Engineers have had a formal language for this for decades: FMEA, where a risk priority number is roughly occurrence × severity × detection. Hardware teams measure it in DPPM (defective parts per million); SaaS teams watch error rates, performance, and availability alongside customer satisfaction and churn.
Teams then decide whether to delay a release, mitigate the problem, or resolve it after shipping through a hotfix, patch, or software or security update.
Cowling’s answer wasn’t that everyone should suddenly understand everything—it was that platforms should be built so that not understanding everything is safe.
His favorite property was composability, which he defined with the best line of the talk: composability is abstractions that care about each other. When they cooperate—checking each other’s assumptions, composing without surprising interactions—people don’t have to hold the entire system in their heads.
Good composition, he argued, is what keeps complexity from exploding combinatorially as systems grow.
Abstractions aren’t the enemy here, and neither is speed—both make modern software development possible. The danger is believing that because an abstraction works, we no longer need to understand what it conceals—or assuming someone else will understand it for us.
2. Designing for Risk
While my kids think I’m pretty cool for writing software for some of their favorite devices, I reminded them that people paved the way for engineers like me.
One of those people is Gerard Holzmann, best known for designing the Spin logic model checker. Spin has been broadly used to verify the logic of multithreaded software systems, including software used in missions such as the Curiosity rover’s landing on Mars.
I don’t recall Holzmann making a single reference to AI during his talk, but the content felt especially relevant to the present moment. His central point was evergreen: all software must be considered potentially flawed.
For software engineers who care about quality and craftsmanship, I’ve included his slide outlining the Power of Ten—Rules for Developing Safety-Critical Code, published in 2006. The rules offer ten technical practices for reducing risk and making software easier to inspect, reason about, and verify.
What stood out to me wasn’t simply the individual rules—it was the mindset behind them. If we assume software is potentially flawed, risk management can’t begin after something breaks; it has to shape the design from the very first decision.
Every system carries risk, but not every risk deserves the same investment.
The technical question: do we understand the system well enough to identify its likely failures?
The business question: do we understand the consequences well enough to decide which failures we can tolerate?
Large companies are expected to manage risk; smaller ones take on more of it to stay nimble and iterate. But regardless of size, systems eventually outgrow their original architecture and assumptions.
At that point the choices get hard: re-architect, keep patching, accept the limits, or sunset the product entirely.
Even sunsetting a product doesn’t eliminate its users’ underlying need. Mint disappeared; the demand for personal financial visibility did not. Products change, but the responsibility to understand the problem, the risks, and the consequences remains.
This is where engineering and entrepreneurship meet. Both mean identifying a problem, betting on a solution, shipping, and learning from what happens next.
Moving quickly doesn’t remove the responsibility to understand what you’re building—it raises the stakes on the tradeoffs.
3. Where Databases Are Heading
The next two ideas are the most technical in this piece, and I’ve kept them that way on purpose. This is where the conference earned its authority, and where the thesis stops being abstract.
One shape recurs across both. A familiar abstraction quietly hides an assumption. Someone returns to the foundation and asks whether that assumption was ever necessary. The deeper understanding buys a better architecture. Eventually the rest of us get access to the result.
Once you have the shape, you’ll see it in every example below.
Databases dominated the hallway conversations. The striking part was how much of it was about tearing them down to the studs and rebuilding from first principles—not adding features to what we already have. Four directions stood out.
Purpose-built beats general. The old default was a general-purpose SQL database bent to fit any workload. TigerBeetle, the conference host, goes the other way: it treats debits and credits as first-class primitives instead of generic rows.
The shift is technical. That specialization is what lets it batch thousands of transactions into a single round trip, by TigerBeetle’s own account.
The impact is on the business. You get order-of-magnitude performance that a general database can’t match—Jim Gray’s old point that databases exist to serve business transactions, not abstract data.
Distributed and fault-tolerant by default. Rather than build a single-node database and bolt on replication later, newer systems assume from line one that machines fail, disks silently corrupt data, and clocks drift.
The shift is technical. TigerBeetle uses Viewstamped Replication over Raft, checksums everything, and is built to survive faults—silent corruption, clock drift—that many databases ignore.
The impact is on the business. Fewer catastrophic, hard-to-diagnose outages in production, and less of the 3 a.m. forensics that follows them.
Reliability as architecture, not afterthought. This is the one I keep coming back to, because it connects straight to Holzmann. TigerBeetle is deterministic by design.
The shift is technical. That determinism lets them run a simulator that—by their own account—compresses roughly two thousand years of rare-failure scenarios into a single day.
It’s Holzmann’s “assume all software is flawed” turned into practice: you don’t hope the edge cases won’t happen, you manufacture them by the millions before you ship.
And this idea isn’t new—it’s finally spreading. FoundationDB was doing deterministic simulation testing more than a decade ago, so thoroughly it became a legend in distributed-systems circles. Apple acquired it in 2015 (which, given my years there, made me smile).
Ten-plus years later the same approach shows up in TigerBeetle—and Antithesis, founded by FoundationDB alumni including CTO Ben Collins, who I was lucky to get to know as a neighbor/fellow-parent/Apple co-worker. Antithesis runs as a product any team can point at their own system.
The impact is on the business. The technique was proven a decade ago; what’s changed is access. You no longer have to build your database from scratch to use it—which is this whole post in one example: the capability was always there, and comprehension and access are catching up.
Even the foundations aren’t settled. For decades the textbook rule for agreement was that every quorum must be a majority—because any two majorities overlap, and that overlap is what stops a system from ever deciding two different things.
The shift is technical. Heidi Howard’s Flexible Paxos shows the rule is stricter than correctness requires: consensus runs in two phases, and quorums only need to intersect across phases, not within them.
The impact is on the business. This unlocks faster, more flexible designs—smaller quorums in the steady state, or quorums reshaped around geography for lower latency across regions.
It’s the comprehension gap in its purest form: we inherited a rule and forgot it was ever a choice.
Put together, these point at a database that looks less like a single monolith and more like a disaggregated datacenter—storage, memory, and compute as independently scalable pools—held together by consensus we finally understand well enough to bend. Which brings me to the people doing that foundational work.
4. Open Source, Academia, and Lost Provenance Contribute to The Comprehension Gap
Five of the talks I attended were given by Andrew Kelley, creator of Zig; Aishwarya Ganesan; Ram Alagappan; Heidi Howard; and Jens Axboe, creator of io_uring. I won’t attempt to summarize all of them, because I’m still working through parts of their work. But one talk made the value of academic systems research concrete for me, so I’ll use it as an example.
Aishwarya Ganesan, of the University of Illinois Urbana-Champaign, gave a talk called “Breaking the Server: Systems for Disaggregated Datacenters.”
Her starting point is a constraint every hardware and infrastructure engineer will recognize: a monolithic server forces compute, memory, and storage to scale together, because they ship as a single unit. Need more memory but not more CPU? You buy the whole server anyway, and over-provision the rest.
Disaggregation breaks that bundle apart. Compute, memory, and storage each become an independent pool you can scale on its own—scale the resource you need, not the whole server.
Storage disaggregation is already mature; as Ganesan put it, “S3 is the disk.” Memory is the harder, newer frontier—moving from RDMA-based approaches toward CXL (Compute Express Link), which lets a host reach a pool of memory over PCIe using ordinary load and store instructions.
CXL 3.0 goes further, promising terabytes of memory shared across multiple hosts. Ganesan illustrated the appeal through Tigon, a CXL-native database design (Huang et al., OSDI ’25): hosts coordinate through shared memory instead of network messages, and her slide put the promise bluntly—cross-host transactions with “no messages, no two-phase commit.”
That is the promise. The rest of her talk was about the catch.
The part that stayed with me was where the abstraction gets leaky. Hardware cache coherence covers only a small slice of that shared memory—a few hundred megabytes—while the large, multi-terabyte region has no cross-host coherence at all. In that region, one host’s processor cache can hold a stale value after another host writes to the same location, producing silent correctness bugs precisely when data is shared across hosts.
I found that detail striking because it is the comprehension gap in miniature. “Shared memory across hosts” is a clean, appealing abstraction. Underneath it sits a partly-coherent model where the convenient mental picture is simply wrong—in ways that surface only as data-corruption bugs at scale.
The naive fix—keep a small coherence record for each object in the scarce coherent region, and put the data itself in the large non-coherent one—collapses under load: once the coherent region fills, sharing one more object means un-sharing another, and with a large dataset the system thrashes until throughput falls off a cliff.
Megalon, her group’s system, escapes the trap with one elegant observation: not all metadata is alike.
The index mapping objects to locations is large but rarely changes, so it can be replicated into each host’s plentiful DRAM and kept consistent cheaply through a shared log. The coherence records are tiny but change constantly, so only those need the scarce coherent region—and read-only objects need no record at all.
The payoff, in their measurements: coherent sharing of terabytes using just a small coherent region, beating the naive approach by more than 8x.
The transferable lesson, which Ganesan drew out explicitly: when a resource is scarce and critical, spend it on the smallest, most critical piece of state—and design around the hardware you actually have, not a simpler picture of it.
It’s the same instinct I watched shape every hardware tradeoff at Apple, where the scarce resource was die area, power, or thermal headroom, and the discipline was always to spend it where it mattered most.
There is the shape again—the one I named at the top of the databases section: a clean abstraction hides an assumption, someone returns to the foundation, and a deeper understanding buys a better architecture.
I’m not doing the work justice in a few paragraphs, and that is rather the point. This is the kind of deep, unglamorous systems research—performance, correctness, language and architecture design—that happens in academia and open source, often years before industry adopts the results. And when industry does, it often loses track of the reasoning, and the people behind them.
That lost provenance contributes to the comprehension gap. We inherit an abstraction but not always the constraints, tradeoffs, or original problem that shaped it. Over time, the abstraction becomes a given: we know how to use it, but not necessarily why it was designed that way, which assumptions it makes, or where it begins to break down.
This isn’t an argument against abstraction—engineers shouldn’t have to reconstruct decades of computer science before they can ship a product. But when we lose the origins of a tool, we can also lose the knowledge required to use it responsibly. We may treat a deliberate tradeoff as an accidental limitation, or assume a tool can solve problems it was never designed to address.
That is why the work happening in academia and open source matters beyond the immediate technical contribution. It preserves and advances the craft behind the capabilities that industry eventually packages and scales. It also gives us somewhere to return when the abstraction stops behaving as expected.
A Book That Ties It Together
One of my takeaways from the conference is that closing the comprehension gap doesn’t mean memorizing more implementation details. It means becoming better at reasoning about systems: making assumptions explicit, evaluating conditions, identifying contradictions, and anticipating failure modes.
For that, I recommend Logic for Programmers by Hillel Wayne. Hillel was also fun to talk to—kindly peppering me with questions about the products I’d worked on in the past. 😉
Closing Thoughts
We’re in an era where abstractions and AI let us ship faster than ever, and that is genuinely valuable. But every layer we add can widen the distance between what we ship and what we actually understand. and, as Cowling’s reframe warns, between what our systems do and what anyone still feels responsible for.
The talks pointed at the same answer from different directions. Composability—abstractions that care about each other—so the burden of understanding doesn’t fall entirely on people. Risk treated as a deliberate business decision, not an accident discovered in production.
Provenance is preserved, so the assumptions behind a tool aren’t lost when the tool is adopted. And architectures designed around the hardware we actually have, not a convenient mental model of it.
Each is a way of keeping the assumptions an abstraction hides from going missing.
So the fuller lesson isn’t simply that we need more comprehension. It’s that reliable systems require someone—or something in the architecture itself—to hold onto the assumptions, constraints, and responsibility that every abstraction quietly conceals.
