Just when we shelter under paper, the rockets come at us sideways.

—Meat Puppets, Backwater

As a software engineer, the chief manifestation of this mistake I see up close is the accumulation of technical debt. Some amount of debt is natural; and, in fact, the amount of debt may reasonably increase over time (implying a tech deficit), so long as the team’s ability to deliver increases at an even greater rate. But if each new integration, customer, etc. is no faster or easier than the last, then you cannot scale any faster than linearly (O(N)), which is typically not good enough for a startup with limited runway; nor, indeed, for any initiative in a competitive market, since somebody else will soon be better at this (whatever “this” is) than you are. If you’re not actively tracking your debt vs. delivery rate, it's likely that your acceleration is negative, meaning your ability to scale isn’t even linear.

When institutional debt—technical or otherwise—interferes with acceleration, a common response is to hire more people. Surprisingly, this makes the problem worse, not better. This principle is sometimes called Brooks’s Law:

Adding manpower to a late software project makes it later.

—Fred Brooks, The Mythical Man-Month

The inclination to expand the team, even when management is intellectually aware of this principle, stems from a misunderstanding of the root cause of the problem. The issue is neither position, nor velocity, but rate of change in that velocity. If you’re going in the wrong direction, then doing that faster makes things worse, not better. A more intuitive phrasing of Brooks’s Law may be the old joke (whose origin eludes me, but please do enlighten me in the comments):

We're losing money on every unit, but we'll make it up in volume.

If the reason you’re not moving fast enough is that you’re accumulating tech debt faster than you’re acquiring reliable automation, then hiring more people is like trying to “make it up in volume.” You’re only making the problem worse.

A lot of Deeply Nested readers will be familiar with the tribulations of trying to deliver market value in the face of accumulated tech debt, bureaucracy, or other institutional inertia. Aside from tech debt, the most common form I’ve seen in person has been badly structured deals, the architects of which are often long gone by the time I arrive. Deal structuring is an underrated art form. If you’re in the process of signing a new customer to a long-term support contract, take care what concessions you make: They will have a real impact on your life (or at least, the lives of your teammates) for years to come.

If it feels like you’re spinning your wheels faster and faster, but not getting anywhere, then it’s likely you’re not understanding the actual impediments to progress. These are often several degrees removed from easily observable metrics like delivery or integration time, instead amounting to things like code complexity or customer support burden. The only way to defeat these impediments once you’ve identified them, as far as I know, is head on. The direction of your velocity is more important than its magnitude, and its derivative acceleration is more important than either. If your weapons are pointed in the wrong direction, you’re liable to suffer more damage from friendly fire than you inflict on your opponents.

If you agree with this sentiment in principle, but simply can’t afford the time to make fundamental changes, then it’s time to move on. You’re like a passenger on a sinking ship who can’t stop bailing water long enough to plug the holes in the hull. So, at some point, you have to ask yourself: Is this boat sufficiently maneuverable to reach a safe harbor before it sinks? You might as well take drastic measures, when the only alternative is going down with the ship.

2 in 1 shampoo is not real, because 1 is not big enough to hold 2. That's why 2 was created. If it was 2 in 1, it would be overflowing.

—Mitch Hedberg

What you must learn is that these rules are no different than the rules of a computer system. Some of them can be bent. Others can be broken.

—Morpheus, in The Matrix

Pairs of countable things are themselves countable. In other words: Any pair of integers can be associated with a single scalar integer, distinct from the integer representing any other pair.

One way is to raise mutual primes, such as 2 and 3, to the original values, then multiply the results to get a single number, as in the following ‘encode’ function:

This is more powerful than simple addition, multiplication, or exponentiation, because it’s guaranteed to be reversible. We can recover the inputs by factoring, as in this ‘decode’ function:

For example, we can encode the pair (3, 5) as the single value 1944, because 2**3 = 8, 3**5 = 243, and 8 * 243 = 1944:

Let’s take this trick for a spin on everyone’s favorite hobbyhorse, the Fibonaccis.

The Fibonacci sequence (OEIS A000045) is a well known plaything of mathematicians and programmers alike. It begins [0, 1], and each subsequent number is the sum of the previous two. The first ten Fibonacci numbers are therefore [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]:

A loop for generating Fibonacci numbers obviously must maintain two integer values as state, because, by definition, it needs them to compute the next value:

But using our handy integer compression trick, we can amaze friends and family by implementing a Fibonacci loop that maintains only a single integer between iterations!

Can we repeat the process, reducing any four values into two scalars, and thence to a single integer? By extension, can we compress arbitrary amounts of data into a single integer? Yes in principle, but no in practice, because the exponentiation quickly causes overflow, or turns BigInts into RAM hogs.

Does this trick have any practical application at all? None whatsoever, as far as I know. It’s just cool enough that I thought you should know about it. 😎

How should we present inline code?

Substack’s Gist integration broke, and they don’t seem inclined to fix it. I’ll be updating old posts to replace the Gists with… something. But going forward, Gists are no longer really an option. I see two viable alternatives:

1. Code blocks. Substack now has inline code blocks that are, dare I say, almost usable. They still have some serious drawbacks: They don’t support syntax highlighting. They’re limited to 73 columns on desktop, and some laughable width on mobile. If we want line numbers, those come out of our line width budget. They don’t support captions or alt text.

2. Screenshots. The most flexible option is to screenshot code, and hyperlink plain text versions in the image captions. The primary limitation of screenshots is that they force readers who can’t visually see the code to click through links.

   By the way, Substack’s alt text support is now completely broken. Seriously, they lose the image entirely if I add alt text. On the bright side, it’s so broken that maybe they’ll actually fix it.

For comparison, here’s a code block, followed by the same code as a linked screenshot:

/// Upper cases text from r, and writes it to w.
fn to_upper<R: BufRead, W: Write>(
    mut r: R,
    w: &mut W,
) -> io::Result<()> {
    let line = &mut String::new();
    while r.read_line(line)? != 0 {
        w.write(line.to_uppercase().as_bytes())?;
    }
    Ok(())
}

Should we stay on Substack?

Other options include Notion, Squarespace, or a bespoke website. It would be a shame to make current subscribers—there are several of you!—switch platforms, or else for Deeply Nested to lose those subscribers. But Substack’s support, especially for technical blogs, is *ahem* lackluster.

Do we want to build a community?

Deeply Nested has always been a mix of posts about careers in software, technical details of software development itself, and short essays about random stuff I personally find interesting.

In the current job market, networking seems especially valuable, as well as building portfolios of work that can be shared online with prospective employers. (Even if you have tons of professional experience, you can’t always publicly share the work you’ve done.) Can Deeply Nested be a resource for helping people connect, personally and professionally? Or is it better off remaining a periodic deep dive/diatribe/pontification?

Regardless of where we publish, we might also expand to a new medium, such as a podcast, that would be more amenable to conversation than a comments section.

Please do share your thoughts on all of the above, or whatever else comes to mind. It’s great to see you again.

STEM trains the mind to model the world precisely and mathematically. We are tested and interviewed primarily on our understanding of such models; sometimes, to such an extent that we lose sight of the realities they abstract. The distinction between model and reality is nevertheless critical. Choosing the right mental model in any situation—and understanding that all models are inherently imperfect, just as no map entirely captures the terrain it represents—is a more valuable skill than the ability to wield any particular model effectively.

The universe is not only queerer than we suppose, but queerer than we can suppose.

—J. B. S. Haldane, Possible Worlds

Models are tools, and like all tools, their utility is situational. No single person has the luxury of unimpeded access to universal truth. The best we can do is to leverage whichever models are most likely to help us achieve specific goals.

When someone else applies a different model than one we’ve accepted, and thus reaches different conclusions than our own, it’s easy to suppose that one of us must be wrong. That disagreement can be frustrating. It’s helpful to remember that we rarely really know anything, especially when deciding the best way to solve a problem. The single most important trait distinguishing senior engineers from junior ones may well be the humility that comes from having been wrong many times before.

All models are wrong, but some are useful.

—Attributed to statistician George E. P. Box

Engineers, at least, routinely deal in space-time trade-offs, margins of error, mechanical tolerances, etc., whereas many people naively believe pure mathematics to be some kind of absolute truth rather than, like all forms of study, a mere representation of our intrinsically flawed understanding of a fundamentally unknowable world. Having “proven” something doesn’t mean it’s True with a capital T, only that we can’t see how it could not be true. We are imperfect beings though, and often miss things.

Kant, who was ten times more distant from Aristotle than we are from him, even held that nothing significant had been added to Aristotle’s views in the intervening two millennia.

In the last century, Aristotle’s reputation as a logician has undergone two remarkable reversals. The rise of modern formal logic following the work of Frege and Russell brought with it a recognition of the many serious limitations of Aristotle’s logic; today, very few would try to maintain that it is adequate as a basis for understanding science, mathematics, or even everyday reasoning.

—Stanford Encyclopedia of Philosophy, Aristotle’s Logic

None of this undercuts the usefulness or merit of scientific models. The medical and technological advances of the last few centuries probably would have been impossible without the scientific revolution. When you disagree with someone though—and can’t understand how they could be so boneheaded, especially in the face of so much evidence!—rather than thinking they must be stupid or evil, try to understand their mental model. And have the humility to realize that it may be no less valid than your own.

What I love about Advent is that it’s a good microcosm of real-world software development, which is something other coding challenges struggle (and generally fail) to model. Aside from the lack of teammates and end users, completing Advent in real time (one challenge per day) takes you on a journey not dissimilar from most professional coding.

The first puzzle is logically easiest, yet in some ways also the hardest. You have to decide what programming language you’ll use, how to lay out your repository, whether to publish your solutions online, and so on. It really feels like you’re at the start of something. The first day also gets you comfortable with the structure of the puzzles, each of which has two parts: You have to complete the first part to unlock the second part, which asks you a new question about the same input. For example, if Part 1 wants you to sum a list of numbers, Part 2 might ask for the maximum of those numbers. So, right away, you’re almost forced to factor your code into three pieces: code that’s useful in both Part 1 and Part 2 (such as an input parser, or hard-coded input values if you’re a killjoy), code that’s useful only in Part 1, and code that’s useful only in Part 2. What an ingenious way to create engineering tasks from simple riddles!

After the first day, the puzzles get gradually harder, often building on earlier puzzles in interesting ways. Again, you’re forced to either refactor or copy/paste code so that you can reuse the logic from earlier solutions in new ones. Sometimes you realize that the data model you used for one puzzle isn’t right for another, even when the two puzzles are semantically similar. The multiple perspectives you gain on the problem space (artificial though it may be) force you to think deeply about it, and give you the incredibly satisfying feeling of understanding it.

Somewhere around December 20th, the difficulty skyrockets. Puzzles go from something you can do before or after work, to something that takes all day, or even multiple days. (Some years, I finish these on time; and some years, I don’t.) Somehow, in three short weeks, you’ve gone from the ease and excitement of a greenfield project to the tension and stress of being up against a deadline. Not the simple deadline of a traditional coding challenge, but a deadline for something you’re invested in. Something you take pride in, and want to show off to other people. You may not have “users,” but you do have peers, and you care what they think. Not even hackathons inspire the sense of urgency that late-stage Advent does. This may sound terrible, but no more so than the climax of a great story, when the hero faces the most powerful villain, and we truly doesn’t know whether the hero will win or lose. It’s an exciting moment.

One other thing that Advent does well is to walk you through sample input for each puzzle, and give you the answer for that sample. This gives you something to test your own code against, which is invaluable for debugging. There’s a lesson to be learned here for actual engineering: If you’re building something to work in the big messy nasty real world, make sure you can also run it on something small and clean and well understood. It not only makes debugging easier, but is often critical to making debugging possible at all.

Kudos to Eric Wastl, the creator of Advent of Code. If you’re participating this year, feel free to share observations and code links here. And if you’ve never tried it before, but you love a good coding challenge, you’re in for a treat. This really is one of the best.

There are some obvious places to look for new-to-you ideas. Sabbaticals, newsletters, or even chatting about work in a social context can plant established ideas from one field in another where they may bloom afresh. Both sides of this pollination can be extremely satisfying: Finding a way to apply the wisdom of others in your own work, or seeing the gleam in someone’s eye when you explain what to you has long since become prosaic.

Ironically, the least obvious place that new ideas originate is from within your own organization, from folks ideating beyond their formal purview. (Flamin’ Hot Cheetos were not actually invented by a janitor, but wouldn’t it be great if they had been?) If, like most development teams, you use an issue tracking system, one of the best things you can do is to make sure everyone at the company can create new tickets in an incoming Triage column (assuming a Kanban-like dashboard in which columns represent stages of work). Periodically (say, once a week), folks with the authority to accept work on behalf of the team should go through any new arrivals in Triage, and respond to them thoughtfully, either adding the issue to the team’s backlog, or explaining why that’s not being done at this time.

If I had asked people what they wanted, they would have said faster horses.

—unknown, but attributed to Henry Ford

One of the saddest mistakes I see folks make with such “suggestion box” Triage columns is to delete or archive the tickets. Keep them! Keep them forever. You can certainly create a separate column—called Triaged, perhaps—but deleting suggestions is unnecessary, and sends the message that you do not value people’s ideas. Moreover, it implies that you are kiboshing their suggestion forever, which you’ll admit is an awfully long time. As long as you keep the triaged tickets around:

• You can refer back to them for the rationale of why an idea was not pursued. This is surprising valuable. All too often, “new” product/eng ideas are greeted by a few old-timers who vaguely recall a similar idea already having been rejected, but cannot recall any compelling reason why

• Suggestions that are rejected for lack of resources to implement them can be revisited later. Some make great intern projects, or chances for new hires to get their feet wet. Often, when an idea rattles around the back of the group’s head long enough, they realize it wouldn’t be all that difficult after all.

• Team members who care deeply about a rejected idea can move tickets from Triaged back to Triage, so that they will be reconsidered. You can cap the frequency of such reversion; say, to once or twice a year. If somebody cares that much about it, maybe it really is worth reconsideration.

• The poobahs who accept or reject work from the Triage column may choose to set aside one day each year for reconsidering rejected tickets. As times and people change, so do priorities and opinions. Think of it as the Big Block of Cheese Day from The West Wing. Of all the seemingly unworkable ideas, some will turn out to be like the ice-shipping scheme that earns Richard Pryor an inadvertent fortune in Brewster’s Millions.

While not every idea is worth pursuing, every idea that someone took the time to document matters to somebody, and should be treated accordingly. Sincerely welcoming ideas from all directions can help you find diamonds in the rough, and humility can help you recognize those diamonds even if they aren’t obvious at first sight. Be open, be thoughtful, and remember that sometimes, long shots pay off.

The past few weeks have entailed brutal layoffs by well-known tech companies. The treatment of Twitter’s staff has been especially bizarre, and speculation continues about whether (and how long) the company can continue operation having decimated1 its headcount.

Email from Elon to the engineering team: "Anyone who can actually write software, please report to the 10th floor at 2pm today. Before doing so, please email me a bullet point summary of what your code commits have achieved in the past 6 months." Elon Musk is also asking for up 10 screenshots of the "most salient lines of code" from Twitter engineers. . . . Yesterday, Twitter told employees the offices were closed and badge access was suspended until Monday. Employees are extremely confused.

—Zoë Schiffer, Managing Editor at Platformer

I once worked at a company where employees were treated more like code to be debugged, or perhaps feral dogs to be domesticated, than professionals to be inspired. Behavior that was not “extremely hardcore” (as Twitter’s new owner puts it), or was seen as disloyal, could severely limit one’s chances of career advancement. Employees were seen more as potential liabilities than vital assets. A wise coworker observed to me that the company’s most valuable resources left at the end of every day, and management didn’t even realize it.

One asinine speech by a senior manager (which I did not witness first hand) implied that product quality should be deprioritized, because maintaining the high bar (and correspondingly excellent repuation) that this company had earned across the industry was costly. Not prohibitively so; merely an expense that could be reduced, thus netting more money (in the short term). When someone objected, asking semi-rhetorically whether we really wanted to reduce our product from a (metaphorical) luxury car to a jalopy, the manager replied: “You can make a lot of money selling jalopies.”

Another coworker once told me something I didn’t understand at the time, but that rings truer and truer to me: Companies are made of people. They’re also made of paperwork and legal entities and sometimes brick and mortar, and the people who start a company aren’t always the same ones who make it tick in later years, but the people are always the parts that matter most. People run the company, and in many ways, those who control the company exercise control over the lives of other people. That’s a responsibility that some executives don’t take seriously enough.

This is a tough time for tech workers, but if you’re a junior engineer trying to enter industry even as hiring is slowing and the market is flooded with laid-off competition, or you’re a senior who’s back on the market after decades of stability, remember that opportunities are still out there, and some company—some people—really need you. You just may not know them yet.

Piezoelectric materials create electricity when you squeeze them. They're like physical functions that transform pressure into voltage. We use them in doorbells and microphones, and lots of other stuff. There’s also a reverse piezoelectric effect whereby some objects, when exposed to an electric field, change their shape or size. Reverse piezoelectrics are used in ultrasound: A rapidly oscillating sheet of reverse piezoelectric ceramic acts as a “speaker” to generate ultrasound waves. It is weird that we can make materials dance for us by subjecting them to fields of invisible energy.

Thermoelectric materials turn temperature differentials into electricity (the Seebeck effect) or vice versa (the Peltier effect). If you're in a warm room in cold weather, and you stick one end of a thermoelectric pole out the window, you get free electricity (except insofar as you're paying to heat that room in the first place). In practice, we use thermoelectrics to build heat pumps for refrigeration; but research is underway toward reclaiming energy from otherwise wasted heat.

Semiconductors either prevent or allow the flow of electricity, depending on their environment. Most materials are either electrical insulators (like the rubber sheath of your laptop’s power cable) or conductors (like the metal wire inside that sheath). Semiconductors are like insulators, except that when we put them in certain electric fields, they become conductors. Computer microprocessors use semiconductors in billions of tiny switches, whereby the state of each switch (either insulating or conducting) controls other switches, and so on, almost the way billions of neurons in the human brain interact to produce higher order phenomena like thought.

Effects so simple they might be mistaken for mere novelties can be combined to create awe-inspiring technology. Consider the mind-blowing example of floppy capacitors. (Floppy isn't a technical term, but an accurate one.) Most modern electronics are made deliberately rigid, so that their size, shape, and behavior are consistent even if you shake them. (If you're old enough to remember portable CD players, you’ll appreciate the fact that modern music players don't skip as you jog.) But it is possible to make floppy electronics that are easily deformed by acceleration. In particular, the electrical properties of simple devices called capacitors change as they're deformed. You know the visceral feeling you get when an elevator starts moving or comes to a stop? Floppy capacitors “feel” it too.

The change in floppy capacitors’ behavior as they get jerked around has been leveraged to create accelerometers. When your smartwatch tells you it's time to get off your duff and stand up for a minute, your watch is using a built-in accelerometer to tell when you last stood up or sat down.

Accelerometers are key components of automobile airbags. Microcontrollers monitor the output of accelerometers, checking for sudden deceleration, and constantly sending a single bit of information (either true or false) to the airbag deployment system: Are we crashing right now? The United States Department of Transportation estimates that over a recent 30-year period, airbags saved the lives of over 50,000 people. All thanks to the magnificent insight that floppy electronics could be more than a novelty. (Inventor Allen Breed earned his degree in Mechanical Engineering after patenting the modern airbag.)

There’s no telling what we'll discover next—what bizarre properties of matter and energy, or what novel uses we'll put them to. Whatever they may be, engineers will be the people who turn the possible into the practical, and the interesting into the invaluable.

Spaghetti

The term “computer program” is misleading, because it implies order. Computer programs are lists of instructions—things for the computer to do—that aren’t necessarily executed in list order. Computers support “jump” (or “branch” or “go to”) instructions that transfer control to different parts of the program, sometimes depending on factors that can’t be known in advance. Programs that are hard to read because of all the jumping around are called “spaghetti code.”

You still see goto in low-level C code sometimes for resource cleanup. For example, a function that tries to copy a file might do the following:

1. Initialize a “success” variable with the value false.

2. Try to open the old file for reading.

3. If (2) failed, go to (10).

4. Try to open the new file for writing.

5. If (4) failed, go to (9).

6. Try reading the contents of the old file, and writing them to the new file.

7. If (6) succeeded, assign the success variable the value “true.”

8. Close the new file.

9. Close the old file.

10. Return the success variable’s value to the caller.

You’ll also see a glorified version of this in garbage collected languages, using defer statements or with or try blocks instead of goto. To do any better than per-call-site resource management, you need either C++/Rust style destructors, or a framework that supports lifecycle callbacks (methods or hooks).

Since the 1950s (and arguably culminating with Dijkstra’s 1968 opinion piece Go To Statement Considered Harmful), goto statements have been considered gauche. In polite company, jumps are used only as part of common patterns like loops and function calls, collectively call structured programming.

When reading well-structured code, you can often skip over details. For example, if you see a function call like factorial(5), you probably don’t have to look at the implementation of the factorial function, because you know what it’s meant to do. That kind of abstraction is critically important for software (and development teams) to scale, because it’s not practical for someone reading code to have to follow all the jumps. There’s an equally important but underappreciated caveat to this kind of structured programming though: Abstraction should never prevent the reader from following the jumps if they want to.

It’s in vogue now to use Dependency Injection (DI), message buses/brokers/queues, and other heavyweight techniques to decouple various parts of complex systems. Tragically, most of these systems throw the baby out with the bathwater: They reason that because you ideally shouldn’t have to look past an abstraction, you should be actively prevented from doing so; or at least, supporting removal of the abstraction is not seen as a priority. Such thinking ignores the fact that abstractions are inherently imperfect. We won’t belabor that point here, because Joel Spolsky has already done a great job coining the The Law of Leaky Abstractions.

Lasagna

Sometimes you see software structured into layers, or tiers if they’re separate programs (such as a database, microservice, and web-based user interface). This kind of structure is occasionally called lasagna.

When you see a Data Access Object (DAO) or other layer-oriented abstraction, make sure you can understand how each layer is actually implemented. At least once, try following a request all the way down the call stack, and the response all the way back up. If you are building such a system, be as transparent (or translucent) as you can about it, so that your layers are easy to reason about when debugging, or assessing cost or performance.

Ravioli

Instead of (or in addition to) layers, software is sometimes structured as bundles of state called objects. These are objects in the sense of Object Oriented Programming (OOP), and while they are generally implemented by objects in the sense we discussed in Variables, they’re a more abstract idea.

The same principle applies to objects (in the OOP sense) as to layers: Don’t presume that an abstract interface is the only thing your clients should have access to. If you can reasonably let them know how that interface is implemented, you should do so. If you’re using someone else’s objects (or classes), take a peek at the source code (if available) and get a sense of how they actually work. Incidentally: Plain Old Data (POD), as opposed to fancy objects with properties and methods and access control, are underrated.

There’s a fallacy gaining traction these days, along the lines of: “Never expose any implementation details, because if you do then someone will come to rely on them, and then you’ll never be able to change your implementation.” All caller/callee relationships are subject to change. Being a service provider doesn’t make you a serf. Your clients are not your enemies. (If your clients are actively hostile, drop them.) The best way to provide value is to clearly distinguish aspects of your system that will remain stable from those that may not, letting your customers lift the hood and see how your machinery works. The possibility that they’ll somehow void the warranty isn’t a good reason to keep them in the dark.

There is surely too much hate in the world. But you know we can all hate together? Bugs. Estimates of the annual worldwide cost of software defects vary, but are generally on the order of trillions of US dollars. The work of finding and fixing bugs is estimated to comprise more than 60% of total software development costs. Formal studies probably underestimate the real cost by a wide margin, because they focus on quantities that can be accounted directly.

Undergraduate engineering coursework includes study of disasters that have resulted in loss of life, including watching video of the Hindenburg and Challenger explosions. Any pecuniary value assigned to such tragedies would be grotesquely inadequate.

🪤 Computer science and bootcamp students are generally not made to study past engineering disasters; yet we continue hiring these folks into engineering roles, despite (or perhaps explaining) the fact that more and more people die because of bugs. (I’m not blaming the students, or even the schools. I’m blaming industry for assigning engineering work to people who were not formally trained as engineers.)

Ripple effects that are hard to measure may in fact be dominant. For want of a bug fix, the account was lost; for want of revenue, earnings were missed; for want of investor confidence, equity was devalued; for want of incentive stock options, talent couldn’t be hired or retained; and for want of an effective team, the company was uncompetitive. The link between product quality and a company’s ability to survive and thrive should be obvious, but it’s hard to quantify, much less prove.

Let’s end them

Humanity has already eradicated (or nearly so) multiple parasites, including smallpox and polio, and (notwithstanding the anti-vax movement) we’re getting better and better at it. In fact, we’re so good at driving species to extinction that we frequently do so accidentally, primarily by killing organisms directly—what the American Museum of Natural History euphemistically calls “overharvesting”—or by destroying habitats.

Software bugs are no less susceptible to extinction than living creatures, and in principle have the same vulnerabilities.

Loss of habitat

The easiest place for bugs to hide is anywhere you have multiple sources of truth. Caches are probably the worst offenders. Caching data that may change over time is obviously a deal with the devil, as you choose to accept the wrong answer now rather than the right answer later.

Suppose a video game saves state on a server; but also caches it in local storage so that the user can start playing right away when they open the game, rather than waiting for state to be retrieved from the server. If the user ever plays on a second device, and then comes back to the first device, the local storage may be out of date. Any gameplay based on that cached state may conflict with changes from the second device. Oops, your player’s health wasn’t supposed to be that high, because they took damage while playing on the second machine. Reconciling differences like these can be tricky, and is a common source of bugs.

Conflicting truths can also come from parts of a codebase that nominally implement the same logic. Suppose we use two separate routines to sort people by height, and the first routine orders Alice, Bob, Charlie, and Dana thus:

The second routine produces a different result:

Both results are correct, because Bob and Charlie happen to be the same height. Yet such gratuitously different results can make software misbehave. We discussed a similar issue in Traversal Order.

The ideal solution to bugs like these is to build systems that are correct by construction. Having a Single Source of Truth (SSOT) guarantees that discrepancies cannot arise. This isn’t usually hard to do, but it requires a certain mindset to even recognize the problem: Redundant code and data storage are potential bug nests.

Never go to sea with two chronometers; take one or three.

—via Fred Brooks, The Mythical Man Month

“Overharvesting”

This may sound obvious, but another critical step toward eradicating bugs is to find and fix them. This doesn’t always happen, mainly because of perceived trade-offs.

Whether to prioritize bug fixes (or other quality measures) over other work is a business decision: The upside of a new feature may outweigh the perceived potential cost of quality issues. However, decision makers often mistakenly think bugs are stand-alone things. That’s like assuming the cockroach you just saw skitter across the restaurant floor lives alone. Bugs live in colonies, and every one you let live is bound to spawn others.

“The only good bug is a dead bug.”

—Edward Neumeier, Starship Troopers

Question #5 on The Joel Test is: Do you fix bugs before writing new code? That’s a pretty good litmus test for distinguishing great teams from mediocre ones. If we really want to end bugs, it’s not enough to “harvest” the obvious, urgent problems. We have to overharvest if we’re to avoid their insidious second-order effects; essentially, to stop them from multiplying.

Eliminating bugs entirely is an ambitious goal, but may be achievable. The first step is to recognize their real cost to both individual initiatives and the world at large, which is frankly staggering. The second step is to destroy (or at least stop constructing) places for bugs to hide; mainly by identifying and protecting SSOTs, and otherwise prioritizing code that is correct by construction. The third step is to hunt and kill them without ruth or delay. Finally, we must prioritize quality in sales and marketing. Customers demand lots of different things, from product features to favorable contract terms to white glove treatment from account managers; but collectively, as an industry, we have to make quality the focus of our shared conversation.

1. Lousy managers use the word to justify reorgs and layoffs.

2. People like to laugh at words (and anything else) they don’t understand.

There’s plenty of jargon we can mock. The phrases “best practices” and “think outside the box” come to mind. But it would be a shame to lose the word synergy, which represents such an important concept.

The idea of synergy is that when you get the right people together, they can collectively accomplish more than they could separately. We’re not talking about the happenstance benefits of serendipitous confluence; we’re talking about deliberate, orchestrated commingling of skills and perspectives, like mashing up two melodies to create surprising and wonderful harmonies.

syn·er·gy (sĭn’ər-jē)
n. pl. syn·er·gies

1. The interaction of two or more agents or forces so that their combined effect is greater than the sum of their individual effects.

2. Cooperative interaction among groups, especially among the acquired subsidiaries or merged parts of a corporation, that creates an enhanced combined effect.

—American Heritage Dictionary

Examples abound in both nature and industry. The most delicious cup of coffee comprises chemicals that taste disgusting on their own (or if not properly balanced). We’ve all met couples that seem like either would be miserable, but both are a delight. Synergy can make the worthless worthwhile, the good great, and the great transcendant.

There’s no such thing as a free lunch.

—Someone mistaken. Paul Mallon, maybe.

The most obvious examples are economies of scale. If you’re building a web service that needs a server, and I’m building an unrelated web service that also needs a server, maybe we can get together and not have to pay for two separate servers. If there are significant per-server costs, and using a single multi-tenant server won’t compromise security or performance, we might be able to save a lot of money and hassle by sharing. That’s synergy in its most mundane form.

If you’ll bear with me through a real example that’s a good metaphor for how surprising synergy can be, I’d like to tell you about a cool technology used in the manufacture of high-end electronic devices. Microprocessors are manufactured in a fascinating way. They’re designed as a set of horizontal layers; that is, 2D slices of the physical 3D object that will eventually be sold to you as the single most expensive part of your phone or desktop computer. Each layer is made of a different material. Professional mask designers decide where, within each layer, the material should go, and where it should not.

At the fab (where processors are “fabricated”), silicon wafers are laid out flat, and covered with the mask for the lowest layer. Particles of the material for that layer (sometimes obtained by heating a chunk of the material until it evaporates) are released into the air above the wafers, slowly settling onto them. The mask is then removed, leaving the material on the wafer only in places where the mask had holes. The process is called deposition. It’s repeated for each layer. Modern processors have dozens of layers.

The feature size of a semiconductor fabrication process is the smallest distance we can realistically control. It’s some fraction of the width of a single transistor. The feature size of current generation processors is about 5 nm; that is, 0.0000005% of a meter. That’s so small that the holes in the masks have to be cut with laser beams. In fact, it’s even smaller than we can cut with a laser beam, because a laser with a wavelength that short (and thus a high frequency) would be too powerful for practical use. So, how do we cut such tiny mask holes? We use two lasers whose waves are almost diametrically out of phase with each other, such that they mostly cancel each other out, much as noise cancelling headphones cancel ambient sound. But they don’t completely cancel each other out; and the little bit of difference is so small that we can use it to cut 5nm holes in masks for microprocessors, making modern digital electronics possible.

Thunder only happens when it’s raining.

—Fleetwood Mac, Dreams

Synergy isn’t merely putting two things together, and getting the sum of their parts; nor even getting more than the sum of their parts, in any naïve sense. Combining two lasers doesn’t necessarily give you one doubly powerful laser. Often, it’s precisely the tension between the composed parts (or people) in a system that makes it special.

Part of great project management is recognizing opportunities for synergy, and turning the workaday into something unique. If you’re able to achieve synergy, and get it moving in the right direction, then you have captured lightning in a bottle. Just maybe keep your voice down when you speak the word “synergy” aloud, because not everyone takes kindly to it.

The great humorist, while not perhaps very robust, is in the best of health. He said: “. . . The report of my death was an exaggeration.”

—Frank Marshall White, New York Journal, “Mark Twain Amused;” via Mental Floss

First, let’s get a little perspective. The practice of encoding precise instructions for transforming available inputs into desired outputs long predates the electronic computer. Knitting patterns and sheet music are “code” by any reasonable definition; and arguably, so are many cooking recipes. The 19th-century advent of player pianos did not eliminate live music, any more than electronic synthesizers did a hundred years later. Mass production of artificial foodstuffs has not ended the art of cooking—some of the best chefs are now even celebrities—nor have people stopped creating (and appreciating) hand-knit socks and sweaters despite the plethora of commonplace, machine-made alternatives.

Technology has already eliminated any number of specific job roles (or nearly so), but that’s nothing new. Somebody in every ancient tribe was probably The Best at starting fires by banging rocks together. Not to be flippant about this: It wasn’t long ago that robots put most automotive assembly line workers out of a job. Now long-haul trucking seems destined to be replaced by self-driving vehicles. People sometimes draw a distinction between blue and white collar work, but that misses the point: White collar jobs may absolutely be replaced by AI. Stenography is certainly white-collar work, but it’s not hard to imagine it being replaced by voice-to-text applications.

By 1977, Bricklin felt that programming was getting to be so easy that he would be out of a job soon.

—Retro Tech Bytes - The Story of VisiCalc

We can acknowledge the potential of technology to disrupt society, while also recognizing that as demand for particular skills waxes and wanes, it makes sense to relegate the most robotic tasks to machines. That’s largely what programming is about. More than that though, programming is about raising the level of abstraction; not only helping us do things more efficiently, but making what used to seem impossible a real part of our everyday lives. AI is the next rung on the ladder of abstraction—which, as should be clear by now, has no upper limit.

Huge parts of what coders now do manually may well be done by AI soon. Consider the ramifications of a similarly staccato step up the abstraction ladder: the leap from machine code to high-level programming languages in the 1950s. The number of jobs available to programmers working directly in machine code declined precipitously, but with two key caveats:

1. Programming didn’t go away. Demand for professional programmers actually increased, because more powerful tools amplified the productivity (and thus the value) of each individual programmer.

2. As low-level knowledge became more rare, it developed into a specialization for which professionals even now charge a premium. Processors did not stop adding new instruction sets (floating point, SIMD, GPU/matrix operations, etc.), nor did CPU-intensive applications stop vying for the most efficient use of hardware.

Even folks who concede point 2—“OK sure, there will always be specialists”—sometimes have a hard time understanding point 1. What seems to trip them up is the idée fixe of economics as a zero-sum game. None of this is zero-sum. The amount of work to be done is unbounded, and so is the reward. Even if the demands society placed on the software industry were to stagnate, it would be naive to think there would be less demand for professional programmers. But of course, demand does not stagnate, it escalates.

If you’re unconvinced—or, if you’re having trouble retaining engineers because they’re unconvinced—see also the very first Deeply Nested post: Is Coding a Dead End Job?

New programmers are taught about variables on day 1, but only through vague analogies like “boxes you can put things in.” Even experienced programmers sometimes have no more than an intuitive understanding of what the word “variable” means. Let’s nerd out a little this week, and take a deeper look at what a variable is.

Mathematics

In mathematics, a variable is a name bound to a value. A statement like x = 5 binds the name x to the value 5. A “name” is usually a letter, word, or short phrase. Sometimes somebody gets wacky and declares a variable whose name is a shape or a cartoon character or something. A “value” can be any timeless concept, like the number 5 or the color blue.

A variable may represent different values in different situations. For example, we can define functions that map argument values to result values, using variables called parameters to represent arguments abstractly. In the function definition f(x) = x + 1, the variable x is a parameter. In the function calls f(3) and f(5), x is bound to the arguments 3 and 5, respectively.

Values never, ever change. A value is forever. Variables that always represent the same value, like π for the ratio of a circle’s circumference to its diameter, are sometimes called constants. In fact, if you call π a variable instead of a constant, you’re likely to annoy a lot of pedants; but in your heart of hearts, know that symbolic constants like π are just variables that don’t vary.

Rose is a rose is a rose is a rose.

—Gertrude Stein, Sacred Emily

Code

In computer programming, a variable is a name bound to an object, which in turn represents a value. This extra level of indirection can be tough to explain, which is why it’s rarely taught to newcomers. Nevertheless, these concepts are fundamental to software engineering, and you ought to understand them if you do this stuff professionally.

An “object” is a region of memory that represents a value. Computer “memory” is a big list of numbers, and a “region” is a contiguous sublist of memory. Modern computers can store billions of small numbers in their Random Access Memory (RAM) all at the same time. Of course, computer software needs to represent lots of different kinds of values, not only small numbers. That’s where types come in.

Types

A “type” is a system for storing values in memory, or interpreting values previously stored there. A region of computer memory (i.e., a list of numbers) on its own isn’t very useful, because you can’t tell what value it represents unless you also know its type.

💡Types (sometimes called “classes”) are often explained to new programmers as being blueprints for objects. That’s not a bad analogy, but it breaks down quickly as you dig deeper into type theory. One rarely hears about “blueprint contravariance” or “higher kinded blueprints.”

Types for representing common kinds of values, such as colors or strings of text, often use standard encodings to map values into numbers. For example, the list of numbers [72, 105, 0] represents an ugly shade of green as an RGB encoded color, but the word Hi when interpreted as UTF-8 encoded text.

None of this is specific to any single programming language, but the C standard offers a pretty good definition of object:

object
region of data storage in the execution environment, the contents of which can represent values.

Note 1 to entry: When referenced, an object can be interpreted as having a particular type.

— N2596 (working draft), 3.15

Static vs. dynamic types

You might think that when a program stores a value in memory (i.e., in an object), it also has to store some type information, so that it doesn’t forget what kind of value was stored. Types used that way are described as dynamic.

Some languages also support static types, which are tracked through analysis of the source code. This is arguably the best of all possible worlds. We get the safety of knowing whether we got our types straight before the program even runs (!), and without the performance overhead of storing and retrieving dynamic type information.

The chief shortcoming of static types is that sometimes, we won’t know what kinds of values we need until the program is already running. For that reason, even “statically typed” languages also have to support dynamic types. (C in particular does so grudgingly, but most newer languages are less judgy about it.)

Variables having static vs. dynamic types are sometimes said to be early or late binding, respectively.

Object footprints

Part of the definition of an object is that its region is contiguous, meaning all the numbers are stored next to each other in memory. The contiguous region is called the object’s footprint. There are two caveats you should be aware of: The footprint may contain padding that does not store any part of the object’s value, and pointers to other regions that do.

Padding

Indexes into memory are called addresses, and they refer to individual bytes, each big enough to store one small number; typically, an integer from 0 through 255. Computers can work with some types efficiently only when their values are stored at addresses divisible by certain numbers (usually 2, 4, or 8). Strange but true. Such types are said to have alignment requirements. So, sometimes an object’s footprint will include meaningless bytes as “padding,” merely so that the next meaningful part of the value lives at a suitably aligned address.

Pointers

A “pointer” is an object whose value is a memory address. A complex object’s footprint might include pointers so that it can share data with other objects, or so that data can live in a specific part of memory; for example, storing huge chunks of data on the heap (which has plenty of room) even when the object’s footprint lives on the stack (which does not).

Move semantics

It’s sometimes useful to move objects around in memory. For example, maybe we’ve stored the color blue at address 1024, but now we want to store it at address 512 instead. Computer hardware doesn’t really support move semantics, so moves are usually implemented as copies. Move semantics are yet another reason for an object’s footprint to contain pointers: When we move an object, it’s usually cheaper to copy a pointer than to copy all the data the pointer points to.

Garbage collected languages move objects all the time. When you hear about mark and sweep GC, remember that “sweep” is a euphemism for “copy.” This is why it’s so hard to get an object’s address (or any unique identifier) in most GC languages: The language designers don’t want you to think of the object as living at any particular address, because it might get moved.

C++ has an, um, interesting approach to this: Rather than moving the object, you move the value from one object to another. The original object is “move copied,” and left in some technically valid but unpredictable state. The introduction of move semantics to C++ in 2011 was a game changer in terms of performance and expressiveness, but it also added one heck of a foot gun to a language already rich in weapons for shooting oneself in the foot.

Exotic variables

Reference types; or, “Dammit, Java!”

If the value we want to represent happens to be a small number, then it’s easy to imagine how we’d store it in memory, which is (after all) a list of numbers. Most programming languages even include built-in types (sometimes called primitive or intrinsic types) for storing such values. User Defined Types (UDTs), on the other hand, get different levels of support in different languages.

C++ bends over backward to give UDTs the same level of support as primitive types. The Design and Evolution of C++, penned by the language’s creator, describes how making UDTs first class entities was an overriding design goal.

At the other end of the spectrum, Java treats primitive and user-defined types completely differently. Java primitives are stored entirely within the object’s footprint, whereas UDTs live entirely outside it. The nomenclature is confusing: Java provides primitive types like int to represent small numbers, and (like all programming languages) stores them in regions of memory; but weirdly, does not consider those regions “objects.” Instead, only variables of reference types are bound to (what Java considers) objects. Ironically, variables of reference types cannot be bound directly to objects. Instead, they may be bound only to references to the actual objects.

In the Java code below, copying a primitive integer and then mutating the copy has no effect on the original integer; which should make perfect sense, because why would manipulating a copy affect the original? These are variables, not voodoo dolls. Copying a variable of reference type and then manipulating it does affect the original though:

Java variables of reference types can also be bound to no object at all, in which case they’re considered null. Null values can be useful when used selectively, as is common in Golang (where null is called nil); but unlike Go, Java asininely makes all variables of non-primitive types nullable, whether you want them to be or not. Golang is proof that a GC language needn’t be burdened by this reference/null stupidity.

🪤 Several languages, including C++ and Rust as well as Java, use the term “reference” to mean a pointer that’s semantically not considered an object in its own right. In other words: Even in non-GC languages, you can’t take the address of a reference (even though it really does live at some address in memory, barring optimizations like register-only variables). If you try, you’ll only get the address of the referenced object; that is, the address that the pointer points to.

The bizarreness of Java’s object model is a historical relic of a mid-1990s fever dream through which Object Oriented Programming (OOP) was seen as some kind of pinnacle of human achievement. Java-style OOP mandates inappropriate conflation of inheritance with polymorphism, as well as other pretentious garbage that’s hard to support efficiently on outré types like, you know, integers.

Lazy variables

Some languages support lazy variables, introducing yet another level of indirection. Rather than binding names to objects that represent values encoded by types (whew!), lazy variables bind names to objects called thunks. Thunks represent whatever computation is necessary to encode the desired value, and delay that computation until the value is actually used.

For example, in this Scala example, x is semantically bound to the value 1; but actually, it’s bound to a thunk that prints a string and then returns the value 1:

Scala papers over some of the more broken parts of Java by unifying primitive types and UDTs under a single base Any type, and blurring the distinction by making collections immutable by default. However, Scala also introduces its own academic nonsense, such as referring to objects as values in official documentation. Objects are not values, even if they’re immutable. Objects represent values, but unlike pure values, objects occupy real space and consume real resources in real computers here in the actual, non-academic world where we humans live.

Function objects

Sometimes, you’ll hear a new programmer say they’re “calling” a variable, when they mean they’re using it. That’s a good habit to break, because it gets confusing when variables can be bound to function objects—that is, objects representing functions. For example, suppose we use such a variable to register a callback in JavaScript:

const handleClick = function () {
  console.log("hello");
};

button.addEventListener('click', handleClick);

We’ve used handleClick here, but we haven’t called it. It won’t be called unless someone clicks the button.

Unsized types

In most type systems, all objects of a given type have the same footprint size. Even types like strings and vectors that include pointers to variable amounts of data have some fixed footprint size, typically big enough to hold the pointer and some metadata. That’s not always good enough: Sometimes, objects of a particular type should be allowed to have different sizes. In most languages, working with such types requires the programmer to be exceedingly careful.

Parameter string starts immediately after this object. Be careful to add padding after string to ensure correct alignment of subsequent dm_target_spec.

—Linux kernel, struct dm_target_spec (dm-ioctl.h)

Rust

We need to talk about Rust for a minute, because Rust gets so much of this stuff staggeringly right.

Unified syntax for early and late binding variables

Rust unifies the syntax for static and dynamic bindings by adding a dyn keyword. I cannot overstate how cool this is. It reflects a profound understanding not only of the Computer Science (CS) involved, but of the actual day-to-day reality of software engineering. For example, suppose we have some abstract interface:

trait MyInterface {
    fn do_something(&self);
}

Here’s a function that accepts a reference to any object whose type implements that interface, so long as the type is known statically (i.e., at compile time):

fn f(x: &impl MyInterface) {
    x.do_something();
}

Now, suppose we realize that oops, we won’t know the type until runtime. Literally the only change we need is to replace impl with dyn:

fn f(x: &dyn MyInterface) {
    x.do_something();
}

You can even have some functions take the type statically, and others dynamically. The decision is made where the variables are defined, not where the type is. So, so cool.

Questionably sized types

In most languages, properly supporting types whose size may vary is tough. How can the language support even simple operations like copying an object, if it doesn’t know where in memory the object ends? Rust works around this issue with spectacular simplicitly: It lets you declare questionably sized (?Sized) types that support only safe operations, and trigger compile-time errors for any operation that the language cannot safely support.

Obviating padding

We discussed Rust’s flexible object layouts in Type Systems and Alignment.

Safe move semantics

Rust supports roughly the same move functionality as C++, but more efficiently, and without any foot guns. When you move an object in Rust, you’re semantically moving it out of the variable (which is now bound to the wrong region of memory), rather than leaving the variable bound to a zombie object (as in C++) or requiring the variable to go through a reference (as in Java). If you try to use a moved-from variable, the compiler reports an error. (There’s a loophole for primitive types, or indeed any types that support bitwise copy, so something like let x = 42; let y = x; doesn’t sacrifice access to x.)

Everybody got that?

There’s a lot more to understand if you want to be a systems programmer, such as volatility and atomicity of variables that are accessed by multiple threads, but this is already a long post for what’s usually passed off as a simple topic. Congratulations on reading this far, and please do leave a comment if you’d care to guide further discussion.

Suppose we’re responsible for a mission-critical service; that is, a computer program that’s supposed to keep running all the time so it can respond to important events like client requests. Nothing in this world is perfect, so there’s always a chance the service will crash, causing downtime for our system and our business. Hardware failures and software bugs (especially race conditions) are the usual culprits, but let’s assume that in any given second of time, there’s very little chance—say, one in a million—that our service will crash. In other words, if our service is humming along happily, there’s a 99.9999% chance it will continue to do so for at least the next second. Let’s call that chance P, and the one-in-a-million chance of failure Q.

$ node
Welcome to Node.js v16.17.0.
Type ".help" for more information.
> Q = 1e-6
0.000001
> P = 1 - Q
0.999999

The odds of failure compound geometrically. For example, the chance that our service will stay up for at least the next three seconds is the product of the chance that it won’t crash in the next second, nor in the second after that, nor the one after that: P * P * P, or P**3.

> P ** 3
0.999997000003

In general, the chance that a happy service will remain so for >=N seconds is P**N. The chance that our service will stay up for at least a minute is P**60, for an hour P**3600, etc. So, how long will it be before our service is likely to crash? What’s the time horizon beyond which the chance of failure exceeds 50%? Not long, as it turns out, because geometric growth is downright nasty.

> Math.log(0.50) / Math.log(P) / 3600 / 24
8.022532800536636

Eight days. That’s it. If you run a single-instance monolithic service in production, you’re liable to have a complete outage every couple of weeks. The one-in-a-million Q value we’ve been using might charitably be called “imprecise,” but this jibes with real-world experience: Too many small companies deal with random outages almost every week. These outages take a harrowing toll not only on productivity, but on the quality of life of the engineering and product support teams, including client-facing staff like Account Managers. People shrug off this absurd burden as though it’s inevitable (“Startup life, am I right?”) and euphemize it in job postings as a “fast-paced environment,” but it’s truly an awful way to live. What’s more, it’s easily avoided.

The first and most obvious approach—and the only one we’ll cover in this post—is to run two instances of every service. Any more than two is gravy.

🪤 If you’re an engineer who has worked at large Software as a Service (SaaS) companies, then advice like “have redundancy in prod” may sound utterly obvious. But if ever you start bopping around startups as an advisor or contractor, you will be stunned at how common these fault-intolerant monoliths are, mostly because the tech teams don’t understand the trade-offs involved.

Because we’re discussing random crashes, the odds of one instance crashing are independent of the other instance. We can thus replace Q with Q*Q in our formulae above, making our new probability of surviving the next second 1 - Q*Q. Let’s call that P2:

> P2 = 1 - Q*Q
0.999999999999

Now how long can we expect at least one instance of our service to stay up?

> Math.log(0.50) / Math.log(P2) / 60 / 60 / 24
8022714.288272493
> _ / 365.25
21964.9946290828

Almost 22,000 years. Individual service instances will still crash as often as before, but those crashes won’t cause major outages, because both instances are extremely unlikely to crash at the same time. (One could crash while the other is already down, but that chance becomes minimal if we address crashes promptly—even if all we do is restart the crashed instance.) We still have to deal with the crashes, but they’re no longer oh-my-god all-hands-on-deck emergencies, and they don’t impact users at all.

Productivity and quality of life improve tremendously if we run more than one instance, especially if our service is a little flaky to begin with. Going multi-instance does impose a couple of architectural requirements though:

1. We need a load balancer (LB) like HAProxy to automatically route traffic to a healthy instance. Our service also needs a health check endpoint, so the LB can tell when an instance isn’t feeling well. (But if we don’t already have a health check, how do we even know when we’re having an outage? Wait for customers to complain? <shiver>)

2. Our service instances must be stateless. The system as a whole needn’t be stateless, and can keep whatever database (DB) it already uses. But now, it’s especially important that we not keep any state where only one instance can access it, such as in-memory variables.
   People sometimes try to skirt this requirement using “sticky sessions,” meaning they ensure that consecutive requests from a single user are always routed to the same instance. Don’t do that. It complicates the LB config, boots users out of their sessions whenever an instance crashes, and normalizes instance-local state that may not even be session-specific. Put your data in a DB, or in shared “data structure servers” like Redis.

In upcoming Deeply Nested posts, we’ll discuss further use of redundancy to improve uptime, and to gain other benefits like economies of scale. In the meantime, please do share relevant thoughts or experience (or horror stories!) in the comments. And here’s a question for you: Going forward, should system architecture posts continue discussing high-level issues like redundancy; dive deeper into topics like orchestration and containerization; or get into the nitty gritty of how to configure particular tools like Kubernetes and Docker?

We are not nouns. We are verbs.

—Stephen Fry

Of course, we pick up some skills naturally on the job. But given that you’re the sort of person who reads a Software Engineering blog, you probably do more than that. How do you decide what to study? Tools, languages, frameworks, paradigms… What, in the interest of career growth or edification, shall you learn? The decision can be daunting, because time is finite. What if you focus on the wrong thing?

I really don’t know. I really don’t know what to do.

—The Rolling Stones, What to Do

Most of the risk in studying the “wrong” thing is minor. Even if you find you’ve spent some time suboptimally, is that so bad? If you learned anything at all, then your time wasn’t entirely squandered, even if not all of it was exactly useful. Another concern is that you’ll study to get a job (or a better job), but that you’ll somehow accidentally acquire only skills for which there’s no demand. Rest assured that this is unlikely. There continues to be tremendous demand for technical adepts of all stripes.

The greater danger is in finding yourself hard at work on something you don’t actually like or care about. As you study, ask yourself sincerely, “Is this for me? Is this what I want to be doing?” If not, you may eventually feel stuck in a role that doesn’t fit you any better than someone else’s clothing.

A man whose desire is to be something separate from himself . . . invariably succeeds in being what he wants to be. That is his punishment.

—Oscar Wilde, De Profundis

“Everyone gets everything he wants. I wanted a mission. And for my sins, they gave me one.”

—John Milius and Francis Coppola, Apocalypse Now

While the type of tech you study probably won’t make or break your ability to find work, it can have a big impact on the kind of work you do. SQL is de rigueur for business analytics, as C is for embedded developers, and neither is a substitute for the other. If you want to develop websites, you should know basic HTML, CSS, and JavaScript, but those are no replacement for R or Python if you want to be a data scientist.

The counter-intuitive point here is that the relative value of different technical knowledge depends as much on you personally as on the world around you. SQL, JavaScript, and C are all languages that make some people deeply happy, and others miserable.

Know yourself.

—Delphic maxim sometimes attributed to Socrates

Now that you know who you are, what do you want to be?

—The Beatles, Baby You’re a Rich Man

Choice is freedom, but it is also a burden. Some questions, like “What should I do with my life?,” are obviously hard. Others, like “Should I learn Biopython so I can fight disease, or Godot so I can make video games?” are surprisingly hard. The pursuit of knowledge is not merely about what you want to learn, but about how you want to spend your days. The only wrong choice is one that’s not true to yourself.

I always ask software engineering interview candidates to implement a factorial function, then (assuming factorial goes well) a Fibonacci function. These are admittedly trite exercises that, for senior engineers, serve as quick warm-ups. Junior engineers didn’t used to have much trouble with them, either; but over the past ~10 years, a strange trend has arisen whereby new college grads consistently go straight for recursive implementations. They all seem to do this. It’s uncanny. (Are entry-level software engineers all reading the same prep book, or watching the same YouTube channel or something?) Recursion usually makes for terrible solutions, and I’d like to show you some reasons why.

🪤 Despite The Problem with Big O, this post bears a light sprinkling of big-O expressions. It’s fine to skip right past them if you want. Moreover, we’ll distinguish space and time complexity using ‘s’ and ‘t’ subscripts; for example, Oₛ(N)Oₜ(2ᴺ) indicates that an algorithm completes in linear space and exponential time. But again, it’s fine to ignore all that, unless you’re a glutton for undergrad-level Computer Science.

Factorial

First, let’s look at a recursive factorial function. We’ll use Rust, but (except where noted) the same principles hold for other languages.

The first thing that should concern you about this code is integer overflow. Factorials get big fast. Integer overflow is easy enough to avoid (potentially at the expense of performance) by switching to a growable return type like num::BigUint. Some languages, like Python, even use growable integers by default.

The bigger problem is stack overflow. Most programming languages (with notable exceptions like Golang, as explained by Dave Cheney) allow each thread only a finite amount of stack space. Each recursive call takes another chunk out of that space. When the stack space is gone, the program crashes, and it crashes ugly.

Some languages (like Scheme, Haskell, and Scala) support Tail Call Elision (TCE), silently optimizing certain functions so they don’t overflow the stack. The implementation above is not amenable to this optimization, and most candidates who jump straight to recursive solutions (or at least, the ones I’ve asked) can’t even explain what a “tail call” is. (The point isn’t that engineers should know about TCE, but that we shouldn’t write code that crashes in the absence of particular optimizations.)

Linear (Oₛ(1)) recursion is exactly the sort of thing that works fine in test code, but blows up in production. Do not recurse in production code unless you have a good reason. Figuring out whether my team will have to explain things like this to new hires is part of the reason I still ask these clichéd, boring questions.

The obvious implementation I would expect most juniors to write—and I swear, they used to do so—is a for-loop:

A plain old loop is absolutely fine for something like factorial. A more sophisticated approach would be to delegate most of the work to a standard library function:

Yet another option is a look-up table of precomputed values:

None of these solutions is liable to blow the call stack, and all of them have better complexity than the recursive solution. These principles aren’t specific to factorial, either: If you want to be a professional computer programmer, you have to be able to write a for-loop. Don’t get any fancier than that unless you can explain your reasons for doing so.

Fibonacci

The reason I ask Fibonacci, even if someone nails factorial, is that efficient implementation requires a translation from the problem domain to the solution domain. That’s something a candidate will do easily if (and only if) they’re already in the habit.

The Fibonacci sequence, as a mathematical object, is defined recursively:

fib(0) = 0
fib(1) = 1
fib(n) = fib(n - 2) + fib(n - 1)

A naive implementation looks like this:

Aside from the overflow issues we already discussed in the context of factorials, this Fibonacci function has a severe scalability problem in the form of exponential time complexity. This is another great example of something that works fine in tests, but can cause production outages in the real world. Exponential isn’t quite as bad as factorial, but it’s bad enough to ruin your day.

The complexity here isn’t due to recursion per se, but to a misunderstanding of how we should translate inductive definitions like fib(n) into computer programs where time and space actually matter.

So many good ideas are never heard from again once they embark in a voyage on the semantic gulf.

—Alan Perlis, Epigrams in Programming

The first step is to identify the state that must be passed from one iteration to the next; in other words, the signature of the recursive function. For Fibonacci, the state is the most recent two results, and the number of iterations remaining. Here’s a linear complexity (tail-)recursive Fibonacci implementation:

When candidates who’ve implemented the naive solution above are asked to address the exponential complexity, they always—and I mean, always—try adding a cache, effectively memoizing the function. That gets us back down to linear space and time (Oₛ(N)Oₜ(N)), but a simple loop would run in constant space and linear time:

Higher Order Functions

Students of Functional Programming spend their first year learning to recurse, and their second year learning not to.

—Unknown

Much as factorial can be expressed as the product of a range of integers, a more sophisticated Fibonacci implementation might use a Higher Order Function (HOF) like successors:

Using HOFs instead of hand-rolled loops is a pretty strong indicator that this ain’t the author’s first rodeo. (Similar HOFs exist in most languages, or else can be easily implemented using iterators or generator functions. For example, what Rust calls ‘successors,’ Haskell and Clojure both call ‘iterate.’)

Higher order functions are often even better than loops, for two main reasons:

1. They express your intent directly. A loop that processes elements of one array and adds them to another is rarely as clear as a call to map and/or filter (or an equivalent comprehension in Haskell, Scala, or Python). Such direct expression of intent makes the code easier to work with for both humans and compilers.

   In some cases for_each may also be faster than a loop, because it will use internal iteration on adapters like Chain.
   —Trait std::iter::Iterator

2. They’re more composable, enabling you to share low-level functionality across functions.

Suppose we want to know how many Fibonacci numbers are below a given value. If we start from the ‘fibonacci_loop’ function above, there’s no good way to factor out the loop logic itself. We’re stuck writing a completely new, though conceptually very similar loop.

With the HOF based version, though, we can easily factor out not only the iteration, but the iteration state; in this case, keeping the value ‘b’ out of the caller’s scope.

Some technical leaders take the preference for HOFs to the extreme. Others positively fetishize raw loops. My own experience is that HOFs are almost always the best option. They scale better than loops, both within individual programs, and as library features. They also tend to go hand in hand with short variable names, and let you mentally separate mechanics from semantics; e.g., separating loop logic (I have a list, and want the Nth item) from the business domain (I’m implementing a Dutch auction, and need the second-highest bid).

But, the glue required to match HOF callback signatures—for example, in our HOF-based Fibonacci functions above, the Some() and .expect() calls—is a pain for small examples. It’s often best to start with a short hand-written loop, but replace it with a HOF at the very first sign of complexity.

OK, fine, recursion is occasionally useful

When, if ever, should you use recursion? Here’s a fancy Fibonacci function I wrote a long time ago that recurses instead of looping. I felt OK writing that because it runs in logarithmic, rather than linear, time and space (Oₛ(log N)Oₜ(log N)). Recursion is also a convenient way to get a data stack for free, if you’re willing to abuse the call stack to hold data; for example, when traversing a binary tree. Even then, it’s usually better to use an explicit stack.

💡 The only difference between depth-first and breadth-first search is whether newly encountered nodes are added to a stack (LIFO) or a queue (FIFO).

If you know what you’re doing and why, then it’s OK for recursion to be one of the tools in your toolbox. It is not, however, a good idea to recurse where a loop or HOF would suffice merely because recursion is what the cool kids seem to be doing.

The degree of civilization in a society can be judged by entering its prisons.

—unknown; often misattributed to Fyodor Dostoyevsky

The most obvious form of system degradation is catastrophic failure. Conventional wisdom is that when a computer program cannot serve its intended purpose, it should fail fast and loudly. The aptly named .NET Environment.FailFast method, for instance, “immediately terminates a process after writing a message.” We’ve previously discussed why Fail-fast isn’t always the best strategy; but when a program absolutely must crash, it should at least crash well.

GEOFFREY: Why, you chivalric fool—as if the way one fell down mattered.

RICHARD: When the fall is all there is, it matters.

—James Goldman, The Lion in Winter

Graceful fail-fast degradation is epitomized by the ‘expect’ method of Rust options. Options are containers that can either be empty, or hold a single value. The expect and unwrap methods both return the contained value, or else print an error message and terminate the program with bad status if the container is empty. The only difference between ‘expect’ and ‘unwrap’ is that ‘expect’ lets you customize the error message. Good error messages are grace incarnate. Rust programmers should always prefer ‘expect’ to ‘unwrap,’ for two reasons:

1. No matter how sure you are that a particular operation won’t fail… what if it does? The moment the impossible happens is precisely when a helping hand would be most appreciated.

   We place absolute confidence in the Titanic. We believe the boat is unsinkable. —White Star Line Vice President P.A.S. Franklin

   The Titanic actually carried just 20 lifeboats . . . This was far too few for the number of people aboard. —Titanic Facts

   Believing your ship unsinkable is no excuse for not providing enough lifeboats.

2. Well-written error messages make code easier to read, because they explain why the author believes a particular state is unreachable, or at least what the relevant values represent. Good inline error messages are among the kindest mercies you can bestow on anyone who would maintain or extend your code, even if no end user ever sees them.

A more common form of graceful degradation is ensuring that media remain accessible to users who have (permanent, temporary, or situational) disabilities. Alternate text for online images, closed captions and audio descriptions for movies, and colorblind-friendly visual distinctions (such as consistent top-down ordering of red, yellow, and green traffic lights) are all forms of graceful degradation. At their best, such “alternate” forms of communication are first-class aspects of design in their own right. While a blind moviegoer may never see the images on screen, most sighted viewers will never see the closed captions or hear the audio descriptions that may be the most entertaining parts of the show.

No real-world system is ever infallible. Something as simple as providing a decent error message can make a big difference to users and developers alike. Getting things right when things go wrong is a mark of craftsmanship that distinguishes good engineers and designers from great ones.

Don’t train alone. It only embeds your errors.

—Vesemir, The Witcher 3: Wild Hunt

Too many cooks spoil the broth.

—Anonymous

Let’s talk about (how to avoid) bad code reviews.

Authors:

1. Thou shalt not send huge change sets for review. Yea, verily, ‘tis far better to submit one hundred Pull Requests of ten lines apiece, a few at a time, than to eventually plop out a single PR for a thousand-line code change. If thou canst not figure out how to present thy changes as a series of small, independently reviewable pieces, thou shouldst brainstorm with a colleague, and invest thy faith in the power of factoring.

2. Thou shalt not send multiple, simultaneous PRs that depend on each other, as any change to one PR may invalidate others. Even if thou dost not mind manually synchronizing several PRs, thou layest a gross burden upon the reviewer, who hath not thy understanding of the interrelated change sets thou proposest, nor desireth to make busy-work for ye. Have a heart, give the reviewer a break, and please, never submit a PR-storm.

3. Thou shalt not send copy pasta for review. Thou not only irritatest the reviewer on general principles, but obligest them to review the same code again and again, and puttest them in the awkward position of leaving the same comment at each location.

Reviewers:

1. Thou shalt be responsive. Thou shalt not leave PRs unattended for days or weeks.

2. Thou shalt not let the perfect be the enemy of the good. Mere room for improvement is not sufficient reason to delay merge. Improvements can be made in later PRs; and if thou feelest strongly enough, thou canst darned well send such improvement PRs thyself, after merging the original. The question thou shouldst ask thyself is not “could this be better?” but “will this make us better?” Will the incoming change, as it stands, be a net positive? If so, thou shalt merge, and that right soon.

3. Thou shalt not delay merges in the name of bike-shedding. Thou art a lookout, not a gatekeeper. Thou art a teammate, not a fashionista. Thou art not the arbiter of good taste. Support and encouragement are more effective than exclusion. Water the flowers, and let the weeds die on their own.

4. Thou shalt be gentle. When thou must request changes (or even reject entire PRs!), no matter the reasons, thou shalt remember thy humanity, and that of they teammates. If thou believest that people should learn not to take professional feedback personally, then thou hast yet much to learn thyself. Disrespecting the work disrespects the worker. Thou hast been invested with power, and shalt wield it with compassion.

It would be a missed opportunity to think of code reviews as solely quality control. I’ve learned at least as much from reviewing other people’s work as from their reviews of my own. As a manager, I’ve found code reviews an invaluable way to break down silos, ensuring that the code is not “owned” by any single individual without whom it would be unmaintainable.

A final note is that code reviews are often worth following even if you’re not an active participant. As workplaces have become remote, and communication increasingly asynchronous, reviews have gained importance as a way for engineers to coordinate their activity. If you want to see how your developers actually work together, look no further than their code reviews.

Movement, in absolute terms, is an illusion. Each of us stands at the origin of our own frame of reference, within which we are still. This principle applies not only to physical movement, but to progress of any kind.

Things fall apart; the centre cannot hold

—William Butler Yeats, The Second Coming

The most facile way to assess our own velocity is relative to whomever or whatever is nearby, yet it is no more meaningful to say we are flying past others than it is to say they are flying past us. In the moment, it is not always clear which way is forward, and which reverse.

I didn't leave the Democratic Party. The party left me.

—Ronald Reagan

Observing progress relative to big things, rather than close ones—mountains in the distance, stars in the sky, or the events of the distant past—is hard, because distance obscures detail. Ever wonder why it seems like music/movies/media were better in some other decade? It’s because, in hindsight, the hits seem closer and closer together, even as the interstitial noise is slowly forgotten.

The phenomenon by which objects appear closer to each other the farther they are from us is called perspective. The related phenomenon by which we seem to move faster relative to nearby objects than to distant ones is called parallax.

💡The parsec, an astronomical unit of distance, is an abbreviation of “parallax second.” It is how close another star would have to be to the plane of our orbit around the sun to seem, from our perspective, to have moved through two arcseconds (0.0003% of the way across the sky) in six months. During that time, we travel nearly 300,000,000 km, yet a parsec is over 30,000,000,000,000 km. There are no stars so close.

Although we cannot meaningfully assess our progress relative to others, what we can determine are changes in our own velocity. Changes in velocity are called acceleration, and like all derivatives with respect to time, acceleration is fundamentally an instantaneous property: We know, in any given moment, whether we are speeding up or slowing down. The sensation is often visceral.

If there is a lesson to be learned from all these observations, it is that progress is an illusory thing. Measuring progress in a changing world, or comparing our own progress to that of other people—or worse yet, other people’s progress relative to ours—is a fool’s errand. The question we can answer is whether we’re doing better than we were yesterday. Whether “doing better” means moving faster or slower… well, that’s up to you.

In a perfect world, things that shouldn’t matter wouldn’t matter. We do not, as you may have noticed, live in such a world.

One thing that shouldn’t matter is the traversal order of data structures representing mathematical sets. Semantically unordered collections (like hash sets and database tables) come up frequently in software development, but they’re not really amenable to the usual “Divide and Conquer” approach.

Divide and Conquer and Subtle Interactions

We once discussed Divide and Conquer as a Project Management technique. This time, we’re discussing it in the context of data processing.

To “Divide and Conquer” is to break (Divide) a big problem into smaller ones that we can solve (Conquer) independently. For example, to add up a bunch of numbers, we can take them two at a time (Divide), and apply our handy binary + operator (Conquer) to get partial results, repeating until we’ve summed the whole collection:

Addition is a seductively flexible operation, because it is order agnostic in two important ways.

• It’s associative, meaning it doesn’t care about the order of application: (1 + 2) + 3 = 1 + (2 + 3). It doesn’t matter how we Divide our collection into pairs.

• It’s commutative, meaning it doesn’t care about the order of its operands: 1 + 2 = 2 + 1. When we Conquer each pair of numbers, it doesn’t matter which number goes on the left or the right.

As it turns out, though, most Software Engineering is a lot harder than adding up integers. In fact, much of computer programming entails being surprised to discover, again and again, that seemingly simple things are not so simple after all. For example, sticking with that handy dandy binary + operator:

Oops! The sum of a collection of floats depends on traversal order, because floating point addition is not associative. Specifically: Epsilon is the smallest number that can be added to 1.0 without being discarded as a rounding error. The sum epsilon/3 + epsilon/2 is greater than epsilon; so, if we add the numbers up in their original, sorted order, we’ve picked up enough “momentum” by the time we hit 1 to tip it over to the next representable value. If we instead sum these numbers in any other order, none of the individual fractions of epsilon is ever big enough to affect the accumulated sum. United they stand, divided they fall. Non-associative operations interact badly with the Divide step of Divide and Conquer dataset processing.

💡 Fun fact: Rust doesn’t allow sets of raw floats, because their comparison semantics are so… shall we say, irregular?

That dirty double-crossing binary + operator, that we thought was our friend, is also commonly used for concatenation. Concatenation has the reverse problem of floating point addition: It’s associative, but not commutative. In other words, “Hello” + “World” is not the same as “World” + “Hello”. The concatenation of a set thus depends on traversal order. Concatenation cares which element comes first, but there is no meaningful “first” element of an unordered collection. Non-commutative operations interact badly with the Conquer step of Divide and Conquer dataset processing.

There’s no such thing as an original sin.

—Elvis Costello, I’m Not Angry

Flakiness

In most languages, hash set traversal order may vary from run to run:

A common consequence is test flakiness. If you have tests that pass or fail seemingly at random, one likely culprit is a set being reduced by a non-associative or non-commutative function:

To work around this kind of flakiness, some implementations guarantee consistent traversal. In JavaScript, for example, iteration recapitulates insertion order:

That’s nice for canned test cases, but it messes up value semantics. It would be weird if two sets compared equal, yet had different sums. JavaScript punts that problem by not providing a set equality operation at all, which seems like throwing the baby out with the bathwater.

Rust takes the opposite approach, making HashSet traversal order deliberately hard to predict. The goal is to prevent programmers (and attackers) from relying on hash implementation details. But the fact that Rust can save us almost entirely from the other main flaky test culprit (data races), yet really can’t help with traversal order issues at all, speaks to how profound this problem is.

Parallelism

It’s tempting to think the solution to set traversal order dependence might be to do all the work in parallel. Unfortunately, we can’t usually do everything all at once, because operations depend on each others’ results. We can’t start smooshing our partial results together into a final, cohesive solution until we’ve done some Conquering, and we can’t start Conquering until we’ve done some Dividing.

While the parallel collections abstraction feels very much the same as normal sequential collections, it’s important to note that its semantics differs, especially in regard to side-effects and non-associative operations.

—Scala Parallel Collections Overview

Takeaway

In that perfect world where we don’t yet live, programming languages would automatically recognize operations that might not be commutative or associative, and would warn us if we used order-dependent operations unordered collections. Until we live in that world, there are two ways to minimize the fallout from the bizarre reality that even unordered sets must be traversed in some order, however meaningless:

1. Be aware of the associativity and commutativity of the operations you rely on, especially when using them to Divide and Conquer (respectively). When designing new functions, try using them with unordered containers in your test suite, and do what you can to make them order-independent.

2. If your data can be meaningfully sorted, logarithmic access time is acceptable, and your standard library offers ordered collections like BTreeSet/Map, std::set/map, and TreeSet/Map, consider choosing them over hash-based implementations.

Please share relevant thoughts, research, or libraries in the comments.