You’ve been assigned to patch servers running the legacy billing system built by L. Jenkins before he left the firm. But before you’re onboarded, Steve from the DevOps team insists you have a chat before proceeding. He sits you down and begins: “So back in 2016, when the great data-center migration was kicked off, Jenkins made an arrangement with the payment processor that required …hmm… let’s call it an ‘unconventional’ certificate workaround…”
Recognition
If you’ve spent time cursing at screens while troubleshooting complex systems and learned via word of mouth about their histories, limitations, pitfalls, and legends; you know what it means to be initiated through a recitation of Lore that mirrors oral traditions predating written language itself.
In any given technical operation, legendary figures are heralded as visionaries or denounced as villains, often a bit of both. Descriptions of mythic structures and epochs are invoked to visualize the network environment from the time of troubles before a reliable WAN link to the backup site was installed. Heroic battles of disaster recovery and seat-of-the-pants refactoring are whispered in reverence. Comical interludes of hapless desktop technicians feuding with helpless end users round out epic tales of forklift deployments and weekend-long conference calls with infamous vendors. “I don’t care if he’s the VP of sales and he’s on vacation in France. Wake him up!”
These “load bearing myths” describe beliefs and legends that form the basis of a culture’s essence and purpose. Legends and lore that explain legacy systems and their pitfalls get passed down to junior engineers and staff, becoming the great stories of every back office. But for some reason significant portions of the real knowledge and wisdom captured, kept, and passed down through shop lore never quite makes its way into the “official” documentation.
Artifacts
The term “Lore” itself feels like it refers to something curious and ancient. Not to be confused with backstory, we’re talking about critical but often informal lessons and information transmitted between people, most often verbally and outside the normal channels.
In the ancient oral traditions of knowledge sharing, we were passing down everything we knew about the world, ourselves, and everything in it by crafting stories we could memorize and tell one another. We’ve had written language for five millennia, tops, but we’ve been making meaningful noises at each other for more like 150,000 years.
Our limited human memory drove the development of complex literary forms. Structural systems of epic poetry used rhythm and repeating phonetic patterns that resisted alteration over enormous timespans. Religions transmitted inherited traditions filled with payloads of practical knowledge between generations. The traditions capable of retaining meaning in spite of ever-changing languages and cultures built the foundations of knowledge. These shared cultural memory systems were our most powerful cognitive technology.
And in spite of all our advancements and achievements, we still use lore to preserve and transmit critical knowledge.
Legends: The heroic network engineer who rebuilt the router configuration on the fly during a major outage with no downtime, all while dialed in through a truck stop wifi connection while on vacation.
Myths: Foundational beliefs about how we do things. “No one knows why we have to label tickets as stories AND set the issue type to story, but if we don’t do it, you will get a nag mail from the project coordinators. Always do both.”
Sagas: Complex multi-part stories about system evolution, data center migrations, storage network failures, and disaster recovery drills. Also the federated procedural “runbooks” that govern a multi-phased deployment.
Parables: Cautionary tales about what happens when you don’t follow order of operations as outlined on the sticky note on the backup tape library.
Ritual knowledge: The tribal wisdom of deployment checklists, two party code reviews, and incident response protocols, including the TTN or Time To Notify a stakeholder of a service interruption.
These legacy patterns are deeply embedded in our shared experience. We start from this foundation whenever we try to build shared understanding. We’ve developed instincts to help us focus on important details, establish context, or refer to common conventions. These are the shared cognitive dance steps we perform around a problem. The dance steps are older than the words. Older than words. These patterns reverberate in the forms that emerged.
Okay, but did you include a README?
Or maybe we should write a blog post (that no one will read?)
Wait…excuse me? What did you mean when you said, “also the User Guide hasn’t been updated?”
We’ve grown well beyond the forms and conventions handed down from the five thousand years of language and literary development since the stone age, but modern documentation is just now becoming “digitally native.”
The costs that once constrained physical form are almost zero now. We can easily do pretty much whatever we want. So that’s kinda what we do. New tech is new and different, and how we know about it needs to keep up with the changes. No longer constrained by the structure of an epic poem, the steps required to achieve a certain goal in the moment can be boiled down and refined into procedure. Likewise, ritual recitation of a genetic lineage helped to preserve history in memory, but our history of software releases is more easily hosted as a simple list and presented on a web page or downloaded from a repository.
So we solved documentation, but where is it?
Accidentally Secret
Always hold your breath strolling past the Information Graveyard.
“Right, so the deployment procedure for the 0.6 release still includes setting the config on the message queue manually. I’ll send you a link to our knowledge base with an example. Don’t bother trying to search for it, you won’t be able to find it.”
Endlessly searching your shop’s wiki site, you’ve wandered in circles around the shared file storage. Futile attempts to find the relevant procedure and the corresponding dependencies have consumed endless cycles. This has happened often enough you’ve learned how to smell the stale dust of the information graveyard, where some diligent soul contrived their own categories and hierarchies based on the operational challenges and context of their day. The team wiki site, the shared storage on SharePoint, the ‘docs’ folders found in various repositories; each reflecting a view of the operation contrived by whomever set them up that first day.
Then a few months passed and the team’s shared storage became cluttered with random files and spreadsheets that didn’t fit well into the scheme. Or maybe the scheme wasn’t obvious and in haste someone made a “best guess.” This continued until the mass of disorganized but still valuable content became a monolith of foggy secrets.
One thing about these ad-hoc taxonomies and hierarchies is that they are arbitrary constructs of their time but must be treated as if they are fixed and future-proof to function. We invent them on the spot with a shrug, then measure and standardize the world with them. We create them on the fly, governed by constraints of the moment. Then those moments are replaced by new goals and challenges but we’re stuck with the previous patterns, which predictably stop working. You recognize the cliché of the information graveyard because this is a perennial problem for every technical operation.
Modern forms of technical documentation are the result of a process of refinement from long explanations full of narrative context into purpose-built portable artifacts. This necessarily occurs because the sum total of knowledge required to develop, maintain, and operate modern systems is often more than a single individual can retain. The amount of knowledge a single individual can wield effectively is a fundamental barrier to scale, so we don’t tell everyone the whole story. We have to deconstruct, federate, and delegate knowledge if we want to scale.
To accomplish this we boiled the knowledge down to its functional essence, but in the process we boiled off the stories, values, and mythology that inform the proper use of our tools, and with it the loyalty to the virtues that built and sustain the environments we work in. We used to embed knowledge in stories because that’s how we experience the world, as embodied agents moving through space over time. Now the stories that would tell a person who they are and where they’re going have to be pieced together from obscure patterns and hidden knowledge. And every reader has to do this alone, in their own mind. We instinctively feel this gap, this demand for context and meaning that’s no longer found on the page.
Lore persists because even well-organized documentation often misses something fundamental: the explanatory substrate that makes everything else comprehensible. It persists because it preserves what was stripped away: the context, history, and narrative perspectives that make documentation usable. Effective documentation must account for the explanatory substrate—typically bound up in lore—because it’s the foundation that makes the rest comprehensible.
The introduction made me laugh out loud! Perfect!