Machine Diaries

The Story

Observability has three pillars — metrics, logs, traces — and one blind spot: biography. Dashboards tell you what a service is doing. Nothing tells you what it has been through.

The knowledge that “the search cluster has been flaky since v41, never fully recovered, and gets worse under cron pressure” lives nowhere in your systems. It lives in people. It decays in Slack. The senior SRE who remembered why cart-service can't be trusted on Mondays quit last spring — and took the only copy of that knowledge with her.

Machine Diaries gives every service a @Persona twin whose @Trait fields capture qualitative operational state — degradation suspicion, confidence in dependencies, deploy anxiety, a running narrative memory — maintained by an LLM from deterministic telemetry digests, persisted with full history, and queryable like inventory. The fleet becomes a population. Postmortems are read, not reconstructed.

How It Is Built

The twin and its trait contracts

@Persona(value = "The payments service: latency-sensitive, PCI-scoped, "
                + "dependent on auth, ledger-db, and the card-network gateway.",
    evolutionSchedule = "@weekly",
    evolutionDescription = "Unexercised code paths and unrotated credentials "
                        + "decay confidence; a quiet week without incident "
                        + "slowly heals trust.",
    internalLoop = true, loopSchedule = "0 0 3 * * *",
    loopDepth = LoopDepth.SHALLOW, loopQuietPeriodHours = 6,
    loopDescription = "Review the day's digests against my biography. Is "
                    + "anything slowly going wrong that no single day would reveal?")
public class PaymentsServiceTwin { /* @Trait fields */ }

The trick is in the trait contracts, which are written in SRE language and encode hard-won wisdom directly into the state rules: “trust recovers slower than it breaks — two clean weeks per ten points, never instantly.” Tribal knowledge stops being tribal the moment it becomes the rule a field is governed by.

Deterministic @CodeFunctions pre-digest the telemetry — RSS trends, p99 drift, deploy outcomes, cron load — and the LOCAL tier updates each twin hourly from those digests. Zero cloud tokens for routine bookkeeping.

Worry travels along the dependency graph

When the database twin's outlook darkens, its dependents hear about it — before their own error budgets start burning. The dependency graph is the social graph: phaseManager().broadcast(dbIncidentEvent, pipeline) with BlastRadius.RELATIONSHIP and AttenuationModel.RELATIONAL, so impact decays with topological distance. Cascading risk assessment, ahead of cascading failure.

The 3 a.m. reflection

The internal loop runs at three in the morning, as is traditional for worrying — but only after six quiet hours (loopQuietPeriodHours = 6): no philosophising mid-incident. The loop is where slow rot becomes visible — the leak, the drift, the creeping p99 that no single day would reveal. A twin whose deep loop revises its beliefs is a twin raising its hand.

Deterministic governance, as always

@PersonaSubscriber
public class FleetGovernor {

    @OnPersonaEvent(persona = ServiceTwin.class,
                    type = PersonaEventType.THRESHOLD_CROSSED,
                    trait = "degradation_suspicion", threshold = 75,
                    direction = Direction.ABOVE)
    public void onSuspicion(TraitChangeEvent<ServiceTwin> e) {
        tickets.open(e);
    }

    @OnPersonaEvent(persona = ServiceTwin.class,
                    type = PersonaEventType.POPULATION_TREND,
                    trend = Trend.FALLING)
    public void onFleetMoodDrop(PopulationTrendEvent e) {
        warRoom.convene(e);
    }
}

For postmortems, report(twin, EvolutionWindow.lastDays(30)) shows when the twin's belief in its own health diverged from the dashboards and which neighbours saw it coming, and customReport(...) drafts the narrative from the diary entries. Even decommissioned services remain consultable via loadRetired() — a queryable fleet graveyard, which is morbid and extremely useful.

Why It Is Unprecedented

This is the senior engineer's hunch, industrialised — with reasons attached. Suspicion defined as a pattern across deploys, rather than a threshold on a gauge, can raise its hand days before any alert fires. The diary is searchable years later and survives every resignation. And the design's quiet generality is worth noticing: PrOOPt States turns out to be state management for everything that has a story.

The Fine Print

A twin's beliefs are an interpretive layer over telemetry, never a replacement for it; alerting on raw SLOs remains primary. Anthropomorphism is the interface, not the ontology — trait contracts must stay tied to observable evidence (“raise when RSS trends up”), or the diary degenerates into fiction. And trust the system only after it runs in shadow mode and earns a precision record. Rollout discipline is part of the design, not an apology.