Skip to content
truffleroot

How it works

A pipeline built from your environment, not pointed at it.

What bootstrapping reads and what it produces. What happens when an alert fires. What an explanation contains, and how the model improves between incidents.

Talk to us

Sources

What bootstrapping reads.

The point is not the volume of data. It is what gets built from it.

  • Past incidents

    What actually broke, when, and what it looked like from the outside.

  • ITSM tickets

    The symptoms people reported and the components they pointed at.

  • Postmortems

    The causes your team already established, and the timelines behind them.

  • Runbooks

    What to check, and in what order, for the failures you already know about.

  • War-room conversations

    The hypotheses that were raised, tested and discarded under pressure.

  • Service topology

    Which components depend on which, and how calls flow between them.

  • Traces, logs and metrics

    The signals events are detected from, and the baselines thresholds are set against.

  • Deployment and configuration state

    What changed, where, and when.

  • Code and infrastructure configuration

    How services are wired, pooled, retried and rate-limited.

  • Your engineers

    The knowledge that never made it into a document.

The model

What it produces.

Domain-specific definitions and rules, reviewed by your engineers, that together configure an RCA pipeline for this environment and no other.

Each definition is inspectable. Your engineers approve what enters the active pipeline.

  • 01

    Event types and properties

    The events that matter here, and the properties that tell one instance from another: which service, which pool, which revision.

  • 02

    Causal rules

    How one kind of event leads to another in this environment, scoped to components and bounded in time.

  • 03

    Lookback windows

    How far back to look for each kind of cause, so a deploy from last week is not blamed for a failure this morning.

  • 04

    Detectors and thresholds

    How each event type is recognised in telemetry, calibrated to your baselines rather than to a default.

  • 05

    Dependency topology

    The graph an investigation is scoped against.

  • 06

    Pipeline configuration

    Everything the RCA engine needs to run against your environment specifically.

The pipeline

What happens when an alert fires.

A structured pipeline, stage by stage. The language model does its work at the end, on a short list of candidates, not at the start on everything.

  1. Alert

    An incident opens. A detector fired, or a person raised it.

  2. Relevant topology

    The engine identifies the portion of the dependency graph that can be involved: the affected component, what it depends on, and the shared infrastructure beneath them.

  3. Relevant events

    It collects events across that scope, inside the lookback window appropriate to each event type.

  4. Causal reasoning

    It applies your causal rules to those events and constructs the causal graph: which events could have led to which.

  5. Ranked candidates

    Plausible root causes, ranked by how well the chain from each one to the symptom is supported.

  6. Investigation

    A reasoning layer examines the leading candidates more deeply, using runtime state, telemetry, runbooks and historical incidents. Candidates that do not hold up are ruled out, with the reason recorded.

  7. Explanation

    A causal narrative with the evidence behind each link. Not a score.

The output

What an explanation contains.

The engineer on call should be able to inspect it, verify it, challenge it and act on it.

  1. 01What happened
  2. 02What happened before it
  3. 03How those events are causally connected
  4. 04What evidence supports each link
  5. 05Why this event is the root cause

INC-2231 · Explanation

Root cause: a connection pool limit reduced by a config change

5 events · 4 causal links

At 14:11 the connection pool limit on payments-svc was reduced from 40 to 20. Under normal afternoon traffic the pool saturated fifteen minutes later, payments-svc latency rose, and checkout-api, which calls it synchronously, breached its SLO at 14:31.

  1. 14:11:04config_changepayments-svcpool.max_connections 40 → 20root cause
    Evidence for the next link
    • Revision cfg-4812 applied by the config service at 14:11:04. The diff sets max_connections from 40 to 20.
    • Rule: a pool config change on a service precedes pool saturation on that service, within 30 minutes.
    • Active connections pinned at 20 from 14:26:40. Before the change they peaked at 31.
  2. 14:26:40pool_saturationpayments-svcpool.wait_p95 3 ms → 1.9 s
    Evidence for the next link
    • 94% of slow payments-svc spans wait on pool.acquire. 2,113 traces sampled.
    • Rule: pool saturation precedes a latency shift on the same service.
  3. 14:29:12latency_shiftpayments-svcp99 180 ms → 2.4 s · timeouts 3.1%
    Evidence for the next link · 3 items
  4. 14:31:55latency_shiftcheckout-apip99 2.6 s against a 1.2 s SLO
    Evidence for the next link · 2 items
  5. 14:32:10alertcheckout-apislo_burn_rate fired
Confirm root causeChallenge a link2 candidates ruled out · reasons recorded

Between incidents

How the model improves.

After an incident, the evidence it produced is used to propose changes to the RCA configuration. Each proposal names the incidents it came from and what it would have changed.

  • New event types
  • Improved event properties
  • New or modified causal rules
  • Namespace consolidation
  • Tuned thresholds and lookback windows
Proposed refinementAwaiting review
Type
Detector threshold
Detector
payments-svc · pool.wait_p95
Change
500 ms → 250 ms
Derived from
INC-2231
Evidencereplayed

Would have fired 4 m 10 s earlier in INC-2231. No additional fires across 90 days of history.

ApproveEditRejectInactive until approved

Nothing modifies itself. A proposal enters the active pipeline when an engineer approves it.

Build RCA around the system you actually run.

Tell us what you run and what keeps breaking.

Talk to us