How it works
From observability to causality.
What truffleroot builds from your environment, what happens when an alert fires, what an explanation contains, and how the model learns between incidents.
Sources
What bootstrapping reads.
The point is not the volume of data. It is what gets built from it.
Read from the systems you already run
- Linear
- Grafana
- Prometheus
- Datadog
- Splunk
- Sumo Logic
- GitHub
- GitLab
- Bitbucket
- Slack
- Linear
- Grafana
- Prometheus
- Datadog
- Splunk
- Sumo Logic
- GitHub
- GitLab
- Bitbucket
- Slack
- Linear
- Grafana
- Prometheus
- Datadog
- Splunk
- Sumo Logic
- GitHub
- GitLab
- Bitbucket
- Slack
Past incidents
What actually broke, when, and what it looked like from the outside.
ITSM tickets
The symptoms people reported and the components they pointed at.
Postmortems
The causes your team already established, and the timelines behind them.
Runbooks
What to check, and in what order, for the failures you already know about.
War-room conversations
The hypotheses that were raised, tested and discarded under pressure.
Service topology
Which components depend on which, and how calls flow between them.
Traces, logs and metrics
The signals events are detected from, and the baselines thresholds are set against.
Deployment and configuration state
What changed, where, and when.
Code and infrastructure configuration
How services are wired, pooled, retried and rate-limited.
Your engineers
The knowledge that never made it into a document.
The model
What it produces.
Explicit definitions and rules, reviewed by your engineers, that describe how this environment fails and no other. A service map says who calls whom. This says how a failure travels.
Each definition is inspectable. Your engineers approve what enters the active pipeline.
Event types and properties
The events that matter here, and the properties that tell one instance from another: which service, which pool, which revision.
Causal rules
How one kind of event leads to another in this environment, scoped to components and bounded in time.
Lookback windows
How far back to look for each kind of cause, so a deploy from last week is not blamed for a failure this morning.
Detectors and thresholds
How each event type is recognised in telemetry, calibrated to your baselines rather than to a default.
Dependency topology
The graph an investigation is scoped against.
Versioned model
Every definition and rule is versioned, so any past investigation can be replayed against the exact model it ran on.
The pipeline
What happens when an alert fires.
Scout, a deterministic causal engine, narrows the search. The AI investigator does its work at the end, on a short list of hypotheses, not at the start on everything.
An incident opens. A detector fired, or a person raised it.
Alert
An incident opens. A detector fired, or a person raised it.
Relevant topology
The engine identifies the portion of the dependency graph that can be involved: the affected component, what it depends on, and the shared infrastructure beneath them.
Relevant events
It collects events across that scope, inside the lookback window appropriate to each event type.
Causal reasoning
It applies your causal rules to those events and constructs the causal graph: which events could have led to which.
Ranked hypotheses
Plausible root causes, ranked by how well the chain from each one to the symptom is supported. The same evidence always produces the same ranking.
Investigation
An AI investigator queries the evidence, tests alternatives and challenges each link, using runtime state, telemetry, runbooks and past incidents. It can overrule the ranking when the evidence demands it. Hypotheses that don't hold up are ruled out, with the reason recorded.
Explanation
A causal narrative with the evidence behind each link, what was ruled out, what is still open, and a suggested fix. Not a score.
The output
What an explanation contains.
The engineer on call should be able to inspect it, verify it, challenge it and act on it.
- What happened
- What happened before it
- How those events are causally connected
- What evidence supports each link
- What was ruled out, and what is still open
- A suggested fix
INC-2231 · Explanation
Root cause: a connection pool limit reduced by a config change
At 14:11 the connection pool limit on payments-svc was reduced from 40 to 20. Under normal afternoon traffic the pool saturated fifteen minutes later, payments-svc latency rose, and checkout-api, which calls it synchronously, breached its SLO at 14:31.
- 14:11:04config_changepayments-svcpool.max_connections 40 → 20root causeEvidence for the next link
- Revision cfg-4812 applied by the config service at 14:11:04. The diff sets max_connections from 40 to 20.
- Rule: a pool config change on a service precedes pool saturation on that service, within 30 minutes.
- Active connections pinned at 20 from 14:26:40. Before the change they peaked at 31.
- 14:26:40pool_saturationpayments-svcpool.wait_p95 3 ms → 1.9 sEvidence for the next link
- 94% of slow payments-svc spans wait on pool.acquire. 2,113 traces sampled.
- Rule: pool saturation precedes a latency shift on the same service.
- 14:29:12latency_shiftpayments-svcp99 180 ms → 2.4 s · timeouts 3.1%Evidence for the next link · 3 items
- 14:31:55latency_shiftcheckout-apip99 2.6 s against a 1.2 s SLOEvidence for the next link · 2 items
- 14:32:10alertcheckout-apislo_burn_rate fired
Ruled out
deploy on inventory-svc v3.8.1: its latency stayed flat through the window.
Unresolved
Did a traffic shift at 14:20 also contribute? Not enough telemetry to say.
Suggested fix
Revert cfg-4812 to restore pool.max_connections on payments-svc to 40.
Then alert on pool wait time, which moved 3 minutes before latency did.
Between incidents
How the model improves.
After an incident, the evidence it produced is used to propose changes to the model. Each proposal names the incidents it came from and what it would have changed.
- New event types
- Improved event properties
- New or modified causal rules
- Namespace consolidation
- Tuned thresholds and lookback windows
- Type
- Detector threshold
- Detector
- payments-svc · pool.wait_p95
- Change
- 500 ms → 250 ms
- Derived from
- INC-2231
Would have fired 4 m 10 s earlier in INC-2231. No additional fires across 90 days of history.
Nothing modifies itself. A proposal is replayed against past incidents, then enters the active model when an engineer approves it.
Find the root cause in your own environment.
Tell us what you run and what keeps breaking.