On-call incident router: an AI triage workflow for production alerts
Checkout is failing. The API has returned 5xx errors for eight minutes, the status page is still green, and peak traffic is twenty minutes away. The on-call question is simple: is this us, and do we page someone?
This workflow answers it in one pass. A cheap scout model reads the alert and labels it HOT or COLD. A router sends real incidents to live checks and a war-room writer, then pauses for a person to approve the page. Known noise, such as a flaky canary, takes a short branch that writes a close note and never pages anyone.
Free in Sim. No account or API key.
Sample input
P1 checkout API 5xx spike last 8 min. Status page still green. Customers posting on social. Peak traffic in ~20 min. On-call: is this us?
How the workflow runs, step by step
1. Start
Alert in
The alert text from PagerDuty, Slack or your monitoring tool starts the run.
Next: Scout
2. Agent
Scout
A small, cheap model classifies the alert as HOT or COLD and says why in one sentence. It only uses the alert text, so it cannot invent telemetry.
Next: Severity
3. Router
Severity
Keyword rules pick the branch. Words like canary, flake or do not page go to the noise path; HOT, P1 or 5xx go to live checks.
Next: HOT path (HOT), Noise note (COLD)
4. Join
HOT path
Fans the incident out to two checks that run side by side.
Next: Status fetch, Outage search
5. Tool
Status fetch
Fetches a public status page, standing in for your own status URL.
Next: Merge checks
6. Tool
Outage search
Searches the web for a known vendor or public outage.
Next: Merge checks
7. Join
Merge checks
Waits for both checks and combines what they found.
Next: War-room writer
8. Agent
War-room writer
A stronger model writes the page card: impact, likely cause and the next check. It never runs for noise.
Next: On-call approve
9. Human approval
On-call approve
The run pauses. A person approves or rejects the page before anyone treats it as real.
Next: Page card
10. Result
Page card
The approved page card is the result of the run.
11. Agent
Noise note
For known noise, a cheap model writes a short close-out instead. No page and no human step.
Next: Close note
12. Result
Close note
The close note is the result of the run.
Why build it as a graph
A single prompt would run the expensive writer and the live checks on every alert, including noise. Splitting the work into a cheap classifier, a router and a gated writer means you only pay for the war-room card when there is a war room, and a human stays in charge of waking people up.
Things to try
- Run it as is in Sim: the alert goes down the HOT path and pauses for your approval.
- Change the alert to “flaky canary, do not page” and run again. The same graph takes the COLD branch and skips the expensive writer.
- Compare the two runs: the route each took and what each one cost.