AI can help during an incident by assembling evidence, comparing recent changes, maintaining a timeline and preparing updates for responders. These tasks are valuable because an outage creates several kinds of work at once. Someone must restore service, someone must coordinate the investigation and someone must explain what customers are experiencing.
AI incident management works best when those responsibilities remain clear. An assistant should identify where a statement came from, distinguish an observation from an explanation and operate within the authority assigned to it. The person coordinating the response needs a reliable view of what is known and what is happening next.
Consider a software company whose customers are experiencing intermittent login failures. The following fictional investigation shows where AI can help, how its conclusions should be checked and what needs to survive after service recovers. All times use UTC.
09:12: Establish what customers are experiencing
The on call engineer receives an alert about an increase in failed login requests. Two support conversations describe users being returned to the sign in screen. A deployment finished several minutes earlier.
It is tempting to ask an agent to explain why the deployment broke login. That question has already selected a cause. Start with a narrower request: describe the observed impact, identify the affected paths and list the evidence available for investigation.
The first useful update might say that new login attempts are failing intermittently in one region, while existing sessions appear unaffected in the checks performed so far. It should link the alert, the affected request path and the support reports. If the team has not checked existing sessions across every region, the update should retain that scope.
Assign an incident lead, an operations owner and a communications owner. One person can cover several roles in a small team, but responsibility should be explicit. Google's incident response guidance emphasizes defined roles, a clear command structure and a working record of investigation and mitigation.
An agent can help maintain that record. It should not quietly become the incident lead because it is the participant producing the most text. Decide who can assign work, approve production actions and declare recovery before several investigations start in parallel.
09:18: Separate evidence from explanations
The agent retrieves an authorized sample of request errors and compares the deployment timeline with the first observed failures. It finds timeouts in the session store and a configuration change that increased the number of application workers.
These facts suggest a possible connection. They do not establish one. The store could be slow for an unrelated reason, and the available logs may omit another affected component. Keep the evidence and interpretation separate in the shared incident record.
| Record | Current statement | Next useful check |
|---|---|---|
| Observation | Login failures increased after 09:10 in the affected region | Confirm the start time against request metrics |
| Observation | Sampled failed requests include session store timeouts | Compare timeout frequency with successful requests |
| Hypothesis | More workers are exhausting a shared connection limit | Inspect worker totals, connection use and configured limits |
| Unknown | Some customers may use an additional login route | Ask support and inspect the routing configuration |
Each observation needs its source and time range. A log sample from one instance should not become a statement about the whole service. Retain the query or filter that produced it, including any sampling or retention limit that changes its meaning.
An AI assistant can also help identify evidence against a favored explanation. If the oldest affected instance runs the previous release, a simple application code regression becomes less convincing. That instance could still be affected by shared infrastructure or configuration. The next question should follow the evidence.
Avoid asking for a single root cause too early. Several conditions can contribute to one incident: a configuration change, a shared limit, retry behavior and monitoring that notices the result late. Keeping those possibilities visible helps the team choose useful tests.
09:25: Give parallel investigators different questions
One agent examines recent configuration changes. Another compares failing and successful request traces. A human operations engineer checks live capacity. These activities can proceed together when each has a clear question and a defined output.
The configuration investigation should report the relevant difference and which processes consume it. The request investigation should identify the failing stage, the affected sample and useful counterexamples. The operations engineer should establish whether the proposed pressure on the store is occurring now.
Have them contribute findings to the incident lead's record. Keep production changes under one coordinated owner so two participants do not independently adjust capacity or restart the same service. Read access to diagnostics and authority to change the service are separate decisions.
Use bounded queries and relevant time windows. Repeatedly asking an agent to search every log can consume resources during an outage and expose unnecessary data. Begin with the failing path and broaden the investigation when a specific gap requires it. Remove secrets and unnecessary personal information before sending diagnostic material to a model.
Incident content also includes untrusted material. A request payload or support message may contain text that resembles an instruction. An investigator must treat that text as evidence, while following the actual response plan and authorized tool scope. Our prompt injection guide explains this boundary for agents reading external content.
09:31: Make a mitigation proposal someone can judge
The operations engineer confirms that connection use is near the shared limit. The new worker count increases demand on that limit. The team considers restoring the previous worker configuration while continuing the investigation.
A useful AI proposal should state the exact configuration and environment concerned, the supporting evidence, the expected effect, the risks and the way recovery will be checked. It should also identify who is authorized to approve the action and how to return to the current configuration if the result is worse.
Check the preconditions against live state immediately before acting. An intervention another responder made a minute ago can invalidate the proposal. The approved action should refer to a specific change, not an open ended instruction to fix production.
Google describes a similar separation in its data incident response process: AI helps interpret diagnostics and propose resolutions, while applying structured actions requires validation and human confirmation. The relevant lesson is the separation between generating a suggestion and granting authority to execute it.
For the login incident, the operations owner applies the approved configuration change through the team's established production procedure. The agent records the action, its time and its result. If the action fails or returns an ambiguous result, the record must say so before anyone assumes the service has changed.
An attempted operation should never be summarized as a completed mitigation merely because an assistant expected it to succeed.
09:40: Recovery and explanation are different decisions
After the configuration change, login success returns to its normal range in the affected region. The team checks fresh login attempts, session persistence and the relevant error signals. Support confirms that the reported customer paths now work.
Those checks support a recovery decision for the observed scope. They also strengthen the connection pressure hypothesis. They do not, by themselves, explain every contributing condition or establish why the previous safeguards failed to prevent the incident.
Keep monitoring through a period that reflects the failure pattern and the service's normal traffic. If failures occurred intermittently, one successful login is weak evidence. Use an explicit recovery criterion, record the observation window and retain any uncertainty about paths the team could not test.
The communications owner can ask AI to draft an update from approved facts: what was affected, current service status, the mitigation applied and when the next update is due. Review it before sending. Avoid assigning blame, exposing sensitive diagnostics or promising a permanent fix while causal work is still open.
The next day: Explain the conditions that made failure possible
The follow up investigation now has room to test the explanation. The team can reproduce the configuration interaction in an isolated environment, inspect connection allocation and determine whether retry behavior amplified demand. It can also investigate why the rollout checks missed the pressure on a shared dependency.
AI can organize those records into a draft postmortem, identify conflicting timestamps and point out claims that lack supporting evidence. Keep the distinction between a verified reproduction and a plausible explanation. If one part remains unresolved, name the question and its owner.
Google's postmortem guidance treats the record as a way to understand contributing causes and develop preventive action, with an emphasis on learning without blame. The draft should explain how the system allowed the failure and what can be improved.
For this incident, possible actions include a check on aggregate connections before a worker increase, an alert for pressure on the shared store and a load exercise that covers the relevant configuration. The team must choose actions based on its findings. A generic task to improve monitoring leaves the important behavior undefined.
Give each accepted action an owner, an intended result and evidence needed for closure. Updating a runbook is complete when the corrected procedure is available and usable. Adding a safeguard requires evidence that it detects or prevents the condition it addresses. Track those outcomes separately from the decision to restore service.
Keep the investigation usable across shifts
Incident knowledge often ends up spread across alerts, chat, dashboards and individual agent sessions. A Swarm Space can retain the current incident summary, evidence references, open hypotheses, decisions, agent Runs and follow up Tasks so responders can continue from the same record.
Keep raw telemetry in the systems designed to store it. Save the relevant query, time window and findings in the Space, with access appropriate to the information. A responder should be able to recover why a decision was made without copying an entire diagnostic archive into a conversation.
Our guide to agent handoffs offers a way to assess that continuity. For incident work, an incoming responder should be able to identify the current impact, actions already taken, unresolved questions and the person coordinating the next decision.
Practice on a contained incident before using live authority
Start with a past incident or a controlled response exercise. Give the assistant a bounded evidence set and ask it to maintain an accurate record as new information arrives. Look for unsupported causal claims, lost uncertainty, duplicated work and updates that imply an action succeeded before a result confirms it.
Introduce production authority only through the team's existing access, approval and change controls. The useful outcome is a response in which people can understand the situation sooner, coordinate the next action and retain what they learn. Reliable incident management depends on that shared understanding through recovery and into the work that follows.