Flaky test investigation
Collect failed runs, cluster symptoms, reproduce the race, and prepare the smallest verified fix.
What this workflow does
Collect failed runs, cluster symptoms, reproduce the race, and prepare the smallest verified fix. It separates observed facts from inference, surfaces conflicting evidence, and delivers a review-ready finding set instead of an opaque answer.
What you gain
Intermittent CI failures become actionable instead of being retried indefinitely.
What the AI agent changes
How this worked before
Traditional automation could collect records, but a person still had to compare sources, resolve conflicts, and write the conclusion.
What the AI agent changes
The agent can plan a search, inspect multiple sources, distinguish facts from inference, and return a cited report.
Agent trigger
A test alternates between pass and fail across comparable CI runs.
Inputs
- Process context: CI logs, timings, seeds, environment, test code, and recent changes
- Approved policies, ownership, and exception rules
Agent flow
- 1
Collect source material from the approved systems and record its timestamp.
- 2
Cross-check conflicting signals and separate facts from inference.
- 3
Prepare a reproduced cause, confidence, and minimal repair with citations, unknowns, and confidence.
- 4
Deliver the approved report and preserve its evidence set.
Human decisions
After step 3
The accountable process owner approves a reproduced cause, confidence, and minimal repair.
Outcome
- Engineer-reviewed flaky-test fix
- Evidence, exceptions, and audit trail
Guardrails
- Label inference separately from observed facts.
- Do not close the investigation while required sources are unavailable.
Risks and mitigations
Missing or stale evidence can produce a confident but incomplete finding.
Show source coverage, conflicts, timestamps, and unanswered questions.
Sources and evidence
Sources establish feasibility or impact. Not every metric comes from an identical implementation.
OpenAI
New tools for building agents
OpenAI documents agents combining web search, files, browser control, tool use, handoffs, guardrails, and traceable multi-step execution.
Langfuse
Evaluating AI agent skills
Langfuse shows agents turning production traces and human feedback into datasets, experiments, and measurable regression checks.