Start here · For everyone

Build the agent, build tests for its paths, rerun them as the agent changes, and use the same suite to validate and optimize it over time. This page walks through each stage in plain language. You don’t need to have read the rest of the series, and you don’t need any tools installed.

The ATLAS lifecycle: eight stages in a loop. 1 Understand the agent, 2 Identify paths and create scenarios, 3 Prepare test data, 4 Run and validate, 5 Change the agent, 6 Synchronize and rerun, 7 Repair regressions, 8 Sweep and optimize. Repeat as the agent evolves.
Figure 1. Eight stages that repeat as the agent evolves. The tests are saved alongside the agent, so they change when it changes instead of going stale.

Who does what

Testing an agent takes two kinds of knowledge. The business expert knows what the agent must get right. The builder knows how to make the tests prove it. Neither can do the job alone.

Business expert (owner, approver, subject-matter expert)Builder (developer, functional consultant, partner)
DefinesThe situations the agent must handle, the answers expected, what it must never do, and the policy exceptionsThe tests, their data, and the connection to Fusion
RunsNothing. No tools needed.The tests, through a coding assistant
ReviewsThe judged answers, the scoring criteria behind them, failed scenarios, and cost against the allowanceEvery report, failed check, and data issue
DecidesWhether a finding blocks release, whether a changed behavior is a defect or the new intended behavior, and the approved AI-unit allowanceHow to fix a problem without weakening the test

Together: you review failures and agree what gets fixed before release.

The eight stages

1. Understand the agent

What happens. Before any test is written, get clear on what the agent is for, who uses it, which records it reads, and anything it can change. If the agent was pulled from Fusion rather than built with the coding assistant, nothing has been tested yet, even if old tests exist.

Business expert. Describe the agent’s job in plain words: who asks it what, and what a good answer looks like. Flag anything it must never do, such as share another person’s data or update a record.

Builder. Build or pull the agent, validate it, and check whether any existing tests score the answers or only the route.

Read more: Why agents and Agentic Apps need ATLAS · Build or pull

2. Identify paths, create scenarios

What happens. ATLAS reads the agent and lists the scenarios worth testing: the main route, each outcome of each decision, fallbacks, app stages, and pauses for a person. It compares them with the tests that already exist. That list is called the sync plan.

Business expert. Add the situations only someone who knows the business would think of: risky requests, policy exceptions, and conversations that take several turns. In the Worker Concierge run, the three tests a person added found two of the real problems.

Builder. Show the sync plan, review it, and create the tests, including the custom ones the business expert asked for.

Read more: Where ATLAS lives, and how to ask · What “thorough” means

3. Prepare test data

What happens. Each test carries the data it needs. Real responses are recorded once from Fusion and replayed on later runs. Cases that are hard to reach with real data, such as an empty team or a missing record, use small made-up data. Sensitive fields can be masked.

Business expert. Point to safe, nonproduction records that represent real cases, say which fields are sensitive, and confirm before any test could create, change, or send something.

Builder. Record, generate, or reuse the data, apply masking, and keep production data and secrets out of the tests.

Read more: Test data

4. Run and validate

What happens. Every test runs. Exact checks confirm the route the agent took and any fixed messages. A judge, which is another AI model, scores the answers the agent wrote against named criteria. Results roll up into reports and a closing summary called Validation and Insights.

Business expert. Review the judged answers and the criteria behind them. A test that passes on its route alone hasn’t checked whether the answer was right.

Builder. Run the suite, read the reports from the top down, and explain each failure.

Read more: Run your tests · Review your test reports

5. Change the agent

What happens. Agents keep changing: a new policy, a new kind of question, a reworded prompt, a different AI model. When the builder makes the change through the coding assistant, ATLAS works out which tests it affects.

Business expert. Ask for changes in business terms, and say what should be different afterwards so it can be tested.

Builder. Make the change through the assistant. If the agent was changed somewhere else, ask ATLAS which tests the change affects before going further.

Read more: Build or pull · Keep tests in sync

6. Synchronize and rerun

What happens. ATLAS brings the tests in line with the changed agent. New routes get new tests, changed ones are updated, and tests for routes that no longer exist are flagged for review, not deleted. Then the whole suite runs again.

Business expert. When a test is flagged for review, decide whether that behavior was meant to go away.

Builder. Complete the test sync and rerun the full suite.

Read more: Keep tests in sync · Test an Agentic App

7. Repair regressions

What happens. A regression is something that used to work and stopped working after a change. ATLAS finds where the problem really is (the agent, the test’s input, its saved data, or a check that’s too strict) and fixes only that. It never loosens a test so it will accept a defect.

Business expert. Decide whether a changed result is a defect or the new behavior you wanted.

Builder. Fix the smallest thing that’s wrong and rerun.

Read more: Keep tests in sync · Review your test reports

8. Sweep and optimize

What happens. Once the tests pass, the same tests become a fair way to compare AI models for each step of the agent, and to catch an agent that has quietly become more expensive to run. Quality always comes first: a cheaper or faster model is recommended only if the tests and judge scores hold.

Business expert. Say how often the main situations happen, approve the AI-unit allowance, and approve any model change before it’s applied.

Builder. Run the sweep, validate the combined plan, and set up cost guardrails.

Read more: Improve cost and quality · Cost guardrails and ATLAS recommendations

Repeat as the agent evolves

Every change starts the loop again from stage 5. A new agent starts at stage 1.

What real test runs found

These come from the real runs in this series. Each one would have reached employees if only the route had been checked.

  • A green suite that checked nothing. The Worker Concierge agent’s existing tests had all passed, but they only checked which steps ran. None of them read an answer, and they covered 4 of 23 scenarios.
  • A raw error shown to an employee. For a question about their own personal details, every step ran in the right order (37 of 37 route checks passed) and the employee still got an internal error message. The judge scored the answer 1 out of 5.
  • A refusal that didn’t help. Asked to change a home address, the agent correctly changed nothing, but never said where the employee could make the change.
  • “No” treated as “yes”. An email agent heard “I do not authorize you to send this email yet” and went on to build and send the email. The test used recorded data, so nothing was actually sent. Once the agent was fixed, the same test passed unchanged.

What the evidence tells you, and what it doesn’t

  • Coverage. Tests cover each distinct behavior: every outcome of every decision, each fallback, each app stage. They don’t try every possible input. A loop gets one representative pass plus its overall result, and some scenarios need a person to design them, for example ones that take several turns or need data the test user doesn’t have. The report lists these as “unable to generate”.
  • Repeatability. Replay keeps the data the agent reads the same on every run. The AI models still run for real, so the wording of an answer can change from run to run, and a borderline answer can pass once and fail the next time. Run it more than once before you trust it.
  • The judge. The judge’s score is only as good as the criteria it scores against. Someone who knows the business should read and approve those criteria, and read the actual answer for anything involving money, policy, or personal data.
  • The live connection. A run on recorded data proves the agent behaves on known data. It doesn’t prove Fusion services answer today. A live run does.

Is it ready for business sign-off?

ATLAS gives you the evidence. The decision to release is still yours. Before you approve, check:

  • The situations you named are covered, including the risky ones.
  • Every failed test is either fixed or knowingly accepted, with the reason written down.
  • Nothing is still waiting for the judge, and you understand why any scenario was marked “unable to generate”.
  • You’ve read the judged answers for anything involving money, policy, or personal data, and you agree with the criteria.
  • The suite has passed against the published version of the agent, and a live run of the tests that can pass on live data is clean.
  • The projected cost is within the approved allowance.

Where to go next

Part of the ATLAS learning path. Back to the learning path: Test every path with ATLAS