Blog 1 · The testing challenge · For everyone

Ask an agent one question and it may take a completely different route than it does for the next. A few test questions prove only those questions. ATLAS is how you test the rest, and keep testing as the agent changes.

After reading this, you’ll be able to: explain why a passing test count doesn’t prove the answers are right, describe what ATLAS checks and what you bring to it, and read a sync plan.

One agent, four questions

The Worker Concierge agent answers employees’ questions about their own pay, compensation, benefits, leave, HR policy, and personal details. Picture four messages from the same employee:

  • “What is my current base salary?” A normal request. The agent should find the employee’s own record and answer with the right amount.
  • “What is my manager’s salary?” A request it must turn down. The agent should decline and share nothing about anyone else.
  • “Change my home address to 123 Main Street.” A request it must not act on. It should explain that it only answers questions, and say where the change can be made.
  • “Switch to my other assignment.” Then: “The second one.” A follow-up. The agent has to remember what it just offered and carry the choice into the next answer.

Each message sends the agent down a different route: a different specialist, a different check, a different reply. Testing the first one tells you nothing about the other three.

Here’s what a real test run of this agent found:

  • The tests already in place had all passed. They only checked which steps ran; none of them read an answer.
  • For a question about the employee’s personal details, every step ran in the right order and the employee still got a raw system error. Only the check that reads the answer caught it.
  • The agent correctly refused the address change, but never said where to make it.
  • “The second one” wasn’t understood when switching assignments.

Blog 8 walks through that run step by step.

Why it gets harder as the agent grows

Behind those four messages, an agent built in AI Studio, whether it runs on its own or behind an Agentic App panel, is an execution graph. Code, IF and SWITCH logic, loops, parallel branches, LLMs and agents, sub-workflows, RAG and vector retrieval, and enterprise data operations connect into many possible paths, depending on inputs, data, conditions, and intermediate results.

One agent with four distinct execution paths: a fallback, a loop with an LLM, a sub-workflow, and retrieval with an agent
Figure 1. Even one agent has many execution paths. Each one needs the right input and the right data to reach it.

Testing an agent like this takes more than a few input-and-output checks. You need to know the paths, decide which ones need coverage, write tests that reach them, prepare the data each one needs, and validate what comes back. Then the agent changes: a new node, a changed condition, an updated prompt, a different LLM, or an edited sub-workflow. New paths appear, old ones shift, and existing tests go stale. The suite has to change with the agent and run continuously.

What “every meaningful path” means

ATLAS aims to test each distinct behavior an agent can show: the main route, every outcome of every IF and SWITCH, each fallback, each app stage, and each pause for a person. It doesn’t try every possible input. A loop gets one representative pass plus a check on its overall result, and a sub-workflow gets its own tests rather than being retested through every parent that calls it. Blog 9 has the checklist for each kind of node.

The questions you need to keep answering

  • What are the meaningful paths through this agent, and which ones already have tests?
  • What data does each path need, and will that data still exist in the next environment?
  • What changed when the agent was updated, and which tests now need to change?
  • Did the latest change break behavior somewhere else in the graph?
  • Is the AI-generated output still correct, even when the wording differs run to run?
  • Which model or configuration gives the best result for the cost and latency?

What ATLAS is

ATLAS (Agentic Testing and Lifecycle Automation Suite) is a testing and simulation framework for agents and Agentic Apps built with AI Studio. It creates, runs, repairs, and reports source-controlled tests throughout the development lifecycle. Each test is organized around a scenario: how the agent is invoked, the path expected to execute, the data that path needs, deterministic expectations, and semantic criteria for AI-generated output.

Builders work through an AI coding assistant such as Codex or Claude. They describe the artifact, the coverage, and the outcome they need; the assistant uses ATLAS to do the work. ATLAS ships as part of the AI Studio skill and runs on the same AI Studio CLI used to build the agent.

Business experts don’t need any of those tools. They bring the situations that matter, the answers they expect, and the things the agent must never do. The full ATLAS lifecycle shows who does what at each stage.

Three ideas behind ATLAS

Fix the data, not the thinking. Mock everything and a test is stable but only proves your mocks work. Keep everything live and you can’t tell whether a failure came from your change or from last night’s data. ATLAS sits in between: it holds the data an agent reads steady and lets every model, agent and decision run for real. The white paper sums it up as “replay the context but execute the intelligence”.

Count behaviors, not tests. Five branches are five business situations the agent has to handle, not five expected strings. The useful question is which distinct behaviors you’ve shown working, not how many tests you have. A few well-chosen scenarios beat many prompts that all take the same route.

Exact where it must be, judged where it can vary. Which steps ran, which branch stayed off, the shape of the output: those are facts, so they get exact checks. Whether an answer is grounded, complete and useful can be phrased many ways, so a judge scores it against criteria instead of a fixed string.

These ideas are set out in more depth in the white paper Testing Agentic Applications.

Glossary

Every ATLAS term used in this series is defined in one place: the ATLAS glossary. Each post lists the terms it relies on at the top.

The sync plan: where every ATLAS request starts

You will see the term sync plan throughout this series, so here it is in full. It is ATLAS’s checklist for one workflow or app: the business scenarios that are already covered, and the ones that still need tests. ATLAS reads the current version, works out the scenarios worth testing (the main path, each IF and SWITCH outcome, fallbacks, app stages, loops, and pauses for a person), and compares them with the tests already in your project.

ATLAS compares the scenarios it discovers in the current workflow with the tests already in the project, and lists an action for each in the sync plan
Figure 2. The sync plan lines up what the workflow needs with what the project already has.
What the plan saysWhat it means
[covered]A current test already proves this scenario. Nothing to do.
[create]No test exists for this scenario yet. ATLAS proposes an input and a data approach for it.
[update]The workflow changed in a way that affects this test, so the test needs updating.
[review]An existing test no longer matches any current path, often because a branch was removed. It’s kept until you decide.
[keep]A custom test you asked for. ATLAS never rewrites or deletes it.

It’s called a sync plan because its job is to keep your tests in sync with the workflow. Asking to show the plan changes nothing. Asking to run or complete it has the assistant carry out every action, run the tests, and report back. Each time the workflow changes, the plan is worked out again. Apps have their own app sync plan, which lists each panel, the state of its panel test, and the backing-workflow tests that need finishing first.

See a sync plan

Show me the workflow test sync plan for my workflow. Explain each scenario, whether it is covered, missing, out of date, or needs review, and the data you would use. Do not create or run anything yet.

Record once, replay anywhere

The Business Object node replays from the test file while condition, LLM and return nodes still execute
Figure 3. Replay is not a replay of the whole run. Only supported data-producing boundaries return recorded responses; the behavior you are testing still executes.

A test built with a particular manager and team keeps the relevant responses as test data, so it can run again in another environment, or after the live team changes, without depending on that population. When the current environment itself is what you want to check, you can still run live.

What replay makes repeatable, and what it doesn’t:

  • Repeatable: the data. Every run reads the same saved responses, so when a test starts failing, the change is in the agent, not in last night’s data.
  • Not repeatable: the wording. The models still run for real, so an answer can be worded differently each time. That’s why a judge scores answers against criteria instead of matching exact text. A borderline answer can pass one run and fail the next, so run it more than once before you trust it.
  • Not checked: the live connection. A recorded-data run doesn’t prove the Fusion services the agent calls still answer today. Keep a live run in your routine for that. Blog 6 covers both.

What ATLAS gives you

  • Scenarios · Tests for every meaningful path: Primary paths, condition and switch outcomes, fallbacks, data conditions, trigger variations, and app stages such as InitDisplay, Query, and Summary.
  • Data · Flexible record and replay: Real recorded data for representative paths, generated data for hard-to-reach ones, and compaction for oversized responses.
  • Validation · Deterministic and semantic checks: Exact checks for routes and fixed messages; judge rubrics for LLM, Agent, and Multi-Agent output.
  • Sync · A suite that follows the agent: New paths get tests, changed paths get updated tests, and unmatched tests are surfaced for review.
  • Regression · Continuous runs and reports: Source-controlled tests run locally and in your CI/CD pipeline, rolling up into workflow, app, and consolidated reports.
  • Optimization · Sweeps on the same scenarios: Compare models and configurations on accuracy, latency, tokens, and cost under identical conditions.

Key takeaways

  • A passing test count isn’t proof the answers are right. Route checks show which steps ran; only the judge reads the answer.
  • Every agent is an execution graph with many paths, and each path needs the right input and the right data to reach it. The test suite has to change with the agent and run on every change, not once.
  • ATLAS replays data at the boundaries while conditions, code, LLMs, and agents still run, so tests are repeatable without becoming shallow mocks. The data stays the same; the wording of answers can still vary.

Part of the ATLAS learning path.

Next: ATLAS or METRO?

Back to the learning path: Test every path with ATLAS