ATLAS learning path ยท Reference

Every ATLAS term used in this series, in plain language, in one place. Each post lists the terms it relies on at the top and links here.

Tests and planning

Scenario : One behavior to test: how the agent is called, the path it should take, the data that path needs, and the expected result.

Test : A file in your project that holds one scenario, its data, and its checks.

Suite : All the tests for one workflow or app, run together.

Sync plan : ATLAS’s checklist for a workflow or app: the scenarios it should be tested on, compared with the tests you already have, each marked covered, create, update, review, or keep.

App sync plan : The same checklist for an app: each panel, the state of its panel test, and the workflow tests that must be finished first.

Test sync : Carrying out the sync plan: creating and updating tests, preparing their data, running them, and judging the results.

Custom test : A test you asked for yourself. ATLAS keeps it and never rewrites or deletes it.

Test Impact : The summary the assistant shows right after an edit, listing which tests the change affects.

Regression : Something that used to work and stopped working after a change. A regression run reruns existing tests to catch it.

Test data

Record and replay : Run once against live Fusion data and save the responses in the test, then reuse them on later runs while the agent’s logic and models still run for real.

Recorded data : Real responses captured from a live run and saved with the test.

Compacted data : Real recorded data trimmed to the rows and fields the workflow uses, when the full response is too big to store.

Generated data : Small, made-up data for paths that are hard to reach with live data, such as an empty team or a missing record.

Baseline replay : Reusing another test’s recorded data, for example for a different app stage that reads the same records.

Masking : Replacing sensitive values in recorded data with consistent stand-ins before tests are committed.

Checks and judging

Exact checks : Checks that the output equals, contains, or doesn’t contain specific text. Used for fixed messages.

Path assertions : Checks that the nodes you expect actually ran, and that the other branches did not.

Node assertions : Checks on what a node in the middle of the agent produced, such as a decision an LLM made.

Execution counts : Checks that a node ran an exact, minimum, or maximum number of times.

Judge rubric : Named criteria an LLM judge uses to score model-written answers, so wording can vary while quality is still checked.

Needs judge : A test whose exact checks passed and whose model-written output is waiting to be scored.

Unsupported claims : Statements the judge found in the output that the test data doesn’t back up.

Running and results

File mode : Running tests with their saved data. The normal choice for regression runs and pipelines.

Live mode : Running tests against the connected Fusion environment, ignoring the saved data.

DRAFT and PUBLISHED : The two versions of a workflow in AI Agent Studio. Tests can run against either; running against PUBLISHED doesn’t publish anything.

Run label : A name that keeps one run’s reports separate from the rest, such as “baseline” and “candidate”, or “red” and “green”.

Validation and Insights : The summary at the end of every run: pass and fail counts, report links, Metrics (AI units, tokens, time), and Optimization.

Test report : The report for one test: what was sent, what came back, every check, the judge’s scores, metrics, and the data used.

Suite report : The report for a whole workflow’s tests, with one row per test and links to each test report.

Consolidated report : The top-level report that rolls up an app’s panel tests and all its workflow suites.

Verification summary : The table at the top of a test report with one line per kind of check and its pass count.

AI units : How Oracle meters model usage. Reports show AI units per test and per suite.

CI/CD pipeline : The automated build, deploy, and test process (continuous integration and continuous delivery) that every change goes through before release. In this series, CI never means Content Intelligence.

Agentic Apps and conversations

App stage : The kind of request an app sends its workflow: InitDisplay when a panel first loads, Query for a typed question, Summary, and others.

Panel test : One test per top-level app panel, checking that the panel loads through its workflow and shows the expected widget.

Widget contract : The widget types a panel may or must show, such as a record table or a card list.

Conversation test : A test with several turns in one conversation, for follow-ups and for steps that pause and resume.

HUMAN and WAIT : Workflow steps that pause for a person’s decision or for a timer before continuing.

Optimization

Optimization sweep : Running the same tests with different models on each LLM node to find cheaper or faster choices that keep quality.

Candidate : One alternative model setting a sweep tries on one node, run against the same tests as the current setting.

Latency gate : A sweep rejects any candidate that makes the workflow more than 20% slower.

AI-unit budget : A limit on the AI units one test may use. Going over it fails the test or warns, as you choose.

Economic Profile : A workflow-level cost guardrail: representative tests, their expected production volume, an approved AI-unit allowance, and separate baselines for recorded and live runs.

ATLAS recommendations : An optional feature that ends each piece of work with a numbered list of next actions, each with a reason, an expected impact, and how it will be checked.

Tools

AI Studio skill : The package your coding assistant loads to build and test AI Studio artifacts. ATLAS and the AI Studio CLI come with it.

Golden dataset : METRO’s curated set of questions with expected answers approved by subject-matter experts.

LLM-as-a-Judge : A judge model that scores an agent’s answer against the expected answer and explains its score.

Part of the ATLAS learning path reference.

Back to the Learning Path for ATLAS