ATLAS: Cost guardrails and recommendations

Blog 13 · After the tests pass · For builders, with a business summary

A suite that passes tells you the agent behaves. It doesn’t tell you whether the agent quietly got more expensive, or what to improve next. ATLAS has two features for that: cost guardrails that treat AI-unit consumption like any other regression, and recommendations that turn everything ATLAS knows about an agent into a short list of next actions.

Part 1: Cost guardrails

An agent can keep giving the right answers while costing more to run: a longer prompt, an extra model call, a lookup that now repeats. Functional tests won’t notice. ATLAS gives you two levels of protection.

• Level 1 · An AI-unit budget on a test: Set a limit on one test. If a run uses more AI units than that, the test fails or warns, your choice. Quick to set up, local to your project, and it changes nothing in Fusion.
• Level 2 · An Economic Profile for the workflow: Connects a handful of representative tests to how often those situations happen in production, projects the AI units that volume will use, and compares it with an approved allowance. This is economic regression testing.

How an Economic Profile works

1. Pick the outcomes that matter. A few tests that represent real usage, for example the one-topic question, the two-topic question, and the greeting.
2. Say how often they happen. Expected volume per day, week, month or year for each.
3. ATLAS measures them twice. Once on recorded data and once live, and keeps a separate baseline for each, because replayed lookups don’t cost what live ones do.
4. It projects usage against an allowance and warns at the thresholds you set, for example at 70% and 90%.
5. You approve. Nothing about the profile, the volumes, the allowance or the baselines is saved or changed without your explicit approval.

After that, normal test runs can include a quick Economic Health check, so cost drift shows up in Validation and Insights next to the functional results. Three more details are worth knowing: runtime metering is the authority whenever it’s available, estimates are labelled as estimates, and in a package-wide run cost results are reported separately as advisory, so they never change the functional pass and fail totals.

Economic Setup:
Action Required: Economic Profile Not Configured
COE Worker Concierge V3 has no Economic Profile and no active runtime AI Unit allocation.
Guided setup would collect fresh replay and live measurements, project production usage,
and propose an allocation. Nothing changes and no profile is saved without your explicit approval.

Set a maximum of 10 AI units for the switch-intent-category-cmp-path test in COE_WORKER_CONCIERGE_V3. Fail it when the limit is exceeded and warn me when AI-unit evidence is unavailable. Run the test and show me the result.

Set up Economic Regression Testing for COE_WORKER_CONCIERGE_V3. Create a fresh Economic Profile using representative replay and live measurements, and propose the runtime control-plane allocation for my approval. Don't save anything until I approve.

Show me the Economic Profile summary for COE_WORKER_CONCIERGE_V3 and evaluate economic regression on recorded data. Tell me whether projected usage is within the allowance.

The assistant runs: get-economic-regression-plan · get-economic-profile-creation-proposal · do-save-economic-profile · do-establish-economic-baseline · do-evaluate-economic-regression · get-economic-profile-summary

Part 2: ATLAS recommendations

Once tests, sweeps and cost checks have run, ATLAS knows a lot about an agent: its structure, which paths are covered, which checks pass, what the judge said, which model each step uses, and what each run costs. Recommendations turn that into a short, numbered list of things to do next. Each one says why, what it should improve, and how the result will be checked.

• What it can suggest: Fix a quality problem, add a missing test, remove repeated work, trim the data a step receives, move a step to a better model, or look into a cost concern.
• When it appears: Automatically, right after Validation and Insights, once a meaningful piece of work finishes. You can also ask for the list any time.
• How to turn it on: It’s off unless your workspace enables it. Add aistudio.atlas-recommendations=true to env.properties in your project.

Picking one

You choose in plain language, for example “do recommendation 2”. The model ranks and explains the options; the CLI is what checks that the evidence is still current, validates the action, and enforces approvals. A source change comes back as a focused proposal for you to review. An exact model placement that a sweep already validated can be approved directly. Anything that touches a cost allowance, profile or baseline keeps its own approval step. After the action runs and passes validation, ATLAS rebuilds the list, and if you change the workflow in the meantime, the old suggestions are dropped as out of date.

Show me the recommended actions for COE_WORKER_CONCIERGE_V3. Don't run any of them yet.

Do recommendation 1. Show me the proposal before you change anything.

The assistant runs: get-atlas-recommendations-status · prepare-atlas-recommendations · prepare-atlas-recommended-action · do-execute-atlas-recommended-action

Recommendations only run in your interactive sessions. Pipeline runs execute the committed tests and stop there: they never start recommendation analysis or repair tests to get a pass.