---
title: The Agent QA Stack That Saves Automation
summary: Agents do not just write text. They make decisions. Build a testable QA stack that checks format, meaning, procedure and cost before a workflow touches the CRM or the CMS.
lede: Test the tools, the loops and the receipts before it ships.
date: 2026-08-16
updated: 2026-08-16
authors: Team COEY
image: /blog/the-agent-qa-stack-that-saves-automation.webp
image_alt: A receipt scroll moving through gears while four QA sentinels inspect structured outputs
keywords: Tools and How-Tos
canonical: /resources/blog/2026/08/16/the-agent-qa-stack-that-saves-automation
source: https://coey.com/resources/blog/2026/08/16/the-agent-qa-stack-that-saves-automation/
---

## The new automation flex is not agents, it is receipts

Everyone wants the latest AI agent. Nobody dreams about the Slack postmortem, phantom costs, or the moment somebody asks what it just did to the lead database. Modern marketing automation is now built on multistep, agent-powered workflows: sourcing data, classifying, updating, shipping, posting. The problem is not that agents are hype. The problem is that most teams race toward "agentic" before they have "operational" down.

Operational, in marketing, means predictable outputs, controlled spend, and receipts you do not have to reconstruct from three log files.

Instead of betting credibility on a vendor demo, anchor agent QA in concrete, testable failure modes. One of the expensive ones is infinite looping: the agent keeps replanning and re-calling tools until the budget becomes a forensic accounting project. A 2026 paper, When Agents Do Not Stop, shows how static analysis can surface these loop risks before you deploy.

```callout
title: The thesis
The new game in AI marketing automation is agent QA. The win is a testable, automated stack: tracking tool calls, catching infinite loops, validating structured outputs, and routing true risk to people before your budget and brand become collateral.
```

## Why this suddenly matters

Remember when AI in marketing meant a catchier subject line or a meeting summary? That is prehistoric. Now it means update CRM fields, generate assets, open tickets, publish to the CMS and launch campaign flows on autopilot. The failure modes shifted with it.

- Old failure: the subject line is vanilla.
- New failure: the agent performed five tool calls, updated the wrong record, output technically valid JSON, and convinced you everything was fine.

If the business runs on scaled publishing, personalized outreach or automated sales ops, the threat is not a hallucinated summary. It is a quiet procedural mistake, hard to spot and brutal to fix after the fact.

## The agent problem in one sentence

Agents do not just write text. They make decisions for you.

The relevant questions have shifted from "does it sound good" to:

- Did the agent use the intended tools, in the right order?
- Did it hand off correct arguments?
- Did it interpret the tool's output properly?
- Did it stop or escalate at the right moment?
- Can you reconstruct and prove what actually happened?

## Recent research confirms what ops teams already know

Correctness is not binary, and "it worked last week" is not a reliability plan. Two research threads stand out:

- Procedure-aware evaluation: agents can "complete" a workflow by skipping steps, reaching an apparently correct result through the wrong process.
- Loop and cost drift: left unchecked, agents retry, replan or endlessly re-execute tool calls until the budget is a post-mortem.

The loop risk is not theoretical. The 2026 work linked above shows how loop patterns show up in real agent repositories and how static analysis can flag them before production.

## Agent QA is four interlocking tests

The real-world options today:

- Most teams ship agents with no QA, cross fingers, and keep the calendar clear for incident response.
- Others over-correct by sticking a human in every loop, grinding the time savings into confetti.

Agent QA is the middle: automated checks for the fatal but predictable failures, and gated human review only on true edge cases.

| QA Layer | What it checks | What it catches |
| --- | --- | --- |
| Format QA | Schema validity, required fields, type correctness | Parsing breaks, downstream crashers |
| Semantic QA | Valid field values, enums match, IDs exist | Valid-but-wrong updates, mismatches |
| Procedural QA | Steps, order, checks, policy adherence | Success that skipped required steps |
| Economic QA | Caps, budgets, retries, tool call limits | Infinite loops, stealth cost overruns |

```stats
Format QA | Layer 1 | Schema and required fields
Semantic QA | Layer 2 | Valid, and also correct
Procedural QA | Layer 3 | The steps actually ran
Economic QA | Layer 4 | Loops and spend stay capped
```

## The dirty secret of structured outputs

Structured outputs are catnip for automation because JSON parses, tables import and logs look tidy. Parsing is not truth. You can have flawless schema adherence and outputs that are spectacularly incorrect, which is worse because they flow downstream cleanly.

```quote
Perfectly structured, terribly wrong is the new silent failure mode for agentic workflows.
```

This is why structured outputs have to travel with active verification.

## Reference architecture

QA cannot be a checkbox. It is a control layer, embedded in every workflow.

```
[Trigger]
  - form submitted, CRM event, scheduled process
[Normalize]
  - generate a machine-readable context object
[Plan]
  - agent drafts a tool plan, step by step
[Validate plan]
  - run procedural, risk and tool allowlist checks
[Execute]
  - execute and log tool calls with input and output
[Validate outputs]
  - run format and semantic QA
[Loop control]
  - enforce budget and retry caps, detect infinite loops
[Route]
  - auto-publish low risk, hold or escalate everything else
[Publish]
  - perform official updates (CMS, CRM, tasks)
[Receipts]
  - capture the whole trail
```

## The core artifact: the tool plan object

Want to test agents? Force them to show their plan before execution. A formal tool plan that is machine-readable, saved and version-controlled lets you catch dumb plans before the agent touches real data.

```
{
  "workflow_id": "",
  "risk_tier": "low|medium|high",
  "tool_plan": [
    {
      "step": 1,
      "tool": "crm.lookup_contact",
      "args": {"email": ""},
      "required": true
    },
    {
      "step": 2,
      "tool": "llm.generate_asset_pack",
      "args": {"schema": "EmailVariantV2"},
      "required": true
    },
    {
      "step": 3,
      "tool": "crm.update_contact",
      "args": {"contact_id": "", "fields": {}},
      "required": false
    }
  ],
  "stop_conditions": {
    "max_tool_calls": 12,
    "max_retries": 2,
    "max_cost_usd": 2.00
  }
}
```

## Economic QA

Agentic workflows blow up in bursts. One mistake, one token loop, and a spot budget is gone.

- Set output, retry and cost caps for every run
- Abort repeated, identical tool calls
- Auto-escalate to a human with log receipts and a short error summary

"It ran four thousand times because the endpoint threw a 404" is not a joke. It is a recurring incident category.

## Procedural QA

What is worse than a failed workflow? One that "succeeds" but cut corners in silent, brand-ending ways.

- Publishing announcements without checking regulatory flags
- Updating CRM status without running suppression checks
- Outreach blasts honoring the wrong compliance region
- Generating ad copy with missing required disclosures

Represent process rules as machine-checkable constraints. Force the agent to pass, or escalate.

| Procedure rule | Automation check | Failure response |
| --- | --- | --- |
| Consent required before outreach | `tool_plan` includes consent verification | Halt workflow, flag record |
| Claims require explicit approval | `claim_ids` in output are on the approval list | Reject and block publish |
| High-risk tier mandates review | `risk_tier` high triggers manual approval | Create an urgent review task |

## Semantic QA

Semantic QA is the difference between "looks valid" and "is correct." The structured output parsed. The field types align. And you attached an expired offer ID, a region code for the wrong locale, or a pricing claim nobody has updated.

Watch:

- Does the CRM field actually exist in production?
- Does that offer or partner ID map to something live?
- Do URLs resolve, and are they allowlisted?
- Is that pricing backed by current official tables?
- Did we flip the wrong region or language rule?

Let QA code validate these first. Never let the agent self-certify.

## Hybrid flows win because irreversibility is real

Once published, a mistake is screenshot, archived and turned into brand lore. High-stakes, fully autonomous agents are still unrealistic for most organizations. The sustainable pattern remains:

- AI drafts and proposes.
- Automation validates, logs and receipts.
- Humans approve only exceptions and high-stakes items.

## What to automate first

1. Structured intake objects
2. Tool plan declarations before execution
3. Schema validation on every agent output
4. Semantic validation on field values
5. Loop and budget caps
6. Risk routing
7. Receipts for every attempt

## What success looks like

You will know agent QA is working when:

- most runs complete with no human involvement
- failures are caught early and routed with context
- human review time drops, and the items they see are high-signal
- costs get boring, because loops are capped
- when a CMO asks why it did that, you have a timeline, not a war story

```quote
Automation first does not mean human absent. It means humans get their time back for judgment and accountability, not for solving mysteries.
```