Product guide

AI Agent Verification

Your agent says it worked. TestNeo proves it.
Wire CrewAI, LangGraph, AutoGen, or a custom agent with the TestNeo Agent SDK. We ingest a thin run summary, verify tools / RAG / memory / (when relevant) live APIs, and return PASS / WARN / BLOCK in Agent Verification.

In one line: Claimed → Observed → Verified. Gate the run before you ship — not another LLM-as-judge.

Your agent finishes → TestNeo Agent SDK (or MCP / REST) ingests summary → Agent Verification runs the gate → Claimed / Observed / Verified → PASS | WARN | BLOCK

Start here: install the SDK, ingest one run, open Verify (~15 minutes).

Jump to quickstart →

How to read results

Term	Meaning
Claimed	What the agent said (`agent_claim`)
Observed	Tools, retrieval, memory, APIs, handoffs from the run
Verified	Independent checks (contracts, RAG/memory oracles, HTTP probes, suites)
PASS / WARN / BLOCK	Ship decision — not a similarity score

If something important could not be verified (for example a refund was claimed but not checked in the app), you will not get a silent PASS.

Two lanes

Same SDK and Verify UI. Different oracles — pick the lane that matches your agent.

Lane	Agent type	What TestNeo verifies	Generate NLP + execute?
A — Product / API	Refund, checkout, tools that hit your staging APIs	Tools + RAG/memory + live HTTP prove-after-write	Yes — API NLP from observed HTTP
B — Research / content	Trip planner, deep research, writers	Tools + retrieval + claim/artifact contract	Usually no — gate only (no invented UI tests)

Quickstart (15 minutes)

Prerequisites

Account + API key at app.testneo.ai (tn_…)
A TestNeo project id
Python 3.10+

Install & env

pip install testneo-agent-sdk

export TESTNEO_BASE_URL=https://app.testneo.ai
export TESTNEO_API_KEY=tn_YOUR_KEY
export TESTNEO_PROJECT_ID=YOUR_PROJECT_ID

Minimal Python

from testneo_agent import TestNeo

sdk = TestNeo.from_env()

with sdk.run(goal="Refund order 430 for the authenticated customer") as run:
    run.authorization("READ_WRITE")
    cid = run.tool_called("search_policy_kb", operation="READ", target="policy_refund_v2")
    run.retrieval(
        doc_id="policy_refund_v2",
        snippet="Full refunds only within 30 days of purchase.",
        emit_action=False,
    )
    run.tool_completed(call_id=cid, result_summary="matched policy_refund_v2")
    run.confirmation(True)
    run.set_claim("Full refund within 30 days per policy v2.")
# ingest on exit when TESTNEO_AUTO_INGEST=true (default)

CrewAI (after kickoff)

from testneo_agent import TestNeo
from testneo_agent.adapters import CrewAIAdapter

sdk = TestNeo.from_env()
with sdk.run(goal="Plan trip from NYC; choose Paris or Rome", source="crewai") as run:
    adapter = CrewAIAdapter()
    adapter.bind(run)
    adapter.from_kickoff(
        tool_calls=[...],      # search / product tools
        handoffs=[...],
        final_output=result,   # claim
    )

Adapters also exist for LangGraph, AutoGen, and Generic event dicts. See the SDK on PyPI.

Then open the UI: project → Agent Verification → select the ingested run → Run verification → read PASS / WARN / BLOCK.

More paths (curl, MCP, n8n, mock refund probe): How to integrate.

UI how-to

Step	Where	What you should see
1	Sidebar → Agent Verification	Project-scoped board
2	Ingested runs	Contexts from SDK / MCP / REST
3	Open a run	Goal, claim, tools, retrieval, APIs
4	Run verification	Gate risk, timeline, Layer 4 findings
5	Eval (if enabled)	Tools / RAG / Memory lenses
6	Semantics / Unified Context	Generate API tests only for HTTP surfaces

Tool / memory / RAG-only nodes are for Verify — not fake browser NLP. When the run touched real HTTP, you can generate deterministic API tests from that evidence.

Validate RAG, memory, and Eval

They ask	Emit / configure	Where it shows
Validate	Summary + `gate_contract`	PASS / WARN / BLOCK
RAG	`retrieved[]` + `expected_retrieval`	Layer 4 + Eval RAG lens
Memory	Memory accesses + memory policy	Layer 4 + Eval Memory lens
Eval	Repeated runs / goldens	Eval-lite (Tools / RAG / Memory)

Soft writing-quality judges (Opik-style) are optional later and never override a BLOCK.

Architecture

YOUR AGENT (CrewAI / LangGraph / AutoGen / custom)
        │  TestNeo Agent SDK (+ adapters)
        ▼
agent_run_summary.v1  →  ingest
        ▼
Agent Verification UI
  Claimed → Observed → Verified
  Gate: PASS / WARN / BLOCK
        │
        ├─ Lane A: optional API NLP suite + execute on staging
        └─ Lane B: research crews stop at gate
        ▼
Optional: PR validation / release gate (same project)

TestNeo does not run your CrewAI/LangGraph process. Your agent runs on your infrastructure; TestNeo proves outcomes and gates release.

Prove HTTP outcomes (Lane A)

When the agent performs a write (refund, transfer, update), TestNeo can independently query your application — for example confirm the refund exists with the right amount and status.

Provide a staging base URL (in the UI or in the gate request). Without a reachable check, that write stays unverified and the gate elevates to WARN — by design.

Try it locally: sample integration includes a tiny mock refund API. Steps: Quickstart recipe.

Golden runs & safety packs

Golden run — save a good path; later runs can be compared for unexpected tools or missing confirmation
Journeys — require a tool sequence for a product flow
Safety packs / Attack — built-in scenarios that check the verifier itself (they do not call your live agent)

PRs & releases

Outcome verification pairs with Release Intelligence when you also need PR risk and release bundles. Tag product tests so the agent gate can run them as part of the same decision (Lane A).

FAQ

Do you replace LangSmith / Opik?: No. They help debug and score model behavior. TestNeo verifies outcomes and gates release.
Do we need golden NLP tests for every agent?: Only for Lane A (product APIs/UI). Research/content crews can gate on tools + retrieval + claim alone.
What do you collect?: Thin evidence: tools, short retrieval snippets, memory keys, final claim, API observations, errors, gate_contract — not full prompts or token dumps.
Can we generate tests from an agent run?: Yes for HTTP surfaces (deterministic API NLP). We do not invent browser UI from API/tool evidence.
Is quality / LLM-as-judge scoring included?: The outcome gate is the product. Optional advisory quality scoring may come later; it never overrides BLOCK.

Checklist

pip install testneo-agent-sdk + env vars set
Agent emits tools/actions (not only the final message)
RAG/memory runs include retrieved / memory fields when those matter
Destructive actions include confirmation correctly
Staging URL available when you need prove-after-write (Lane A)
QA can re-run the gate from the UI
Optional: CI fails the job on BLOCK

AI Agent Verification

How to read results

Two lanes

Quickstart (15 minutes)

Prerequisites

Install & env

Minimal Python

CrewAI (after kickoff)

UI how-to

Validate RAG, memory, and Eval

Architecture

Prove HTTP outcomes (Lane A)

Golden runs & safety packs

PRs & releases

FAQ

Checklist

Related