Release Intelligence — setup & end-to-end

Business flow (end to end)

Five stages connect organizational memory to every ship decision:

Trust: Calibration suggestions are read-only — TestNeo never auto-changes your gate threshold.

The intelligence lifecycle

Teams with few tests today: start with MCP Customer E2E (generate → run), then return here for PR validation.

What is a PR validation?

One PR validation = one full review of a pull request or branch diff:

1. Impact analysis on changed files
2. Risk score (0–100) and AI release brief
3. Optional impacted test execution (uses your test runs allowance separately)
4. Evidence, fix plan, and PR comment draft

Each run is stored at /web/pr-validation/<workflow_id> — share the link in GitHub or Slack. Re-opening the same workflow does not use another credit.

Three ways to validate: MCP in Cursor/VS Code · comment @TestNeo validate on a GitHub PR (when GitHub Release Intelligence is enabled) · manual trigger in the web app.

GitHub Release Intelligence shipped

Opt-in per project. When enabled, every PR open/sync (or a comment trigger) runs the same engine as MCP — including parallel impacted test execution by default.

Setup (5 minutes)

1. PR Validation or Release Readiness → GitHub Release Intelligence → Step 1 — Connect repository (owner/repo + PAT with repo scope)
2. Index from GitHub on your target branch — see Code index from GitHub (replaces ZIP for connected repos)
3. Enable auto-validation → Register webhook (or use Run on a PR # for smoke test)
4. Toggle Execute impacted tests (default on) and optional Jira sync before validate
5. On any open PR, comment @TestNeo validate to re-run

Release gate in CI: use the testneo-release-gate GitHub Action with a bundle_id — returns GATE_PASS or fails the job on GATE_BLOCK.

Code index from GitHub

No ZIP upload when your repo is connected — clone, index, and reuse across PR validations for blast radius, impacted tests, and execution evidence.

Step by step

1. Connect GitHub — Release Intelligence → Step 1 — Connect repository (owner/repo + PAT)
2. Pick branch — Code index → Branch to index (default or feature branch)
3. Index from GitHub — Wait 1–3 minutes; confirm Active index shows file/function counts
4. Validate PRs — Leave Use code impact when index is available ON in GitHub Release Intelligence

Without a code index, PR validation still runs but shows neutral execution evidence and no blast radius. Air-gapped repos: AI Assistant → Code Scanning (ZIP) or MCP testneo_sync_code_structure.

Hands-on demo — GitHub to ship readiness

~15 minutes on app.testneo.ai: connect repo → index → Engineering Memory → validate a PR.

Full customer setup: Customer PR validation flow. Self-hosted / local stack: Realworld App demo.

Setup — MCP in your IDE

1. Account & API key

Sign up → Settings → API keys (tn_…).

2. MCP config (Cursor or VS Code)

Cursor: ~/.cursor/mcp.json · VS Code + Copilot: .vscode/mcp.json in your project — then Developer: Reload Window (no npm install).

{
  "mcpServers": {
    "testneo": {
      "command": "npx",
      "args": ["-y", "@testneo/mcp-server@0.2.1"],
      "env": {
        "TESTNEO_BASE_URL": "https://app.testneo.ai",
        "TESTNEO_API_KEY": "YOUR_API_KEY",
        "TESTNEO_MCP_ALLOW_WRITE": "false"
      }
    }
  }
}

Set TESTNEO_MCP_ALLOW_WRITE to true only when you want test execution inside validation.

3. Verify

In chat: Validate my TestNeo connection or call testneo_validate_connection.

4. IDE playbooks (auto-behavior)

Optional IDE playbooks (Cursor, VS Code Copilot, and other agents) auto-detect git context and call testneo_pr_validation_workflow first when configured in your project.

Full install steps: MCP Quickstart.

End-to-end checklist (PR focus)

Greenfield (no tests yet)? Run Customer E2E first, then repeat from step 4 on every PR.

Prompt techniques (copy-paste)

Replace <PROJECT_ID> and <PR_NUMBER> with your values. More prompts: MCP Prompt Packs — PR Validation.

Validate PR <PR_NUMBER> for project <PROJECT_ID>.
Use git to detect base_sha, head_sha, and changed_files.
Call testneo_pr_validation_workflow with confirm false.
Show PASS/WARN/BLOCK, risk score, top findings, and the PR comment draft.
Give me the workflow_id link to the web board.

I'm modifying [list files] in project <PROJECT_ID>.
What's the historical failure rate and flakiness on related tests?
Have we seen BLOCK-level PRs on these paths recently?

Explain why this PR is blocked — root cause per finding.
Then give me the NOW/NEXT fix plan and which tests to rerun.

Is project <PROJECT_ID> ready to release to production?
Give me release confidence and a Go / No-Go verdict with top risks.

Show recent PR validation history for project <PROJECT_ID> —
what's the risk trend over the last two weeks?

Define release v2.4.0 for project <PROJECT_ID>.
Include these PR validation IDs: [<WF_ID_1>, <WF_ID_2>, <WF_ID_3>].
Gate policy: both. Min confidence: 85.
What do I need to fix to ship?

I'm about to change: app/services/order_service.py, app/models/order.py.
Before I push — any historical risk patterns for these files? Project <PROJECT_ID>.

Call testneo_refresh_engineering_memory_patterns with project_id <PROJECT_ID>.
Call testneo_list_engineering_memory_patterns with project_id <PROJECT_ID>.
Summarize worsening patterns, source_mix, and best_resolution for each.

Create release bundle v2.4.0 for project <PROJECT_ID> with workflow_ids [...].
Call testneo_evaluate_release_gate with bundle_id, policy both, min_confidence 85.
After deploy: testneo_record_release_outcome with rollback / incident_within_7d.
Call testneo_get_release_calibration — leadership summary.

Validate PR <PR_NUMBER> for project <PROJECT_ID>.
Use git for base_sha, head_sha, changed_files.
Call testneo_pr_validation_workflow with sync_jira_before_validate true and confirm false.
Show Historical Incident Matches and explain they match by file/area — not PR number.

Engineering Memory — Jira & CSV → live PR matches

Store historical bugs in TestNeo (bug CSV, Jira Bug/Incident issues, postmortems). When a PR is validated, memory rows are scored against that PR's changed files — there is no permanent bug-to-PR link.

Phase A flow (shipped)

QA files bug in Jira (label: orders)
     ↓
Dev validates PR (MCP / webhook / open PR in UI)
     ↓
Auto: Jira → Engineering Memory  ← Phase A
     ↓
Auto: score memory vs PR changed files
     ↓
"Engineering Memory Match" on PR board

Setup (one-time)

1. TestNeo → AI Assistant → Jira Integration → connect → select project key (e.g. SCRUM)
2. Team-managed Jira: use labels (orders, checkout) — Components may not exist
3. Or upload bug CSV with affected_area column (no Jira required)

MCP: Workflow 8 · Workflow 10 · Prompts EM1–EM9 · Tools: Engineering Memory reference

Unified patterns & trend

After ingest, Re-cluster groups memory entries into unified failure patterns across Jira, CSV, and postmortem sources. Each pattern shows:

• Occurrence count with source mix (e.g. Seen 5× (2 Jira, 1 postmortem))
• Trend badge — Worsening if last 30 days > prior 30 days; otherwise Stable
• Best-known fix from your RCAs
• Shareable drill-down: /web/engineering-memory?project_id=<ID>&pattern_id=<ID>&view=patterns

Web UI — step by step

1. Open Engineering Memory → select project → Patterns tab
2. Click Re-cluster after bulk ingest
3. Click a pattern row → detail panel with trend narrative + contributing entries
4. View contributing entries filters the Entries tab by pattern_id

PR validation link-through

On /web/pr-validation/<workflow_id>, Engineering Memory Match cards include View pattern → — opens the pattern detail with context preserved.

MCP

testneo_refresh_engineering_memory_patterns(project_id)
testneo_list_engineering_memory_patterns(project_id)          // list + trend
testneo_list_engineering_memory_patterns(project_id, pattern_id)  // detail
testneo_list_engineering_memory(project_id, pattern_id)       // contributing entries

Release Bundles — named release candidates v1.1

A Release Bundle is a named group of PR validations representing a specific release candidate — "Release v2.4.0 = these 8 PRs" instead of "the last 14 days of history". The confidence score, unlock actions, and AI Release Brief are all scoped to exactly that bundle.

What you get from a bundle

• Confidence score — 5-factor unified score scoped to exactly the PRs in the bundle
• Unlock actions — "Fix PR #278 → +8 confidence" — specific, not vague
• AI Release Brief — Go/No-Go verdict, release notes, deploy checklist, rollback plan
• CI/CD gate — API endpoint your pipeline calls; returns GATE_PASS or GATE_BLOCK
• Shareable link — /web/release-readiness?bundle_id=<id> — PM and on-call don't need Cursor

MCP tools (C22–C25)

Unified confidence formula (v1.1 — 5 factors)

CI/CD Gate policies

Override: EM-level users can pass override_reason to force-pass the gate. The override is recorded with user, timestamp, and reason for audit.

Gate + outcome stub

When testneo_evaluate_release_gate returns GATE_PASS, TestNeo automatically creates an outcome stub row you can update after deploy. Use testneo_evaluate_release_gate from MCP or Evaluate Gate in Release Readiness.

Outcome learning — ship → measure → calibrate

Close the loop after production deploy. TestNeo correlates gate decisions with post-ship signals (rollback, incidents, hotfixes) and surfaces read-only calibration suggestions — e.g. “Releases that ignored HIGH memory patterns had 3× rollback rate.”

Web UI — step by step

1. Open Release Readiness → select a release bundle (?bundle_id=<uuid>)
2. Click Evaluate Gate (GATE_PASS stubs an outcome row) or Mark bundle deployed
3. After deploy window: Release Outcome Learning → Record outcome — rollback, incident within 7d/30d, hotfix count, notes
4. After 2–3 recorded outcomes: Calibration panel shows suggestions

Note: Post-ship Jira/PagerDuty webhook setup is API/MCP advanced only in the current release — not in the web UI. Day-1 ship loops use manual Record outcome above.

MCP tools

Recipe: MCP Workflow 10 · Prompts EM5–EM8

15-minute customer demo

Quick timeline for a live call. Step-by-step setup: Hands-on demo · Code index from GitHub.

Web UI (same data as MCP)

• PR Validation board — /web/pr-validation
• PR detail — /web/pr-validation/<workflow_id> (live Engineering Memory · auto Jira sync)
• Engineering Memory — /web/engineering-memory — CSV, Jira sync, postmortem, Patterns tab with trend
• Pattern drill-down — /web/engineering-memory?project_id=<ID>&pattern_id=<ID>&view=patterns
• Branch health — Analytics → Branch (CI + PR risk overlay)
• Release Readiness — /web/release-readiness — confidence, bundle picker, Release Brief, gate, Outcome Learning panel
• Create Bundle wizard — /web/release-bundle-create — 3-step: name + policy → select PRs → create
• Shareable bundle link — /web/release-readiness?bundle_id=<id> — share with PM/on-call, no login required to view

Tool reference and JSON contracts: PR Validation & Release Confidence.

Plans & limits

Errors: release_verification_limit_reached (monthly PR validation cap) or MCP 403 (daily write cap). See MCP troubleshooting — limits.

FAQ

Current release scope

Related documentation