Release Intelligence — setup & end-to-end

Business flow (end to end)

Five stages connect organizational memory to every ship decision:

Trust: Calibration suggestions are read-only — TestNeo never auto-changes your gate threshold.

The intelligence lifecycle

Teams with few tests today: start with MCP Customer E2E (generate → run), then return here for PR validation.

What is a PR validation?

One PR validation = one full review of a pull request or branch diff:

1. Impact analysis on changed files
2. Risk score (0–100) and AI release brief
3. Optional impacted test execution (uses your test runs allowance separately)
4. Evidence, fix plan, and PR comment draft

Each run is stored at /web/pr-validation/<workflow_id> — share the link in GitHub or Slack. Re-opening the same workflow does not use another credit.

Three ways to validate: MCP in Cursor/VS Code · comment @TestNeo validate on a GitHub PR (when GitHub Release Intelligence is enabled) · manual trigger in the web app.

GitHub PAT & permissions required first

Do this before you paste a token into TestNeo or ask a teammate to share repo access. Wrong scopes = empty branch list, no PR comments, failed webhook register, or silent Check Run skip.

What TestNeo needs access for

Option A — Classic PAT (best combination)

1. GitHub → profile → Settings → Developer settings → Personal access tokens → Tokens (classic)
2. Generate new token (classic) — note: not “Fine-grained”
3. Note: e.g. TestNeo Release Intelligence · Expiration: 90 days (or org policy)
4. Select scopes below → Generate token → copy once (ghp_…)
5. If the repo is under an organization with SSO: open the token → Configure SSO → Authorize for that org before pasting into TestNeo

Minimum that still works for index + manual Run: repo only. Add admin:repo_hook before you rely on auto-validation.

Option B — Fine-grained PAT (org policy)

1. GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token
2. Resource owner: you or the org that owns the repo
3. Repository access: Only select repositories → pick the app repo(s)
4. Under Repository permissions, set exactly:

Before you share access (security checklist)

• Create a dedicated token for TestNeo — do not reuse a personal “god” token
• Scope to one repo (fine-grained) or accept that classic repo covers all repos the account can see — use a machine/bot user if you need least privilege
• Set an expiration; calendar a rotate every 90 days
• Never commit the PAT to git, Slack, or ticket comments — paste only into TestNeo Connect repository / Update token
• Org SSO: Authorize the token for the org or clone/index will fail with 404/empty branches
• Self-hosted TestNeo: set GITHUB_ENCRYPTION_KEY in server .env before first connect; if the key changes, use Update token and paste a new PAT
• Revoke the token in GitHub immediately if it leaks

Where to paste the token in TestNeo

1. Open app.testneo.ai → select your project
2. PR Validation (or Release Readiness) → expand GitHub Release Intelligence
3. Step 1 — Connect repository → owner/repo (e.g. acme/shop-web) + paste PAT → connect
4. Confirm the connected repo name appears. Empty branch list → fix scopes / SSO → Update token

Next: UI walkthrough (index → memory → validate → release confidence) · auto-validation toggles.

How to use it from the UI step by step

One path from a fresh project to ship confidence. Do these in order on app.testneo.ai. MCP in Cursor is optional and parallel — webhooks do not require MCP.

Phase A — One-time setup (~30–45 min)

Phase B — Every PR (automatic after webhook)

Does not auto-run: push to a branch with no open PR. Open a PR, or use Run / MCP.

Phase C — Engineering Memory → richer PR risk

1. AI Assistant → Jira Integration (optional) → connect → pick project key (e.g. SCRUM) → Engineering Memory → Sync from Jira
2. Or Upload Bug Reports (CSV) / Add postmortem with file paths in the body
3. Patterns → Re-cluster
4. Leave Sync Jira bugs before validate ON in GitHub Release Intelligence
5. Validate a PR whose changed files overlap memory paths → expect Engineering Memory Match cards on PR detail

Matching is by changed file paths (and impact areas), not by PR number or repo name. Details: Engineering Memory.

Phase D — Release confidence (ship decision)

1. Confirm the PRs you care about are validated (PASS/WARN/BLOCK) on the PR Validation board
2. Open Release Readiness → Create Bundle (or /web/release-bundle-create)
3. Name the release (e.g. v2.4.0) → select the PR validation rows in the lookback window → create
4. Review confidence score, unlock actions, and optional AI Release Brief
5. Click Evaluate Gate (policy both recommended) → GATE_PASS / GATE_BLOCK
6. Share /web/release-readiness?bundle_id=<id> with EM/PM
7. After deploy: Mark deployed / Record outcome (rollback, incidents 7d/30d) → later review Calibration

Deep dives: Release Bundles · Outcome learning.

Quick map — which page does what

GitHub Release Intelligence shipped

Opt-in per project. When enabled, every PR open/sync (or a comment trigger) runs the same engine as MCP — including parallel impacted test execution by default. Create the PAT first: GitHub PAT & permissions. Full UI order: How to use it from the UI.

Setup (5 minutes)

1. PR Validation or Release Readiness → GitHub Release Intelligence → Step 1 — Connect repository (owner/repo + PAT — classic repo + admin:repo_hook)
2. Index from GitHub on your target branch — see Code index from GitHub (replaces ZIP for connected repos)
3. Enable auto-validation → Register webhook (or use Run on a PR # for smoke test)
4. Toggle Execute impacted tests (default on) and optional Jira sync before validate
5. On any open PR, comment @TestNeo validate to re-run

Release gate in CI: use the testneo-release-gate GitHub Action with a bundle_id — returns GATE_PASS or fails the job on GATE_BLOCK.

Code index from GitHub

No ZIP upload when your repo is connected — clone, index, and reuse across PR validations for blast radius, impacted tests, and execution evidence.

Step by step

1. Connect GitHub — Release Intelligence → Step 1 — Connect repository (owner/repo + PAT)
2. Pick branch — Code index → Branch to index (default or feature branch)
3. Index from GitHub — Wait 1–3 minutes; confirm Active index shows file/function counts
4. Validate PRs — Leave Use code impact when index is available ON in GitHub Release Intelligence

Without a code index, PR validation still runs but shows neutral execution evidence and no blast radius. Air-gapped repos: AI Assistant → Code Scanning (ZIP) or MCP testneo_sync_code_structure.

Hands-on demo — GitHub to ship readiness

~15 minutes on app.testneo.ai: connect repo → index → Engineering Memory → validate a PR.

Full customer setup: Customer PR validation flow. Self-hosted / local stack: Realworld App demo.

Setup — MCP in your IDE

1. Account & API key

Sign up → Settings → API keys (tn_…).

2. MCP config (Cursor or VS Code)

Cursor: ~/.cursor/mcp.json · VS Code + Copilot: .vscode/mcp.json in your project — then Developer: Reload Window (no npm install).

{
  "mcpServers": {
    "testneo": {
      "command": "npx",
      "args": ["-y", "@testneo/mcp-server@0.2.1"],
      "env": {
        "TESTNEO_BASE_URL": "https://app.testneo.ai",
        "TESTNEO_API_KEY": "YOUR_API_KEY",
        "TESTNEO_MCP_ALLOW_WRITE": "false"
      }
    }
  }
}

Set TESTNEO_MCP_ALLOW_WRITE to true only when you want test execution inside validation.

3. Verify

In chat: Validate my TestNeo connection or call testneo_validate_connection.

4. IDE playbooks (auto-behavior)

Optional IDE playbooks (Cursor, VS Code Copilot, and other agents) auto-detect git context and call testneo_pr_validation_workflow first when configured in your project.

Full install steps: MCP Quickstart.

End-to-end checklist (PR focus)

Greenfield (no tests yet)? Run Customer E2E first, then repeat from step 4 on every PR.

Prompt techniques (copy-paste)

Replace <PROJECT_ID> and <PR_NUMBER> with your values. More prompts: MCP Prompt Packs — PR Validation.

Validate PR <PR_NUMBER> for project <PROJECT_ID>.
Use git to detect base_sha, head_sha, and changed_files.
Call testneo_pr_validation_workflow with confirm false.
Show PASS/WARN/BLOCK, risk score, top findings, and the PR comment draft.
Give me the workflow_id link to the web board.

I'm modifying [list files] in project <PROJECT_ID>.
What's the historical failure rate and flakiness on related tests?
Have we seen BLOCK-level PRs on these paths recently?

Explain why this PR is blocked — root cause per finding.
Then give me the NOW/NEXT fix plan and which tests to rerun.

Is project <PROJECT_ID> ready to release to production?
Give me release confidence and a Go / No-Go verdict with top risks.

Show recent PR validation history for project <PROJECT_ID> —
what's the risk trend over the last two weeks?

Define release v2.4.0 for project <PROJECT_ID>.
Include these PR validation IDs: [<WF_ID_1>, <WF_ID_2>, <WF_ID_3>].
Gate policy: both. Min confidence: 85.
What do I need to fix to ship?

I'm about to change: app/services/order_service.py, app/models/order.py.
Before I push — any historical risk patterns for these files? Project <PROJECT_ID>.

Call testneo_refresh_engineering_memory_patterns with project_id <PROJECT_ID>.
Call testneo_list_engineering_memory_patterns with project_id <PROJECT_ID>.
Summarize worsening patterns, source_mix, and best_resolution for each.

Create release bundle v2.4.0 for project <PROJECT_ID> with workflow_ids [...].
Call testneo_evaluate_release_gate with bundle_id, policy both, min_confidence 85.
After deploy: testneo_record_release_outcome with rollback / incident_within_7d.
Call testneo_get_release_calibration — leadership summary.

Validate PR <PR_NUMBER> for project <PROJECT_ID>.
Use git for base_sha, head_sha, changed_files.
Call testneo_pr_validation_workflow with sync_jira_before_validate true and confirm false.
Show Historical Incident Matches and explain they match by file/area — not PR number.

Engineering Memory — Jira & CSV → live PR matches

Store historical bugs in TestNeo (bug CSV, Jira Bug/Incident issues, postmortems). When a PR is validated, memory rows are scored against that PR's changed files — there is no permanent bug-to-PR link.

Phase A flow (shipped)

QA files bug in Jira (label: orders)
     ↓
Dev validates PR (MCP / webhook / open PR in UI)
     ↓
Auto: Jira → Engineering Memory  ← Phase A
     ↓
Auto: score memory vs PR changed files
     ↓
"Engineering Memory Match" on PR board

Setup (one-time)

1. TestNeo → AI Assistant → Jira Integration → connect → select project key (e.g. SCRUM)
2. Team-managed Jira: use labels (orders, checkout) — Components may not exist
3. Or upload bug CSV with affected_area column (no Jira required)

MCP: Workflow 8 · Workflow 10 · Prompts EM1–EM9 · Tools: Engineering Memory reference

Unified patterns & trend

After ingest, Re-cluster groups memory entries into unified failure patterns across Jira, CSV, and postmortem sources. Each pattern shows:

• Occurrence count with source mix (e.g. Seen 5× (2 Jira, 1 postmortem))
• Trend badge — Worsening if last 30 days > prior 30 days; otherwise Stable
• Best-known fix from your RCAs
• Shareable drill-down: /web/engineering-memory?project_id=<ID>&pattern_id=<ID>&view=patterns

Web UI — step by step

1. Open Engineering Memory → select project → Patterns tab
2. Click Re-cluster after bulk ingest
3. Click a pattern row → detail panel with trend narrative + contributing entries
4. View contributing entries filters the Entries tab by pattern_id

PR validation link-through

On /web/pr-validation/<workflow_id>, Engineering Memory Match cards include View pattern → — opens the pattern detail with context preserved.

MCP

testneo_refresh_engineering_memory_patterns(project_id)
testneo_list_engineering_memory_patterns(project_id)          // list + trend
testneo_list_engineering_memory_patterns(project_id, pattern_id)  // detail
testneo_list_engineering_memory(project_id, pattern_id)       // contributing entries

Release Bundles — named release candidates v1.1

A Release Bundle is a named group of PR validations representing a specific release candidate — "Release v2.4.0 = these 8 PRs" instead of "the last 14 days of history". The confidence score, unlock actions, and AI Release Brief are all scoped to exactly that bundle.

What you get from a bundle

• Confidence score — 5-factor unified score scoped to exactly the PRs in the bundle
• Unlock actions — "Fix PR #278 → +8 confidence" — specific, not vague
• AI Release Brief — Go/No-Go verdict, release notes, deploy checklist, rollback plan
• CI/CD gate — API endpoint your pipeline calls; returns GATE_PASS or GATE_BLOCK
• Shareable link — /web/release-readiness?bundle_id=<id> — PM and on-call don't need Cursor

MCP tools (C22–C25)

Unified confidence formula (v1.1 — 5 factors)

CI/CD Gate policies

Override: EM-level users can pass override_reason to force-pass the gate. The override is recorded with user, timestamp, and reason for audit.

Gate + outcome stub

When testneo_evaluate_release_gate returns GATE_PASS, TestNeo automatically creates an outcome stub row you can update after deploy. Use testneo_evaluate_release_gate from MCP or Evaluate Gate in Release Readiness.

Outcome learning — ship → measure → calibrate

Close the loop after production deploy. TestNeo correlates gate decisions with post-ship signals (rollback, incidents, hotfixes) and surfaces read-only calibration suggestions — e.g. “Releases that ignored HIGH memory patterns had 3× rollback rate.”

Web UI — step by step

1. Open Release Readiness → select a release bundle (?bundle_id=<uuid>)
2. Click Evaluate Gate (GATE_PASS stubs an outcome row) or Mark bundle deployed
3. After deploy window: Release Outcome Learning → Record outcome — rollback, incident within 7d/30d, hotfix count, notes
4. After 2–3 recorded outcomes: Calibration panel shows suggestions

Note: Post-ship Jira/PagerDuty webhook setup is API/MCP advanced only in the current release — not in the web UI. Day-1 ship loops use manual Record outcome above.

MCP tools

Recipe: MCP Workflow 10 · Prompts EM5–EM8

15-minute customer demo

Quick timeline for a live call. Step-by-step setup: Hands-on demo · Code index from GitHub.

Web UI (same data as MCP)

• PR Validation board — /web/pr-validation
• PR detail — /web/pr-validation/<workflow_id> (live Engineering Memory · auto Jira sync)
• Engineering Memory — /web/engineering-memory — CSV, Jira sync, postmortem, Patterns tab with trend
• Pattern drill-down — /web/engineering-memory?project_id=<ID>&pattern_id=<ID>&view=patterns
• Branch health — Analytics → Branch (CI + PR risk overlay)
• Release Readiness — /web/release-readiness — confidence, bundle picker, Release Brief, gate, Outcome Learning panel
• Create Bundle wizard — /web/release-bundle-create — 3-step: name + policy → select PRs → create
• Shareable bundle link — /web/release-readiness?bundle_id=<id> — share with PM/on-call, no login required to view

Tool reference and JSON contracts: PR Validation & Release Confidence.

Plans & limits

Errors: release_verification_limit_reached (monthly PR validation cap) or MCP 403 (daily write cap). See MCP troubleshooting — limits.

FAQ

Current release scope

Related documentation