TestNeo CLI v0.3.0

Release assurance in your terminal — run .testneo tests, pick cases from Test Manager, validate PRs, run API Risk Discovery (paste cURL), query Engineering Memory, check release readiness, and Ask the AI Assistant. Same account and projects as app.testneo.ai and MCP.

npm install -g @testneo/cli@0.3.0
testneo login
testneo config set baseUrl https://app.testneo.ai
testneo config set webAppUrl https://app.testneo.ai
testneo config set projectId YOUR_PROJECT_ID
testneo config set defaultEnvironment dev
testneo doctor

# Run an exported .testneo file (cloud execution — safe default for most users)
testneo run ./login.testneo --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error

# Before you merge (from your application git repo)
testneo pr validate --project YOUR_PROJECT_ID
testneo readiness --project YOUR_PROJECT_ID
testneo ask --project YOUR_PROJECT_ID "What is my release readiness?"

Get started free (14-day trial) · Release Intelligence product overview

Also from TestNeo — pick your surface

Getting started

Point the CLI at https://app.testneo.ai. Do this once.

1) Sign up & create a project

1. app.testneo.ai/signup — create an account (14-day trial).
2. Projects → Create project.
3. Note your project ID (shown in the app or via testneo doctor).

2) Environment + API key

1. Project → Environments → create dev with a base_url for the app under test (example: https://www.saucedemo.com).
2. Settings → API keys → create a key (tn_…).

3) Install & configure

npm install -g @testneo/cli@0.3.0

testneo login
# or: testneo config set apiKey tn_your_key

testneo config set baseUrl https://app.testneo.ai
testneo config set webAppUrl https://app.testneo.ai
testneo config set projectId YOUR_PROJECT_ID
testneo config set defaultEnvironment dev
testneo doctor

Expect ✓ API reachable and your projects listed. Dashboard links in CLI output will use https://app.testneo.ai/….

4) Export tests from Test Manager (recommended)

1. Open Test Manager → select cases → Export → TestNeo CLI → unzip.
2. Run the exported file (see examples below).

Which command should I use?

Sample project journey (end-to-end)

Walkthrough for a typical web project (Sauce Demo–style login + catalog). Replace YOUR_PROJECT_ID with your ID from testneo doctor.

1. 1 · Smoke — Prove auth & execution with run / try.
2. 2 · Inventory — Ask “how many tests?” / “which tests cover login?”
3. 3 · Diff risk — From your app git repo: memory match + pr validate.
4. 4 · Ship decision — readiness + Ask “safe to ship?” / release decision report.
5. 5 · CI — Same commands in GitHub Actions with secrets.

P=YOUR_PROJECT_ID   # from: testneo doctor

# --- 1) Smoke (cloud — works without a local agent) ---
testneo try --project $P --env dev --cloud --wait --fail-on-error \
  --url https://www.saucedemo.com \
  "Enter username as standard_user
Enter password as secret_sauce
Click Login
Verify Products is visible"

# Or run an exported / example file
testneo run "$(npm root -g)/@testneo/cli/examples/saucedemo/login-explicit-url.testneo" \
  --project $P --env dev --cloud --wait --fail-on-error

# Or run existing Test Manager cases (no new cases created)
testneo run --pick --project $P --cloud --wait --fail-on-error
# testneo run --test-case 12,34 --project $P --cloud --wait --fail-on-error

# --- 2) Inventory (Ask) ---
testneo ask --project $P "How many test cases do I have?"
testneo ask --project $P "Which tests cover login?"
testneo ask --project $P "List all test cases"
testneo ask --project $P --chat    # interactive; type questions, exit to quit


# --- 2) API Risk Discovery (FakeStore sample after npm install) ---
testneo api-risk scan --project $P \
  --file "$(npm root -g)/@testneo/cli/examples/api-risk/fakestore-products.curl" \
  --cloud --fail-on-critical --brief-out api-confidence.md
# Promote failures → Risk_* NLP + Engineering Memory (write guard)
# testneo api-risk run --project $P -f that.curl --cloud --json > run.json
# testneo api-risk promote --project $P --run-file run.json --confirm

# --- 3) PR / memory (run from the *application* git repo) ---
cd /path/to/your-app-repo
testneo memory match --project $P
testneo pr validate --project $P
# optional: execute impacted tests on cloud (write guard)
testneo pr validate --project $P --run-impacted --confirm --cloud --wait --fail-on-error
testneo pr history --project $P

# --- 4) Release readiness ---
testneo readiness --project $P
testneo ask --project $P "What is my release readiness?"
testneo ask --project $P "Show release metrics"
testneo ask --project $P "Generate release decision report"
testneo ask --project $P "Is it safe to ship?"

Run · pick · try

All three execute NLP against your project on app.testneo.ai. Prefer --cloud unless you intentionally use a local agent (next section).

File / folder run

# Cloud (recommended default)
testneo run ./login.testneo --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error
testneo run ./tests/smoke/ --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error

# Local agent (headed browser on your machine — agent must be online)
testneo run ./login.testneo --project YOUR_PROJECT_ID --env dev --use-agent --wait --fail-on-error

Bundled Sauce Demo sample (after global install):

testneo run "$(npm root -g)/@testneo/cli/examples/saucedemo/login-explicit-url.testneo" \
  --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error

Pick Test Manager cases (TTY)

testneo run --pick --project YOUR_PROJECT_ID --cloud --wait --fail-on-error
# filter: testneo run --pick --search login --cloud --wait

Run by ID (no duplicate cases — best for CI)

testneo run --test-case 12,34 --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error

One-shot natural language (try)

No .testneo file — NLP as arguments. Upserts a case by name, then executes.

testneo try --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error \
  --url https://www.saucedemo.com \
  "Enter username as standard_user
Enter password as secret_sauce
Click Login
Verify Products is visible"

Live steps stream in the terminal; open the printed dashboard URL on app.testneo.ai.

Status

testneo status EXECUTION_ID --wait

Local agent vs cloud

Every execute prints a line like execution: cloud / EC2 (--cloud) or execution: local agent #12 … so you always know where the browser ran.

# Check whether a local agent is registered / online
testneo agent status
testneo doctor

# --- Cloud (pin it — clearest for demos & CI) ---
testneo run ./login.testneo --project YOUR_PROJECT_ID --env dev --cloud --wait --fail-on-error
testneo run --pick --project YOUR_PROJECT_ID --cloud --wait
testneo try --project YOUR_PROJECT_ID --cloud --url https://www.saucedemo.com \
  "Verify Products is visible" --wait

# --- Local agent ---
# 1) In app.testneo.ai → Settings → Agent → create & start the agent
# 2) Confirm:
testneo agent status
# 3) Run:
testneo run ./login.testneo --project YOUR_PROJECT_ID --env dev --use-agent --wait --fail-on-error
testneo run --pick --project YOUR_PROJECT_ID --use-agent --wait --fail-on-error

API Risk Discovery — paste cURL

Deterministic risk variants from a single cURL (no LLM). Same backend as API Testing → Risk in the app and MCP testneo_api_risk_* tools. Product overview: API automation — Risk Discovery · MCP recipe: Workflow 13.

Commands

Flags

All examples (copy-paste)

P=YOUR_PROJECT_ID
SAMPLE="$(npm root -g)/@testneo/cli/examples/api-risk/fakestore-products.curl"
# From monorepo after build: SAMPLE=examples/api-risk/fakestore-products.curl

# --- Discover only (no HTTP to target) ---
testneo api-risk discover --project $P --file "$SAMPLE"
testneo api-risk discover --project $P --file "$SAMPLE" --json
testneo api-risk discover --project $P --curl - --max-variants 20 < "$SAMPLE"
testneo api-risk discover --project $P --file "$SAMPLE" --injection

# --- Run (cloud) ---
testneo api-risk run --project $P --file "$SAMPLE" --cloud
testneo api-risk run --project $P --file "$SAMPLE" --cloud --max-variants 10
testneo api-risk run --project $P --file "$SAMPLE" --cloud --fail-on-critical
testneo api-risk run --project $P --file "$SAMPLE" --cloud --min-confidence 50
testneo api-risk run --project $P --file "$SAMPLE" --cloud --brief-out api-confidence.md
testneo api-risk run --project $P --file "$SAMPLE" --cloud --body-oracles
testneo api-risk run --project $P --file "$SAMPLE" --cloud --json > run.json

# --- Run (local agent) ---
testneo agent status
testneo api-risk run --project $P --file "$SAMPLE" --use-agent --max-variants 20 --timeout 90

# --- One-shot scan ---
testneo api-risk scan --project $P --file "$SAMPLE" --cloud \
  --fail-on-critical --brief-out api-confidence.md

testneo api-risk scan --project $P --file "$SAMPLE" --cloud \
  --fail-on-critical --brief-out api-confidence.md \
  --promote --confirm

testneo api-risk scan --project $P --file "$SAMPLE" --use-agent --max-variants 20 \
  --brief-out api-confidence.md

# --- Brief from saved run or run_id ---
testneo api-risk brief --project $P --run-file run.json --out api-confidence.md
testneo api-risk brief --project $P --run-id RUN_UUID --format markdown
testneo api-risk brief --project $P --run-file run.json --format json --out brief.json

# --- Promote (Remember) ---
testneo api-risk promote --project $P --run-file run.json --confirm
testneo api-risk promote --project $P --run-file run.json --confirm --no-remember
testneo api-risk promote --project $P --run-file run.json --confirm --remember-warnings

Typical next step after a successful scan

# You already have a brief. Save a run JSON, then promote:
node dist/cli.js api-risk run -p 6 -f examples/api-risk/fakestore-products.curl --cloud --json > run.json
node dist/cli.js api-risk promote -p 6 --run-file run.json --confirm

# Or one-shot re-scan + promote:
node dist/cli.js api-risk scan -p 6 -f examples/api-risk/fakestore-products.curl --cloud \
  --brief-out api-confidence.md --promote --confirm

PR validate — Release Brief

Builds a Release Brief from local git (or a connected GitHub PR): risk level (PASS / WARN / BLOCK), impacted tests, Engineering Memory matches, and a board URL. Product deep-dive: Release Intelligence.

# From your app repo (must have commits + origin/main or origin/master)
cd /path/to/your-app

testneo pr validate --project YOUR_PROJECT_ID
# prints brief + workflow_id + https://app.testneo.ai/web/pr-validation/<id>

# Connected GitHub PR number
testneo pr validate --project YOUR_PROJECT_ID --pr 142

# Run impacted tests (requires explicit --confirm)
testneo pr validate --project YOUR_PROJECT_ID --run-impacted
# → errors without --confirm (write guard)

testneo pr validate --project YOUR_PROJECT_ID \
  --run-impacted --confirm --wait --fail-on-error

# Force cloud for impacted runs (CI / no local agent)
testneo pr validate --project YOUR_PROJECT_ID \
  --run-impacted --confirm --cloud --wait

testneo pr history --project YOUR_PROJECT_ID --limit 5

What to look for: 🟢 PASS / 🟡 WARN / 🔴 BLOCK · risk score · historical incident matches · merge recommendation · open the board link for the full UI.

Engineering Memory

Prior incidents, patterns, and live matches against your current diff.

cd /path/to/your-app

testneo memory match --project YOUR_PROJECT_ID     # score / tier vs current changes
testneo memory list --project YOUR_PROJECT_ID      # stored entries
testneo memory patterns --project YOUR_PROJECT_ID  # clustered patterns

Use match before merge when you want a fast “have we broken this area before?” signal; use pr validate for the full brief.

Release readiness

Project-level confidence (0–100) and recommendation (e.g. SAFE TO RELEASE).

testneo readiness --project YOUR_PROJECT_ID
testneo readiness --project YOUR_PROJECT_ID --json   # scripts / CI

# Same question via Ask (narrative + links)
testneo ask --project YOUR_PROJECT_ID "What is my release readiness?"
testneo ask --project YOUR_PROJECT_ID "Show release metrics"
testneo ask --project YOUR_PROJECT_ID "Generate release decision report"

UI board: app.testneo.ai/web/release-readiness

Ask — AI Assistant in the terminal

Same endpoint as web /web/ai-assistant and MCP testneo_ai_assistant_query. Operational questions (inventory, readiness, navigate, k6/codegen) are answered from your project data via the server command router — not a hard-coded FAQ.

Modes

testneo ask --project YOUR_PROJECT_ID "How many test cases do I have?"
testneo ask --project YOUR_PROJECT_ID --detailed "Give me an executive summary"
testneo ask --project YOUR_PROJECT_ID --json "List all test cases"
testneo ask --project YOUR_PROJECT_ID --chat

In --chat, type the question only (e.g. Go to dashboard). Pasting a full testneo ask … line also works — the CLI strips the wrapper.

High-value questions

Interactive chat script

testneo ask --project YOUR_PROJECT_ID --chat

1. How many test cases do I have?
2. Which tests cover login?
3. Show latest test run results
4. What is my release readiness?
5. Generate release decision report
6. Open release readiness
7. exit

Demo reel — Ask loop (copy-paste)

Walk inventory → latest runs → readiness → ship decision → UI link. Great for demos. Set P to your project ID from testneo doctor.

P=YOUR_PROJECT_ID

for q in \
  "How many test cases do I have?" \
  "List all test cases" \
  "Which tests cover login?" \
  "Show latest test run results" \
  "What is my release readiness?" \
  "Show release metrics" \
  "Generate release decision report" \
  "Is it safe to ship?" \
  "Open release readiness"
do
  echo ""
  echo "════════ $q ════════"
  testneo ask --project "$P" "$q"
  sleep 2
done

Navigate answers print absolute links under https://app.testneo.ai/… (from your webAppUrl config).

Commands reference (v0.3.0)

Global flags

run / try / impacted PR flags

TUI & NDJSON agent mode

testneo tui

testneo try "Verify Products is visible" --wait --agent
# → NDJSON events, then a final result line

.testneo file format

# Test: Sauce Demo login smoke
# Project: YOUR_PROJECT_ID
# Environment: dev
# Tags: smoke

Navigate to {{base_url}}
Enter username as "standard_user"
Enter password as "secret_sauce"
Click on Login button
Verify "Products" is visible

Code:
  await page.locator('[data-test="add-to-cart-sauce-labs-backpack"]').click()
End Code:

Same NLP as the VS Code extension and Test Manager.

CI (GitHub Actions)

- name: TestNeo smoke
  run: |
    npm install -g @testneo/cli@0.3.0
    testneo run tests/smoke/ --cloud --wait --fail-on-error
  env:
    TESTNEO_BASE_URL: https://app.testneo.ai
    TESTNEO_WEB_APP_URL: https://app.testneo.ai
    TESTNEO_API_KEY: ${{ secrets.TESTNEO_API_KEY }}
    TESTNEO_PROJECT_ID: "47"
    TESTNEO_PREFER_AGENT: "false"


- name: API Risk Discovery (optional)
  run: |
    testneo api-risk scan --project ${{ env.TESTNEO_PROJECT_ID }} \
      --file node_modules/@testneo/cli/examples/api-risk/fakestore-products.curl \
      --cloud --fail-on-critical --brief-out api-confidence.md
  env:
    TESTNEO_BASE_URL: https://app.testneo.ai
    TESTNEO_API_KEY: ${{ secrets.TESTNEO_API_KEY }}
    TESTNEO_PROJECT_ID: "47"

- name: PR release brief (optional)
  run: |
    testneo pr validate --project ${{ env.TESTNEO_PROJECT_ID }} --json
  env:
    TESTNEO_BASE_URL: https://app.testneo.ai
    TESTNEO_API_KEY: ${{ secrets.TESTNEO_API_KEY }}
    TESTNEO_PROJECT_ID: "47"
  # checkout must include full git history for local impact

Environment variables

Troubleshooting