TestNeo Playwright AI SDK

Build Playwright tests with natural language while keeping full control of native Playwright code.

Why this SDK

• Playwright-native execution model (not black-box remote-only runs)
• Hybrid testing: mix ai.run(...) and normal page.* / expect(...)
• Deterministic execution modes for CI and production
• Real-time run visibility with SSE and run artifacts
• Guest mode support for quick trials in playground

Prerequisites

• Node.js 18+ and npm.
• @playwright/test and @testneo/playwright-ai-sdk in the same Playwright project.

Install

npm install -D @playwright/test @testneo/playwright-ai-sdk

Install browsers (required for local runs):

npx playwright install chromium

Configure

Set environment variables (the SDK reads TESTNEO_BASE_URL, not TESTNEO_API_URL):

export TESTNEO_BASE_URL="https://app.testneo.ai"
export TESTNEO_API_KEY="tn_your_api_key"
export TESTNEO_PROJECT_ID="123"

Guest trial (no API key):

export TESTNEO_GUEST_MODE=true

Quick Start

import { test, expect } from "@testneo/playwright-ai-sdk";

test("hybrid web flow", async ({ page, ai }) => {
  await ai.run(`
    Navigate to https://www.saucedemo.com
    Enter username as "standard_user"
    Enter password as "secret_sauce"
    Click on Login button
    Verify "Products" is visible
  `, { mode: "balanced" });

  await page.click('[data-test="add-to-cart-sauce-labs-backpack"]');
  await expect(page.locator(".shopping_cart_badge")).toHaveText("1");
});

Core APIs

ai.run(commands, options?)

Runs one or more NLP commands in sequence.

await ai.run([
  "Navigate to https://example.com",
  'Verify "Example Domain" is visible',
], { mode: "strict" });

ai.step(command, options?)

Runs a single NLP step (useful for fine-grained control and debugging).

await ai.step('Click on "Sign in" button');

ai.verify(command, options?)

Assertion-focused NLP helper.

await ai.verify('Verify "Checkout" is visible');

ai.api(commands, options)

Runs API orchestration commands through the TestNeo API automation pipeline.

const apiRun = await ai.api(
  [
    "Call API GET https://httpbin.org/status/200",
    "Verify API response status equals 200",
  ],
  {
    projectId: Number(process.env.TESTNEO_PROJECT_ID),
    mode: "balanced",
    waitForCompletion: true,
  },
);

Execution Modes

• strict — max determinism, CI-friendly, minimal recovery
• balanced — practical fallback behavior
• adaptive — best effort and maximum recovery

Publishing and Links

Hybrid/local runs can be published to TestNeo so users can open:

• dashboard run details
• live execution view
• video / screenshot artifacts (when available)

Guest Mode

Guest mode is intended for no-login playground experiences.

• strict free-run limit (currently 10)
• server-side quota enforcement
• security checks on command payloads
• anti-abuse controls (rate limits and token checks)

Recommended CI Setup

• use strict mode in CI
• pin base URLs and test data
• keep deterministic test commands
• fail fast on parsing/execution errors

Versioning and Compatibility

• API contract is versioned server-side
• SDK is backward-compatible across minor updates where possible
• Keep your SDK version reasonably aligned with the latest TestNeo release

Links

• Playground: https://app.testneo.ai/playground
• Product signup: https://app.testneo.ai/signup
• SDK package on npm: @testneo/playwright-ai-sdk

Overview & Quickstart

The TestNeo Playwright AI SDK (@testneo/playwright-ai-sdk) lets teams write natural-language test steps alongside normal Playwright code: planning from TestNeo where needed, execution in your runner for transparency and debugging.

Who this is for

• QA engineers who want faster authoring with NLP
• SDETs who want CI-friendly strict / balanced modes
• Teams outgrowing recorder-only or opaque remote-only runners

What you get

• ai.run / ai.step / ai.verify for web NLP; native page / expect unchanged
• Data-driven web and API: ai.each, ai.eachApi (connected mode)
• ai.api for API NLP orchestration on TestNeo (connected mode)
• Optional dashboard sync: autoPublish or ai.publish with TESTNEO_API_KEY + project
• Guest trial without API key (10 runs per fingerprint, server-enforced); see Guest doc

Prerequisites

• Node.js 18+ and npm
• Network access to https://app.testneo.ai (the value of TESTNEO_BASE_URL)

Install

Peer dependency @playwright/test must be installed with the SDK:

npm install -D @playwright/test @testneo/playwright-ai-sdk
npx playwright install chromium

Use npx playwright install if your playwright.config uses multiple browsers.

Trial without API key or project (SDK guest)

export TESTNEO_BASE_URL="https://app.testneo.ai"
export TESTNEO_GUEST_MODE=true
unset TESTNEO_API_KEY TESTNEO_PROJECT_ID
npx playwright test

Quota file: .testneo-guest.json in the directory you run tests from (unless overridden). Exhausted quota → signup at https://app.testneo.ai/signup.

Connected mode configuration

Dashboard history, ai.api, autoPublish, ai.publish:

export TESTNEO_BASE_URL="https://app.testneo.ai"
export TESTNEO_API_KEY="tn_your_api_key"
export TESTNEO_PROJECT_ID="123"
unset TESTNEO_GUEST_MODE

Minimal test

import { test, expect } from "@testneo/playwright-ai-sdk";

test("login smoke", async ({ page, ai }) => {
  await ai.run(
    `
    Navigate to https://www.saucedemo.com
    Enter username as "standard_user"
    Enter password as "secret_sauce"
    Click on Login button
    Verify "Products" is visible
  `,
    { mode: "balanced" },
  );

  await expect(page).toHaveURL(/inventory/);
});

Hybrid + optional dashboard publish

test("hybrid + publish", async ({ page, ai }) => {
  await ai.run(`... NLP steps ...`, {
    mode: "balanced",
    autoPublish: {
      projectId: Number(process.env.TESTNEO_PROJECT_ID),
      testName: "smoke checkout",
      metadata: { suite: "e2e" },
    },
  });

  expect(await page.locator(".inventory_item").count()).toBeGreaterThan(0);
});

Open printed dashboard links while logged in as the API-key user. Paths use /test-runner/execution/....

Public Playground (no Playwright install)

Marketing / try-before-install: https://app.testneo.ai/playground. Separate from SDK guest mode; still subject to guest policies.

Execution modes

Recommended defaults

Next steps

• API surface: API Reference
• Guest + Playground: Guest Mode & Playground
• FAQ & setup: Troubleshooting & FAQ
• NPM README: Introduction

API Reference

Reference for @testneo/playwright-ai-sdk. For install and env setup, see Overview & Quickstart and Troubleshooting.

Prerequisites

• @playwright/test and @testneo/playwright-ai-sdk in the same project (peer dependency).
• npx playwright install chromium (or npx playwright install).
• TESTNEO_BASE_URL pointing at a reachable TestNeo API.

Imports

import { test, expect } from "@testneo/playwright-ai-sdk";
import {
  loadTestNeoConfigFromEnv,
  TestNeoSdkClient,
  normalizeTestNeoDashboardPath,
  TESTNEO_GUEST_SIGNUP_URL,
  getOrCreateStableGuestFingerprint,
} from "@testneo/playwright-ai-sdk";

Most tests only need test and expect. Advanced use can construct TestNeoSdkClient or read config via loadTestNeoConfigFromEnv.

test fixtures

ai fixture surface

• ai.run(commands, options?)
• ai.step(command, options?)
• ai.verify(assertion, options?)
• ai.each(rows, commands, options?) — data-driven web NLP ({{column}} templates)
• ai.eachApi(rows, commands, options) — data-driven API NLP
• ai.api(commands, options) — remote API test execution on TestNeo (requires project + key)
• ai.publish({ projectId, testName, nlpCommands?, metadata? }) — push last local run to dashboard (connected only)
• ai.lastRunReport() — last aggregated LocalExecutionReport (or null)
• ai.guestInfo() — { remainingRuns, maxRuns } in guest mode; else null

Default execution mode

If you omit mode, ai.run / ai.step / ai.verify / ai.each default to adaptive. Pass mode: "balanced" or mode: "strict" explicitly when you want tighter CI behavior.

ai.run(commands, options?)

Runs one or many NLP lines (string with newlines, or string[]).

Options (common)

autoPublish (connected mode)

autoPublish: {
  projectId: number;
  testName: string;
  metadata?: Record<string, unknown>;
  /** If true, logs every sync; default is one summary after the test ends. */
  logEachPublish?: boolean;
}

Each ai.run / ai.step / ai.verify with the same sticky autoPublish can create multiple dashboard executions; the SDK prints one end-of-test summary pointing at the latest id (full combined NLP history unless you use logEachPublish: true). Dashboard paths resolve to /test-runner/execution/{id} (and legacy /execution/... is normalized when building links).

Example

await ai.run(
  [
    "Navigate to https://example.com",
    'Verify "Example Domain" is visible',
  ],
  { mode: "strict", onStepEnd: (e) => console.log(e.command, e.status) },
);

ai.step(command, options?)

Single NLP command. Same options as ai.run except commands is one string.

await ai.step('Click on "Checkout" button', { mode: "balanced" });

ai.verify(assertion, options?)

Wraps assertions; if the string does not start with Verify / Assert, the SDK prefixes Verify.

await ai.verify('"Order confirmed" is visible');

ai.each(rows, commands, options?)

• rows: array of objects (e.g. from CSV or fixtures).
• commands: string / string[] / function (row, index) => string[].
• Placeholders: {{columnName}} substituted from each row keys.

Options: same as ai.run plus continueOnRowError.

await ai.each(
  [
    { username: "standard_user", password: "secret_sauce" },
    { username: "locked_out_user", password: "secret_sauce" },
  ],
  [
    'Navigate to https://www.saucedemo.com',
    'Enter username as "{{username}}"',
    'Enter password as "{{password}}"',
    'Click on Login button',
  ],
  { mode: "balanced", continueOnRowError: true },
);

ai.api(commands, options) Connected only

Runs NLP API automation on TestNeo infrastructure. Requires projectId and a valid TESTNEO_API_KEY.

Options

projectId, testName, mode, environmentId, environmentName, timeoutMs, waitForCompletion (default true), pollIntervalMs, includeSteps, metadata, idempotencyKey, planHash, replayToken.

Example

const result = await ai.api(
  [
    "Call API GET https://httpbin.org/bearer with header Authorization: Bearer my-token",
    "Verify API response status equals 200",
  ],
  {
    projectId: Number(process.env.TESTNEO_PROJECT_ID),
    mode: "balanced",
    waitForCompletion: true,
  },
);

console.log(result.executionId, result.status, result.dashboardUrl);

ai.eachApi(rows, commands, options) Connected only

Same templating as ai.each, but each row calls ai.api. Returns AIApiRunResult[].

ai.publish(...) Connected only

Publishes the last local hybrid report to the dashboard (after ai.run / ai.step / ai.verify). Optional nlpCommands overrides the command list sent to the server.

Result and dashboard links

• ai.api / published runs: responses include dashboard and live/manage URLs; log in as the API key user to open them.
• Hybrid local runs: no product video by default; live popup is most relevant for remote TestNeo browser runs.
• Programmatic helpers on TestNeoSdkClient: getExecutionLinks, normalizeTestNeoDashboardPath.

Environment variables

Signup when guest quota is exhausted: https://app.testneo.ai/signup (also available as TESTNEO_GUEST_SIGNUP_URL export).

Best practices

• Use strict for CI gates; balanced for daily automation; adaptive for local exploration.
• Keep NLP lines short and explicit.
• Use ai.step when you need precise stop points for debugging.
• For API chains, prefer inline Call API ... with header ... so headers apply reliably.
• Mix native expect / page for critical assertions.

Guest Mode & Public Playground

This page covers two guest experiences: the no-login public Playground in the web app, and SDK guest mode when running Playwright on your machine without an API key.

Comparison

Install prerequisites for the SDK (Node, @playwright/test, package, playwright install): see Overview & Quickstart and Troubleshooting.

What guest mode is (SDK + product)

• Limited trial before signup
• No authenticated project for SDK guest runs (nothing written to your project history)
• Strict server-side quotas and payload checks
• Upgrade path: https://app.testneo.ai/signup

Current policy (high level)

• ~10 free NLP runs per guest fingerprint (subject to server configuration)
• Session expiry enforced server-side
• Anti-abuse: session-creation throttles, fingerprint / IP limits, daily caps

Playground (public website) intent

• Run Live as the primary action: real execution, timeline, step progress
• Clear queue / capacity messaging when many concurrent guests
• Run outputs (video, summaries) via guest-safe URLs where applicable
• Explicit upgrade / signup when quota is exhausted

What guest users do not get

• Full authenticated product navigation without an account
• Unlimited free runs via repeated "resets" (fingerprint + IP controls)
• Raw internal debug endpoints as the default UX

SDK guest fingerprint (local machine)

With TESTNEO_GUEST_MODE=true and no API key:

1. The SDK stores a stable id in .testneo-guest.json under the current working directory (unless you set TESTNEO_GUEST_STATE_FILE or TESTNEO_GUEST_FINGERPRINT).
2. Each npx playwright test counts against the same server-side quota (not reset to 10 every run).
3. When quota is exhausted, APIs return 402 / 429 and the SDK error text includes https://app.testneo.ai/signup.

Playground NLP: data loops

The public playground supports Data: / End Data and For each row: / End For with placeholders like {firstName}. Put the full per-row journey inside the loop when the app returns to an end-state after each iteration (for example Sauce Demo order confirmation).

NPM SDK limitations in guest mode

• ai.api, ai.eachApi, ai.publish, and autoPublish require connected mode (TESTNEO_API_KEY + TESTNEO_PROJECT_ID, TESTNEO_GUEST_MODE unset).

Example user journey (Playground)

1. Open /playground
2. Run a live scenario and watch progress / timeline
3. Review outputs and links where available
4. Hit guest limits → CTA to signup
5. Use a full account for projects, dashboards, API keys, unlimited runs within plan

Troubleshooting & FAQ

Common setup issues, fixes, and answers to the questions teams ask most.

First-time setup checklist

1. Node.js (LTS) and npm installed.
2. npm i -D @playwright/test @testneo/playwright-ai-sdk — both are required.
3. npx playwright install chromium — required for browser launch (see errors below).
4. TESTNEO_BASE_URL points at a running TestNeo API (not legacy TESTNEO_API_URL for this SDK).
5. Either guest (TESTNEO_GUEST_MODE=true) or connected (TESTNEO_API_KEY + TESTNEO_PROJECT_ID for publish / API features).

Common issues

Executable doesn't exist / Playwright browser missing

Cause: Browser binaries not installed.

Fix:

npx playwright install chromium

Authentication required opening a dashboard URL

Cause: Product routes require login; SDK guest runs are not tied to your project.

Fix: Sign in as the API-key user for autoPublish / ai.api links, or signup at https://app.testneo.ai/signup.

Wrong route: /execution/... shows login loop

Cause: SPA route for execution detail is /test-runner/execution/:id; older links used /execution/:id.

Fix: Use the URL from current SDK/console output or normalize with normalizeTestNeoDashboardPath (@testneo/playwright-ai-sdk).

Project not found or API key errors

Cause: Invalid or missing TESTNEO_PROJECT_ID / TESTNEO_API_KEY, or key user does not own the project.

Fix: Set env vars correctly; verify project id in TestNeo UI.

Guest quota never drops between separate playwright test runs

Cause: SDK builds before ~0.1.4 used a random fingerprint every process.

Fix: Upgrade SDK; ensure .testneo-guest.json is written under your cwd, or set TESTNEO_GUEST_STATE_FILE / TESTNEO_GUEST_FINGERPRINT. After exhausted quota → https://app.testneo.ai/signup.

No tests found / wrong Playwright project

Cause: Running from wrong directory or wrong --project name.

Fix: Run from the package root that contains playwright.config.ts; align --project=api with your config.

Requiring @playwright/test second time

Cause: Two physical copies of @playwright/test (e.g. SDK linked as file:../packages/... with its own node_modules).

Fix: Depend on @testneo/playwright-ai-sdk via npm pack tarball or hoisted npm layout so Playwright resolves once.

fetch failed / ECONNREFUSED calling the API

Cause: API down, wrong TESTNEO_BASE_URL, firewall, or TLS issues.

Fix: Run a curl or browser health check against TESTNEO_BASE_URL (https://app.testneo.ai) and confirm your network/firewall allows outbound HTTPS.

Unsupported action type / NLP parsing errors

Cause: Command not mapped or ambiguous.

Fix: Rephrase; for API chains use Call API GET ... with header Authorization: Bearer ... style lines.

FAQ

Is execution local or remote?

Hybrid (default UX): NLP is planned via the API; Playwright drives the browser on your machine. ai.api runs API automation through TestNeo runners (remote orchestration).

Default mode if I omit it?

adaptive for ai.run / ai.step / ai.verify / ai.each. Override with strict / balanced for CI.

Can I mix NLP and plain Playwright?

Yes — that is core to the SDK.

Does guest mode allow unlimited runs?

No. Roughly 10 runs per fingerprint plus server-side anti-abuse; then signup at https://app.testneo.ai/signup.

Debug checklist

• TESTNEO_BASE_URL, TESTNEO_API_KEY, TESTNEO_PROJECT_ID, TESTNEO_GUEST_MODE correct for the scenario
• npx playwright install done
• Guest file .testneo-guest.json present if testing quota continuity
• Run from correct directory and Playwright project name
• For playground issues: /playground flow and SSE tab / network