False Positive Elimination

Your CI is red.
Nothing is actually broken.

Cookie banners. Maintenance notifications. Live metrics. Chatbot rotation.
Native toHaveScreenshot() fails on all of them. BugHunters Vision understands context — it doesn't.

We ran the same 4 E-Commerce scenarios twice. Here's what happened.

75%
of native Playwright visual failures
in our E-Commerce suite were false positives
0
false positives with
BugHunters Vision AI
€0
AI cost when screenshots
are identical (Fast Pixel Match)
The Score

Same 4 scenarios. Completely different results.

We ran the NovaTech E-Commerce suite twice — once with native toHaveScreenshot(), once with vision.check(). Here's every verdict, side by side.

Scenario Native toHaveScreenshot() BugHunters Vision
FALSE +
Scenario F — GDPR cookie banner
Fresh CI session → consent dialog appears over page
✗ FAIL (false +) ✓ AI PASS
FALSE +
Scenario G — Maintenance banner
Ops adds notice → all content shifts 44px down
✗ FAIL (false +) ✓ AI PASS
TRUE +
Scenario H — Broken price €NaN
JS data binding failure → price renders as "€NaN"
✗ FAIL (no context) ✗ AI FAIL + reason
FALSE +
Scenario I — Chatbot agent rotation
Support widget shows different agent (name, avatar, status)
✗ FAIL (false +) ✓ AI PASS
Final Score 4 FAIL · 0 actionable 3 PASS · 1 FAIL with reason
Terminal output

Same test suite. Read the difference.

4 E-Commerce scenarios. Native Playwright vs BugHunters Vision.

Native toHaveScreenshot()
DEMO_PHASE=demo npx playwright test shop-native.spec.ts
✗ Scenario F — GDPR cookie banner
Screenshot comparison failed: 142,832 pixels differ
✗ Scenario G — maintenance notification banner
Screenshot comparison failed: 891,206 pixels differ
✗ Scenario H — broken price display
Screenshot comparison failed: 3,847 pixels differ
✗ Scenario I — chatbot agent rotation
Screenshot comparison failed: 28,419 pixels differ
4 failed (12.3s)
Which one is the real bug? Open each diff to find out.
BugHunters Vision
DEMO_PHASE=demo npx playwright test shop.spec.ts
✓ Scenario F — GDPR cookie banner (21.3s)
AI PASS: Standard GDPR consent dialog on fresh session — not a regression
✓ Scenario G — maintenance notification banner (19.8s)
AI PASS: Planned maintenance notice — core product page intact
✗ Scenario H — broken price display (18.5s)
AI FAIL: Price shows '€NaN' — JS data binding failure. Add to Cart disabled. Revenue-critical.
✓ Scenario I — chatbot agent rotation (20.1s)
AI PASS: Agent rotation is normal live-support behaviour
3 passed, 1 failed (79.7s)
Exactly 1 real bug. Identified immediately. No diff images needed.

Note: BHV AI evaluation takes 20–40s per check on cold start. Fast Pixel Match is <1s. The extra time pays for zero false alarms and instant triage.

The targets

Two live demo pages — nine scenarios in total.

Both pages are hosted on bughunters.dev. URL params inject bug states — no server needed. Run the quickstart guide against either target.

Cloud Infrastructure Dashboard
✅ Scenario A
Fast Pixel Match
Identical screenshots → 0 AI calls, $0 cost
?bug=none ↗
🤖 Scenario B
AI Pass — chart data refresh
Bar heights differ slightly → AI: normal variation, passes
?bug=chart-tick ↗
❌ Scenario C
AI Fail — wrong icon on Deploy
Trash icon replaces deploy arrow — button text still "Deploy"
?bug=deploy-icon ↗
✅ Scenario D
Multi-check — all pixel-perfect
3 visual checks in one test, all pass — smart expand shows list without auto-opening
?bug=none ↗
❌ Scenario E
AI Fail — broken modal interaction
Click Deploy → modal opens → circuit-breaker error instead of confirm dialog. Selectors pass, AI catches it.
?bug=modal-broken ↗
NovaTech E-Commerce Store  ★ False positive elimination demos
✅ Scenario F native FAIL
AI Pass — GDPR cookie banner
Fresh CI session shows consent dialog. AI identifies it as standard browser behaviour, evaluates the product page beneath it.
?bug=cookie-banner ↗
✅ Scenario G native FAIL
AI Pass — maintenance notification
Ops deploys a banner — every pixel shifts 44px. Native pixel-diff: 100% of the page "changed". AI reads the banner and passes.
?bug=maintenance ↗
❌ Scenario H true positive
AI Fail — broken price €NaN
JS data binding failure → price = "€NaN", Add to Cart disabled. BHV explains the bug immediately; native gives you a diff PNG.
?bug=price-broken ↗
✅ Scenario I native FAIL
AI Pass — chatbot agent rotation
Support widget shows a different agent — different name, avatar colour, status. Native fails. AI scopes to core product UI and passes.
?bug=chatbot ↗
Prerequisites
Node.js 18 or later
Check: node --version
BugHunters Vision API token
Format: bhv_xxx... — get yours free →
Step-by-step guide
1
Create a new project and install dependencies
A fresh directory, Playwright, and the BugHunters Vision package.
terminal
# Create project directory
$ mkdir bhv-demo && cd bhv-demo
$ npm init -y

# Install Playwright + BugHunters Vision
$ npm install @playwright/test @bughunters/vision --save-dev

# Download Chromium browser (takes ~1 min)
$ npx playwright install chromium
2
Set your API token
The token authorises AI calls to the BHV cloud API. Never commit it — use a .env file or CI secret.
.env
# Get your token at bughunters.dev — it's free to try
BUGHUNTERS_VISION_TOKEN=bhv_your_token_here

Add .env to your .gitignore. In CI, set it as a repository secret.

3
Configure Playwright
Register the BugHunters Vision reporter and tune visual testing behaviour for your team.
playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  timeout: 90_000, // AI evaluation can take 20–40s on cold start
  reporter: [
    ['list'],
    ['@bughunters/vision/reporter', {
      snapshotsDir: './bhv-snapshots',
      reportDir:    './bhv-report',
    }],
  ],
  use: {
    screenshot:       'off',        // BHV handles screenshots manually

    // ── Core behaviour ──────────────────────────────────────
    bvMode:           'ai',         // 'ai' | 'strict' | 'off'
    bvFailMode:       'hard',       // 'hard' throws | 'soft' collects all failures
    bvUpdateBaseline: false,        // true = force-save current as new baseline

    // ── AI data tolerance — the big one ─────────────────────
    // 'tolerant' (default): live data values — metrics, counters,
    // timestamps — are treated as normal variation and ignored.
    // 'strict': flag ALL changes, including numeric values.
    bvDataTolerance:  'tolerant',   // 'tolerant' | 'strict'

    // ── Snapshot storage ─────────────────────────────────────
    bvSnapshotsDir:   './bhv-snapshots',
    bvApiUrl:         'https://bugvision-backend.vercel.app', // override for on-premise
  } as Record<string, unknown>,
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
});

All options can also be set via environment variables: BHV_MODE, BHV_FAIL_MODE, BHV_DATA_TOLERANCE, BHV_UPDATE_BASELINE=true. Useful for overriding in CI without changing config files.

4
Write the five-scenario test file
DEMO_PHASE=setup makes all tests use the clean URL (for baseline creation). DEMO_PHASE=demo sends tests B, C & E to bugged states.
tests/dashboard.spec.ts
import { test } from '@playwright/test';
import { vision } from '@bughunters/vision';

const DASHBOARD = 'https://bughunters.dev/dashboard';
const phase     = process.env.DEMO_PHASE ?? 'demo';

function url(bug: string) {
  return `${DASHBOARD}?bug=${phase === 'setup' ? 'none' : bug}`;
}

test.describe('BugHunters Vision — Cloud Dashboard', () => {

  // ── A: identical screenshots → 0 AI calls, $0 cost ─────────
  test('Scenario A — identical pages (Fast Pixel Match)', async ({ page }) => {
    await page.goto(`${DASHBOARD}?bug=none`);
    await page.waitForLoadState('networkidle');
    await vision.check(page, 'cloud-dashboard-A',
      'Verify: Deploy button visible, metrics shown, charts rendered.');
  });

  // ── B: chart bar heights vary slightly → normal data tick ────
  // Regular PW: no computed-style assertion can catch this cleanly.
  // AI: "minor data variation, all good." → PASS
  test('Scenario B — chart data refresh (AI Pass)', async ({ page }) => {
    await page.goto(url('chart-tick'));
    await page.waitForLoadState('networkidle');
    await vision.check(page, 'cloud-dashboard-B',
      'Slight bar height variation in the chart is normal metric ' +
      'refresh — ignore it. Verify Deploy button and layout are intact.');
  });

  // ── C: Deploy button shows trash icon → selectors still PASS ─
  // expect('#deploy-btn').toBeVisible()           → PASSES
  // expect('#deploy-btn').toContainText('Deploy') → PASSES
  // AI sees the trash icon → critical regression → FAIL
  test('Scenario C — wrong icon on Deploy button (AI Fail)', async ({ page }) => {
    await page.goto(url('deploy-icon'));
    await page.waitForLoadState('networkidle');
    await vision.check(page, 'cloud-dashboard-C',
      'Inspect the icon inside the "Deploy" button. It should be a ' +
      'deployment arrow. A trash or delete icon is a critical regression.');
  });

  // ── D: 3 vision.check() calls in one test → all pixel-perfect ─
  // Shows multi-check workflow: header, main content, full page.
  // Report "smart expands" to show checks list without auto-opening.
  test('Scenario D — multiple visual checks (smart expand demo)', async ({ page }) => {
    await page.goto(`${DASHBOARD}?bug=none`);
    await page.waitForLoadState('networkidle');
    await vision.check(page.locator('header, nav').first(), 'dashboard-D-header',
      'Check the navigation header area renders correctly.');
    await vision.check(page.locator('main').first(), 'dashboard-D-metrics',
      'Check the main content area renders correctly.');
    await vision.check(page, 'dashboard-D-fullpage',
      'Check the full dashboard page renders correctly.');
  });

  // ── E: interaction test — Deploy modal broken ─────────────────
  // Check 1: clean page → PIXEL PASS
  // Check 2: open modal → shows red circuit-breaker error → AI FAIL
  // Check 3: close modal → back to clean page → PIXEL PASS
  // Report auto-opens the first failing check ("smart expand").
  test('Scenario E — modal regression (smart expand of failing check)', async ({ page }) => {
    await page.goto(`${DASHBOARD}?bug=none`);
    await page.waitForLoadState('networkidle');
    await vision.check(page, 'dashboard-E-clean',
      'Clean dashboard before interaction.');

    await page.goto(url('modal-broken'));
    await page.waitForLoadState('networkidle');
    await page.click('#deploy-btn');
    await page.waitForSelector('#deploy-modal.open');
    await vision.check(page, 'dashboard-E-modal',
      'The Deploy confirmation modal is open. Correct state: blue ' +
      'deployment panel with version and Confirm button. A red error ' +
      'or disabled button is a critical regression — flag as FAIL.');

    await page.keyboard.press('Escape');
    await page.waitForSelector('#deploy-modal:not(.open)');
    await vision.check(page, 'dashboard-E-after-close',
      'Modal closed — verify dashboard is back to normal state.');
  });

});
5
Create baselines (run once)
All three tests visit the clean dashboard (?bug=none). BHV saves the screenshots as the reference baseline for each test.
terminal
# Load the token from your .env file
$ export $(cat .env | xargs)

# Create clean baselines for all 5 tests (run once)
$ DEMO_PHASE=setup BHV_UPDATE_BASELINE=true npx playwright test

  Running 5 tests using 1 worker
  📸 Baseline created: cloud-dashboard-A
  ✓  Scenario A — identical pages (Fast Pixel Match)          (1.3s)
  📸 Baseline created: cloud-dashboard-B
  ✓  Scenario B — chart data refresh (AI Pass)                (1.1s)
  📸 Baseline created: cloud-dashboard-C
  ✓  Scenario C — wrong icon on Deploy button (AI Fail)       (1.0s)
  📸 Baseline created: dashboard-D-header, D-metrics, D-fullpage
  ✓  Scenario D — multiple visual checks (smart expand demo)  (1.2s)
  📸 Baseline created: dashboard-E-clean, E-modal, E-after-close
  ✓  Scenario E — modal regression (smart expand)             (1.7s)
  5 passed (8.8s)

Baselines are saved to ./bhv-snapshots/. Commit them to your repo so your team shares the same reference.

6
Run the demo 🚀
Tests B & C now visit the bugged dashboard states. Watch BHV make the right call on each scenario.
terminal
$ DEMO_PHASE=demo npx playwright test

  Running 5 tests using 1 worker
  ✅ FAST PIXEL MATCH: cloud-dashboard-A
  ✓  Scenario A — identical pages (Fast Pixel Match)          (1.3s)
  🤖 AI PASS: cloud-dashboard-B
  ✓  Scenario B — chart data refresh (AI Pass)                (8.0s)
  ✗  Scenario C — wrong icon on Deploy button (AI Fail)      (10.4s)
     Visual regression: Deploy button icon changed from deploy arrow
     to a cloud-upload icon — critical visual regression.
  ✅ FAST PIXEL MATCH: dashboard-D-header, D-metrics, D-fullpage
  ✓  Scenario D — multiple visual checks (smart expand demo)  (1.3s)
  ✗  Scenario E — modal regression (smart expand)             (5.8s)
     Visual regression: Modal shows red "Deployment blocked"
     circuit-breaker error instead of blue confirmation panel.
  3 passed, 2 failed (34.9s)
7
Open the HTML report
A local, standalone report is generated after every run — no upload, no cloud. Or skip ahead and see a live example ↗
$ open bhv-report/index.html      # macOS
$ start bhv-report/index.html    # Windows
$ xdg-open bhv-report/index.html # Linux
BugHunters Vision — Report
bhv-report/index.html
5 tests 3 passed 2 failed 0 baselines
⚡ Scenario A — identical pages PIXEL
✓ Scenario B — chart data refresh AI
✗ Scenario C — wrong icon on Deploy AI
⚡ Scenario D — 3 checks, all pixel-perfect PIXEL
✗ Scenario E — modal regression AI
Configuration reference

Tune it for every environment.

Every option can be set in playwright.config.ts and overridden via env vars — ideal for staging vs. production CI pipelines.

🧠
bvDataTolerance 'tolerant' (default) | 'strict'
BIG SELLING POINT

Most visual testing tools break when deployed against shared staging environments. Why? Because counters tick, charts refresh, timestamps change — and every pixel diff triggers a failure.

BugHunters Vision understands live data. With bvDataTolerance: 'tolerant', the AI is instructed to treat changing metric values, percentages, request counts, and timestamps as expected real-world variation — not regressions. Only structural layout changes, missing elements, or broken UI get flagged.

tolerant (default)
Numbers, %, timestamps, counters — all ignored as normal "live data" variation. Structural regressions still caught.
strict
Flag ALL changes including numeric values. Use when testing against controlled environments with deterministic data.

Override in CI: BHV_DATA_TOLERANCE=strict npx playwright test

⚙️ bvMode 'ai' | 'strict' | 'off'

Controls the evaluation pipeline for every visual check.

'ai' Fast Pixel Match first → AI only when pixels differ. Best cost control.
'strict' Fail immediately on any pixel diff. No AI call. Zero AI cost — for deterministic UIs.
'off' Disable all visual checks — useful to skip BHV on a specific CI pipeline without removing code.

Env: BHV_MODE=strict

🚦 bvFailMode 'hard' | 'soft'

Controls what happens when BHV detects a visual regression.

'hard' Throws immediately — test stops. Default for strict CI gates.
'soft' Uses expect.soft() — all checks run, failures collected and reported together. Great for thorough visual audits.

Env: BHV_FAIL_MODE=soft

📸 bvUpdateBaseline

Force-save the current screenshot as the new baseline for every check. Use when you intentionally changed the UI and want to promote the new state.

BHV_UPDATE_BASELINE=true npx playwright test

Tip: commit updated baselines to git so your whole team shares the same reference.

📁 bvSnapshotsDir

Directory where baseline and current screenshots are stored. Defaults to ./bhv-snapshots.

Filenames include Playwright project + test title + step name to prevent collisions when running chromium/firefox/webkit in parallel.

Env: BHV_SNAPSHOTS_DIR=./custom-dir

🖼 Pass screenshots in report

By default, the HTML report shows screenshots only for failed or errored checks. Passed checks show a brief verdict note to keep the report compact.

Click the 📸 Pass screenshots button in the report header to toggle baseline (and current, for AI-evaluated) images visible for passed steps too.

No config needed — it's an interactive toggle in the generated report.

What's next
🛒
Try the E-Commerce false positive demo

Run the NovaTech Shop scenarios yourself. Set up baselines, then watch native Playwright produce 3 false positives while BHV catches the only real bug.

Open E-Shop demo →
🔗
Add to CI/CD

Add BUGHUNTERS_VISION_TOKEN as a GitHub Actions secret and run npx playwright test in your workflow. BHV publishes a rich Markdown summary to GitHub Step Summary automatically.

See CI/CD integration →
🧠
Customise with prompts

The third argument to vision.check() is a plain-English hint to the AI. Tell it what to ignore, what to verify, and what matters.

See prompt example →
Get started

Ready to stop fighting false positives?

Get a free API key and add AI visual testing to your Playwright suite today.

Request Access → Back to home