Cookie banners. Maintenance notifications. Live metrics. Chatbot rotation.
Native toHaveScreenshot() fails on all of them. BugHunters Vision understands context — it doesn't.
We ran the same 4 E-Commerce scenarios twice. Here's what happened.
We ran the NovaTech E-Commerce suite twice — once with native
toHaveScreenshot(), once with vision.check().
Here's every verdict, side by side.
4 E-Commerce scenarios. Native Playwright vs BugHunters Vision.
Note: BHV AI evaluation takes 20–40s per check on cold start. Fast Pixel Match is <1s. The extra time pays for zero false alarms and instant triage.
Both pages are hosted on bughunters.dev. URL params inject bug states — no server needed. Run the quickstart guide against either target.
node --versionbhv_xxx... — get yours free →# Create project directory $ mkdir bhv-demo && cd bhv-demo $ npm init -y # Install Playwright + BugHunters Vision $ npm install @playwright/test @bughunters/vision --save-dev # Download Chromium browser (takes ~1 min) $ npx playwright install chromium
.env file or CI secret.# Get your token at bughunters.dev — it's free to try
BUGHUNTERS_VISION_TOKEN=bhv_your_token_here
Add .env to your .gitignore. In CI, set it as a repository secret.
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './tests',
timeout: 90_000, // AI evaluation can take 20–40s on cold start
reporter: [
['list'],
['@bughunters/vision/reporter', {
snapshotsDir: './bhv-snapshots',
reportDir: './bhv-report',
}],
],
use: {
screenshot: 'off', // BHV handles screenshots manually
// ── Core behaviour ──────────────────────────────────────
bvMode: 'ai', // 'ai' | 'strict' | 'off'
bvFailMode: 'hard', // 'hard' throws | 'soft' collects all failures
bvUpdateBaseline: false, // true = force-save current as new baseline
// ── AI data tolerance — the big one ─────────────────────
// 'tolerant' (default): live data values — metrics, counters,
// timestamps — are treated as normal variation and ignored.
// 'strict': flag ALL changes, including numeric values.
bvDataTolerance: 'tolerant', // 'tolerant' | 'strict'
// ── Snapshot storage ─────────────────────────────────────
bvSnapshotsDir: './bhv-snapshots',
bvApiUrl: 'https://bugvision-backend.vercel.app', // override for on-premise
} as Record<string, unknown>,
projects: [
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
],
});
All options can also be set via environment variables:
BHV_MODE, BHV_FAIL_MODE, BHV_DATA_TOLERANCE, BHV_UPDATE_BASELINE=true.
Useful for overriding in CI without changing config files.
DEMO_PHASE=setup makes all tests use the clean URL (for baseline creation).
DEMO_PHASE=demo sends tests B, C & E to bugged states.
import { test } from '@playwright/test';
import { vision } from '@bughunters/vision';
const DASHBOARD = 'https://bughunters.dev/dashboard';
const phase = process.env.DEMO_PHASE ?? 'demo';
function url(bug: string) {
return `${DASHBOARD}?bug=${phase === 'setup' ? 'none' : bug}`;
}
test.describe('BugHunters Vision — Cloud Dashboard', () => {
// ── A: identical screenshots → 0 AI calls, $0 cost ─────────
test('Scenario A — identical pages (Fast Pixel Match)', async ({ page }) => {
await page.goto(`${DASHBOARD}?bug=none`);
await page.waitForLoadState('networkidle');
await vision.check(page, 'cloud-dashboard-A',
'Verify: Deploy button visible, metrics shown, charts rendered.');
});
// ── B: chart bar heights vary slightly → normal data tick ────
// Regular PW: no computed-style assertion can catch this cleanly.
// AI: "minor data variation, all good." → PASS
test('Scenario B — chart data refresh (AI Pass)', async ({ page }) => {
await page.goto(url('chart-tick'));
await page.waitForLoadState('networkidle');
await vision.check(page, 'cloud-dashboard-B',
'Slight bar height variation in the chart is normal metric ' +
'refresh — ignore it. Verify Deploy button and layout are intact.');
});
// ── C: Deploy button shows trash icon → selectors still PASS ─
// expect('#deploy-btn').toBeVisible() → PASSES
// expect('#deploy-btn').toContainText('Deploy') → PASSES
// AI sees the trash icon → critical regression → FAIL
test('Scenario C — wrong icon on Deploy button (AI Fail)', async ({ page }) => {
await page.goto(url('deploy-icon'));
await page.waitForLoadState('networkidle');
await vision.check(page, 'cloud-dashboard-C',
'Inspect the icon inside the "Deploy" button. It should be a ' +
'deployment arrow. A trash or delete icon is a critical regression.');
});
// ── D: 3 vision.check() calls in one test → all pixel-perfect ─
// Shows multi-check workflow: header, main content, full page.
// Report "smart expands" to show checks list without auto-opening.
test('Scenario D — multiple visual checks (smart expand demo)', async ({ page }) => {
await page.goto(`${DASHBOARD}?bug=none`);
await page.waitForLoadState('networkidle');
await vision.check(page.locator('header, nav').first(), 'dashboard-D-header',
'Check the navigation header area renders correctly.');
await vision.check(page.locator('main').first(), 'dashboard-D-metrics',
'Check the main content area renders correctly.');
await vision.check(page, 'dashboard-D-fullpage',
'Check the full dashboard page renders correctly.');
});
// ── E: interaction test — Deploy modal broken ─────────────────
// Check 1: clean page → PIXEL PASS
// Check 2: open modal → shows red circuit-breaker error → AI FAIL
// Check 3: close modal → back to clean page → PIXEL PASS
// Report auto-opens the first failing check ("smart expand").
test('Scenario E — modal regression (smart expand of failing check)', async ({ page }) => {
await page.goto(`${DASHBOARD}?bug=none`);
await page.waitForLoadState('networkidle');
await vision.check(page, 'dashboard-E-clean',
'Clean dashboard before interaction.');
await page.goto(url('modal-broken'));
await page.waitForLoadState('networkidle');
await page.click('#deploy-btn');
await page.waitForSelector('#deploy-modal.open');
await vision.check(page, 'dashboard-E-modal',
'The Deploy confirmation modal is open. Correct state: blue ' +
'deployment panel with version and Confirm button. A red error ' +
'or disabled button is a critical regression — flag as FAIL.');
await page.keyboard.press('Escape');
await page.waitForSelector('#deploy-modal:not(.open)');
await vision.check(page, 'dashboard-E-after-close',
'Modal closed — verify dashboard is back to normal state.');
});
});
?bug=none).
BHV saves the screenshots as the reference baseline for each test.
# Load the token from your .env file $ export $(cat .env | xargs) # Create clean baselines for all 5 tests (run once) $ DEMO_PHASE=setup BHV_UPDATE_BASELINE=true npx playwright test Running 5 tests using 1 worker 📸 Baseline created: cloud-dashboard-A ✓ Scenario A — identical pages (Fast Pixel Match) (1.3s) 📸 Baseline created: cloud-dashboard-B ✓ Scenario B — chart data refresh (AI Pass) (1.1s) 📸 Baseline created: cloud-dashboard-C ✓ Scenario C — wrong icon on Deploy button (AI Fail) (1.0s) 📸 Baseline created: dashboard-D-header, D-metrics, D-fullpage ✓ Scenario D — multiple visual checks (smart expand demo) (1.2s) 📸 Baseline created: dashboard-E-clean, E-modal, E-after-close ✓ Scenario E — modal regression (smart expand) (1.7s) 5 passed (8.8s)
Baselines are saved to ./bhv-snapshots/. Commit them to your repo so your team shares the same reference.
$ DEMO_PHASE=demo npx playwright test Running 5 tests using 1 worker ✅ FAST PIXEL MATCH: cloud-dashboard-A ✓ Scenario A — identical pages (Fast Pixel Match) (1.3s) 🤖 AI PASS: cloud-dashboard-B ✓ Scenario B — chart data refresh (AI Pass) (8.0s) ✗ Scenario C — wrong icon on Deploy button (AI Fail) (10.4s) Visual regression: Deploy button icon changed from deploy arrow to a cloud-upload icon — critical visual regression. ✅ FAST PIXEL MATCH: dashboard-D-header, D-metrics, D-fullpage ✓ Scenario D — multiple visual checks (smart expand demo) (1.3s) ✗ Scenario E — modal regression (smart expand) (5.8s) Visual regression: Modal shows red "Deployment blocked" circuit-breaker error instead of blue confirmation panel. 3 passed, 2 failed (34.9s)
$ open bhv-report/index.html # macOS $ start bhv-report/index.html # Windows $ xdg-open bhv-report/index.html # Linux
Every option can be set in playwright.config.ts and overridden via env vars — ideal for staging vs. production CI pipelines.
bvDataTolerance
'tolerant' (default) | 'strict'
Most visual testing tools break when deployed against shared staging environments. Why? Because counters tick, charts refresh, timestamps change — and every pixel diff triggers a failure.
BugHunters Vision understands live data. With bvDataTolerance: 'tolerant', the AI is instructed to treat changing metric values, percentages, request counts, and timestamps as expected real-world variation — not regressions. Only structural layout changes, missing elements, or broken UI get flagged.
Override in CI: BHV_DATA_TOLERANCE=strict npx playwright test
bvMode
'ai' | 'strict' | 'off'
Controls the evaluation pipeline for every visual check.
'ai'
Fast Pixel Match first → AI only when pixels differ. Best cost control.
'strict'
Fail immediately on any pixel diff. No AI call. Zero AI cost — for deterministic UIs.
'off'
Disable all visual checks — useful to skip BHV on a specific CI pipeline without removing code.
Env: BHV_MODE=strict
bvFailMode
'hard' | 'soft'
Controls what happens when BHV detects a visual regression.
'hard'
Throws immediately — test stops. Default for strict CI gates.
'soft'
Uses expect.soft() — all checks run, failures collected and reported together. Great for thorough visual audits.
Env: BHV_FAIL_MODE=soft
bvUpdateBaseline
Force-save the current screenshot as the new baseline for every check. Use when you intentionally changed the UI and want to promote the new state.
BHV_UPDATE_BASELINE=true npx playwright test
Tip: commit updated baselines to git so your whole team shares the same reference.
bvSnapshotsDir
Directory where baseline and current screenshots are stored. Defaults to ./bhv-snapshots.
Filenames include Playwright project + test title + step name to prevent collisions when running chromium/firefox/webkit in parallel.
Env: BHV_SNAPSHOTS_DIR=./custom-dir
By default, the HTML report shows screenshots only for failed or errored checks. Passed checks show a brief verdict note to keep the report compact.
Click the 📸 Pass screenshots button in the report header to toggle baseline (and current, for AI-evaluated) images visible for passed steps too.
No config needed — it's an interactive toggle in the generated report.
Run the NovaTech Shop scenarios yourself. Set up baselines, then watch native Playwright produce 3 false positives while BHV catches the only real bug.
Open E-Shop demo →Add BUGHUNTERS_VISION_TOKEN as a GitHub Actions secret and run npx playwright test in your workflow. BHV publishes a rich Markdown summary to GitHub Step Summary automatically.
The third argument to vision.check() is a plain-English hint to the AI. Tell it what to ignore, what to verify, and what matters.
Get a free API key and add AI visual testing to your Playwright suite today.