Building an AI QA Agent That Tests Your Web App

A small team with one well-built AI QA agent runs the kind of regression coverage a dedicated QA hire used to deliver. The agent walks the app on every deploy, validates the flows that pay the bills, files a clean bug report when something drifts, and goes quiet on green builds. No manual checklist on release day, no engineer babysitting a test run, and no untested code path waiting to surprise a customer.

For most SMBs in 2026, the practical build is Playwright as the browser driver and Claude as the reasoning layer that decides what to test, interprets the page, and decides whether the result is right. Selenium still has a place in the conversation, but only for the narrow cases where a browser other than Chromium is the deciding factor. The cost of running this setup is far below a full-time QA hire, and the work to stand it up is measured in weeks rather than quarters.

This post covers the patterns that hold up in production, the patterns that look promising and then fail in week three, and how to think about the cost and ownership of an AI QA agent inside an SMB engineering team.

Why Playwright Beats Selenium for the AI Layer

The driver choice frames the rest of the build, so it is worth getting right early.

Playwright is the better default for an AI-driven QA agent. It is built around the same Chromium DevTools Protocol that modern AI tooling reads from natively, it ships with first-class auto-waiting (the agent rarely needs to insert manual sleeps), and its getByRole, getByLabel, and getByText locators line up cleanly with the way an AI agent reads a page. Claude is more accurate when the locators it reasons about resemble the labels a human would describe.

Selenium remains useful in two narrow cases: legacy test suites that already run on it and would be expensive to port, and cross-browser coverage that requires Firefox or Safari engines with full feature parity. For greenfield builds in 2026, the Playwright + Claude pairing is the lower-friction starting point.

A useful test: if the answer to "what browser do most of our users actually run" is Chrome or Edge, Playwright is the right choice. If a meaningful share of revenue runs through Safari or Firefox flows that have to be validated end to end, the conversation gets more nuanced and a cross-driver layer is worth the extra weight.

The Three Patterns That Work

How ScaleIT handles this: See AI Quick-Start

Most production AI QA agents fall into one of three shapes. Each one is a different slice of the same loop, suited to a different shape of work.

1. The Scripted Walk With AI Adjudication

The agent runs a deterministic Playwright script (log in, navigate to checkout, fill the form, submit) and Claude inspects the page at each checkpoint to decide whether the result matches the expected behavior described in natural language. The script handles the mechanical work, the AI handles the judgment call.

What makes this pattern hold up:

• The Playwright script is deterministic, so the agent reproduces the same path on every run.
• Claude only judges the rendered page against a written acceptance criterion ("the order confirmation page shows the correct total and order number"), so the prompt stays short and focused.
• A regression in the visual or copy layer (a misaligned price, a swapped button label) gets caught even though the click path still works.

This is the pattern most SMB teams should start with. It pairs the reliability of traditional E2E tests with the AI-native ability to notice things a brittle assertion misses.

2. The Exploratory Agent

Claude drives the browser directly through a Playwright tool, with the system prompt naming a goal ("sign up for an account, complete onboarding, send a test message") and a small tool set. The agent reads the page, decides what to click, and continues until it reaches the goal or gets stuck.

This pattern is right for:

• Smoke testing a new feature where the exact UI path is still in flux.
• Crawling the app weekly to find pages that 500 or throw console errors.
• Validating that the marketing site's primary call-to-action paths still funnel to the right next step.

It is overkill for stable flows that already have scripted coverage. Exploratory runs cost more in tokens, take longer, and produce more flake.

3. The Visual Regression Pair

Playwright takes screenshots of the canonical pages on every deploy, Claude compares them against the previous run, and flags differences with a short written description ("the pricing card on /plans is 8 pixels narrower and the heading wraps to two lines"). Traditional pixel-diff tools flag every Tailwind upgrade as a regression. Claude reads the change and decides whether it is meaningful.

The pattern earns its cost on marketing pages and core product surfaces where a layout shift can quietly damage conversion. It is overkill for admin tools that only the internal team sees.

The Anti-Patterns That Look Promising and Fail

A handful of approaches feel right in a demo and then break in production. They are worth naming so the team can skip them.

• Letting Claude write the locators from scratch on every run. The agent invents inconsistent selectors, the suite gets flaky, and the report is noise. Pin the locators in the Playwright script and let Claude judge results, not pick elements.
• Pointing the agent at the whole app with no goal. Open-ended exploration produces volume, not signal. Every agent run should have a named outcome.
• Skipping the deterministic baseline. Teams that go straight to a fully exploratory agent end up with no regression coverage on the flows that matter most. Scripted walks first, exploratory second.
• Treating the agent's output as ground truth. Claude's pass/fail decision is a strong signal, not a verdict. Every failed run still needs a human glance before a release is held.

Cost, Honestly

The dollars spread across three buckets: driver infrastructure, model calls, and human review time.

Driver infrastructure is the smallest line item. A Playwright runner fits on a single small CI worker for most SMB workloads, and the major CI vendors (GitHub Actions, CircleCI, Buildkite) all include enough free or low-tier minutes that a daily suite rarely pushes the bill into a new pricing band.

Model calls are the variable cost. A scripted-walk suite that judges roughly a dozen checkpoints per run on Claude Sonnet stays well below a coffee per day for an SMB-sized suite. An exploratory agent that walks the app for several minutes per run is meaningfully more expensive (still well short of a QA hire, but no longer rounding error). Caps on tool-call count per run and on total monthly spend keep the bill predictable.

Human review time is the cost most teams underestimate. Every failed run needs a fast glance. A noisy suite that cries wolf twice a week burns more engineering attention than a quiet suite that catches one real regression a month. The signal-to-noise ratio is the metric to watch, not the absolute pass rate.

Recommended Setup by Team Profile

The 5 to 15-person team with no QA hire. Start with two or three scripted-walk suites covering the flows that earn revenue (signup, checkout, primary product action). Pair each with a Claude adjudicator. Run on every deploy. This delivers the largest share of the value with the lowest token bill and the cleanest signal.

The 15 to 40-person team with one full-stack QA or QA-leaning engineer. Keep the scripted walks as the baseline and add a weekly exploratory agent that targets the parts of the app that are changing fastest. Use the QA engineer to triage exploratory findings and feed the useful ones back into the scripted suite.

The 40 to 150-person team with a dedicated QA function. Run scripted walks on every deploy, exploratory agents on a daily schedule, and visual regression pairs on the marketing surface and core product surfaces. The QA function owns the prompts, the tool set, and the budget caps. Engineering owns the deploy hooks and the failure-handling path.

What ScaleIt Sees in Practice

The teams that get the most out of an AI QA agent treat the build the same way they would treat any other production system. They pick one pattern, ship it well, run it for a month, and only then add a second pattern. The teams that struggle treat the agent as a side project, give it three goals at once, and never trust the output.

Most ScaleIt engagements that touch QA automation start with the consolidation step: take whatever ad-hoc agent or partial test suite already exists, narrow it to one named pattern, pin the locators, and put the signal-to-noise ratio on a dashboard. That single pass is usually enough to make the suite trustworthy. From there, adding a second or third pattern is incremental work.

Capabilities of a Mature AI QA Setup

When the patterns above are in place and the team has settled into the operational rhythm, the day-to-day looks like this:

• Every deploy runs the same scripted walks and the team trusts the green build to mean the revenue paths are healthy.
• Visual regressions land as one-line notes in the deploy channel, with a screenshot diff attached, and the reviewer can decide in seconds whether to block the release.
• Exploratory runs surface a small handful of real findings each week, not a queue of false alarms.
• The model bill is predictable, capped, and small relative to a QA headcount.
• Engineering time goes to building features, not to triaging flaky tests.
• The QA function (when there is one) spends its time on the parts of testing AI still does badly: complex multi-user scenarios, performance under load, and security cases.

That is what a quiet, useful AI QA layer looks like for an SMB engineering team.

Next Step

Are you looking to put an AI QA agent in front of your web app without spending a quarter wiring it up? We can help. Contact ScaleIt for a free consultation.

Q: Does an AI QA agent replace a human QA hire?

For most SMBs in the 5 to 40-person range, it replaces the part of QA work that is regression coverage and routine smoke testing. It does not replace exploratory testing on complex flows, performance work, or security testing. Teams with a QA function are more productive with the agent in the loop, not less.

Q: Why Playwright and not Cypress?

Cypress is a strong tool, especially for teams that already use it. The reason to default to Playwright for a new build is the broader cross-browser support, the auto-waiting that the AI layer benefits from, and the locator API that maps cleanly to how Claude reads a page. Cypress teams that already have a working suite should keep it.

Q: Should the agent run in CI or on a separate schedule?

Both. Run the scripted walks in CI on every deploy so a failing build blocks the release. Run the exploratory and visual regression agents on a separate daily or weekly schedule so a long agent run does not slow down the engineering feedback loop.

Q: What model is right for the adjudicator step?

Claude Sonnet is the practical default. It judges page state and visual diffs accurately, the per-call cost is low enough for daily use, and it handles the multi-modal cases (screenshot plus page text) cleanly. Opus is overkill for the scripted-walk pattern and worth considering only for the most ambiguous exploratory runs. Haiku is the right choice when the suite is large and the checkpoint judgments are simple.

Q: How do we handle authentication and test data?

Pin a dedicated test account with a known data fixture for every scripted walk. Reset the fixture before each run so the suite is reproducible. For exploratory runs, give the agent a sandboxed environment with synthetic data so a runaway agent cannot affect production records.

Verified against the Playwright 2026 documentation, the Selenium 4 documentation, the Anthropic Claude API documentation, and the Anthropic agent-building patterns reference. Last reviewed 2026-05-10.