// AI-Powered QA
Using AI in testing, with judgment attached
Where Claude Code, Cursor, and MCP-assisted workflows speed up my QA work, how I check what they produce, and what I hold them to.
Where AI does the work
- First-draft test generation and automation scaffolding with Claude Code and Cursor, at LiveSwitch and Triomics.
- Failure investigation: working through traces and logs faster to separate product defects from automation issues.
- Coverage-gap analysis: comparing what the suite exercises against the features it's meant to protect, to surface missing scenarios.
- An MCP server in the loop for faster first drafts and faster debugging of failing tests.
- In public: a Playwright MCP setup with planner, generator, and healer agents in my multi-layer framework, with human review before anything is committed.
How I validate what it produces
- Every AI-generated scenario is checked against the business requirement it claims to cover, not just whether it passes.
- Generated code has to land in the framework's existing architecture (page objects, fixtures, API clients) instead of sprawling into one-off scripts.
- Nothing joins a suite without a manual review. Deciding what actually needs coverage stays a human call.
Why the assertions stay deterministic
An AI-drafted test will happily assert whatever the application currently does — which means it passes on a bug. So the steps can be drafted by AI, but the expected values come from the requirement or from known test data, and the review checks that every assertion would fail if the behavior were wrong.
| Common in AI drafts | What ships instead |
|---|---|
| Asserts on whatever the page currently shows | Asserts on expected values from the requirement or seeded test data |
| Fixed waits and sleeps | Web-first assertions that wait for a specific state |
| Brittle CSS or XPath locators | Role- and label-based locators that survive markup changes |
| One long test covering many behaviors | Focused tests, each failing for one clear reason |
How I measure the benefit
The direct number is time: AI-assisted workflows cut the time I spend writing and debugging tests by 70%+ at Triomics and LiveSwitch — measured on authoring and debugging, not on total QA time.
The downstream number is regression execution time: ~40% lower at LiveSwitch after the Selenium-to-Playwright migration plus the AI-assisted workflow. That figure covers both changes together, so I don't claim it for AI alone.
The cost side is review effort. AI makes drafting cheap; it doesn't make reviewing free, and a workflow that saves authoring time but ships weaker assertions is a net loss.
Testing AI itself
Practice project — not yet public
Testing LLM features needs different tools. I built a DeepEval suite against a synthetic support assistant and a small RAG pipeline: 33 evaluation cases across four levels, from answer relevancy, correctness, toxicity, and bias on the chatbot, to faithfulness, hallucination, and contextual precision/recall on the RAG pipeline, custom G-Eval rubrics, and a golden-dataset regression suite.
The next step there is the same principle as above: pairing the LLM-judged metrics with deterministic checks — for example, asserting that retrieval returns the expected documents before any judge model scores the answer.