AI can help a software team identify test cases, write test code, investigate failures and maintain a regression suite. Its usefulness depends on what the tests actually establish. A passing test that checks the wrong expectation can make a defect harder to notice.
The most productive place to begin is a behavior the team can describe clearly. Give the agent the requirement, relevant code and existing test conventions. Review its proposed cases, run the tests and inspect whether a plausible mistake would cause them to fail.
This approach keeps testing connected to the reason for the change. It also produces evidence that another developer can use when the work moves between people, agents and releases.
A small import change with several ways to go wrong
Suppose a software company is updating a bulk import feature. Customers upload rows containing a product reference and an order quantity. The feature should accept quantities from 1 through 100, report invalid rows clearly and avoid creating orders from a rejected preview.
An agent working on the importer can generate many tests quickly. Yet several important questions are outside the quantity parser. What happens when the same product appears twice? Does a preview write anything? If a customer corrects one row, does the next preview still contain the original error? Can someone import into an account they do not administer?
Separate the agreed behavior from questions that need a product decision. An assistant may propose rejecting duplicate products, but the intended feature could require combining their quantities. Turning that proposal directly into a test would silently create a requirement.
Write a short acceptance record before generating code. For this example, it should cover input format, row validation, duplicate handling, preview behavior, access and final submission. The product owner resolves ambiguous behavior. Engineering identifies which existing components already implement it.
Microsoft's guide to testing existing code with AI recommends using requirements to establish expected behavior and inspecting generated assertions. Tests inferred entirely from the current implementation can preserve its mistakes.
Decide what each test needs to observe
A unit test can check one small function in isolation. An integration test can establish whether components work together. A browser test can exercise the customer interaction. Each should have a specific reason to exist.
For the importer, a useful division might be:
| Concern | Useful observation | Appropriate test boundary |
|---|---|---|
| Quantity validation | Exact accepted values and rejected inputs | Parser unit tests |
| Preview behavior | Validation results appear without creating orders | Import service with its persistence boundary |
| Account access | An unauthorized account cannot receive an import | Authenticated service boundary |
| Correcting a row | Updated input replaces the old validation result | Preview flow or browser interaction |
| Final submission | Accepted rows produce the intended records once | Submission flow with persistence checks |
The table is a plan for evidence, not a requirement to implement every concern through a full browser test. Keep inexpensive, deterministic checks close to the behavior they establish. Use broader tests for interactions that a small isolated test cannot observe.
Before asking an agent to add a suite, have it inspect the repository's framework, fixtures and commands. Reuse a representative existing test. Adding another runner or a parallel fixture system creates maintenance work that may have little to do with the feature.
Make the expected result independent of the implementation
For the parser alone, suppose the agreed input is a string of decimal digits representing a whole number from 1 through 100. Leading zeroes, spaces and scientific notation are rejected. These format decisions should be explicit because a general numeric conversion accepts some of them.
The following standalone JavaScript example includes a small implementation and checks for that contract. It uses Node.js and no external packages.
import assert from 'node:assert/strict';
function parseQuantity(raw) {
const value = typeof raw === 'string' ? Number(raw) : NaN;
if (!Number.isInteger(value) || value < 1 || value > 100 || String(value) !== raw) {
throw new RangeError('Use a whole number from 1 to 100');
}
return value;
}
for (const [raw, expected] of [['1', 1], ['37', 37], ['100', 100]]) {
assert.equal(parseQuantity(raw), expected);
}
for (const raw of ['', '0', '101', '2.5', '12x', ' 5', '05', '1e2', '5\n', null]) {
assert.throws(() => parseQuantity(raw), RangeError);
}
console.log('Quantity checks passed');Save it as quantity.mjs and run node quantity.mjs. The valid expectations are written as literal values. They are not calculated by calling the parser again or copying its internal logic into another helper.
The comparison with the number's standard string form rejects alternate spellings, including extra whitespace and scientific notation. The explicit bounds and integer check establish the accepted numerical range.
Notice what these checks do and do not observe. They verify the quantity function's returned values and rejection behavior for the listed inputs. They do not establish whether the import preview creates database records, whether account permissions are enforced or whether a customer understands the error message. Those need checks at the relevant boundaries.
The example also exposes a useful review question: if the upper limit accidentally became 101, would a test notice? The rejected input 101 checks exactly that mistake. If the implementation returned the parsed quantity plus one, the valid examples would catch that different defect.
Check the tests by introducing a controlled defect
Code coverage shows which parts of the program ran during testing. It does not establish that the assertions would detect an incorrect result. A test can execute the parser and discard its output, covering code without checking its meaning.
On an isolated copy, deliberately change the upper limit in the example from 100 to 101. The rejection check should now fail. Restore the original condition before continuing. Repeat with a different plausible defect, such as changing the returned number, if that helps inspect another important expectation.
This is the idea behind mutation testing: make controlled changes to code and check whether tests detect them. A surviving change deserves investigation, although some changes do not alter observable behavior and cannot be distinguished by a test.
For a larger codebase, focus this work on important behavior and changed code. A mutation score is another signal to interpret, not a substitute for deciding whether the contract itself is correct. Avoid expensive, indiscriminate runs when a few targeted checks can answer the immediate review question.
AI can propose plausible defects to test against. A developer should decide which ones represent meaningful risks, and the test runner should establish the result. Keep intentionally broken copies isolated from the branch being prepared for release.
Keep mocks from answering the question for you
Mocks make a dependency controllable during a test. They become misleading when they replace the behavior the test is supposed to verify.
Consider the import preview. If the test replaces the preview service with a mock configured to return “no orders created,” asserting that message proves little about persistence. The check needs to observe whether the real preview path attempted or performed a write at the appropriate boundary.
Similarly, a permission test should exercise the relevant authorization path. Replacing the whole request handler with an allowed response removes the decision being tested. Use a test account or fixture with the intended permissions, then inspect the actual response and relevant side effects.
Ask the agent to explain the role of each major mock. Which external behavior is being controlled? Which production behavior remains under test? What would fail if the requirement were violated? These questions are often more useful than asking for a larger number of tests.
Keep test inputs independent, especially around time, shared state and execution order. A test that passes only after another test creates an account can conceal a missing setup step. A test that depends on a live third party service can fail because of network conditions unrelated to the change.
Investigate a failing test before asking AI to fix it
A failure may reveal a product defect, an incorrect expectation or a problem in the test environment. Giving an agent the broad instruction “make the suite pass” leaves room for the wrong response.
For the import change, imagine that a preview test reports a newly created order. Inspect the execution path and fixture before changing anything. If preview is supposed to be read only, weakening the assertion to accept one order would erase the signal that matters.
Ask for a diagnosis with the failed command, relevant output, expected behavior and proposed explanation. Require a separate decision before changing implementation, acceptance criteria or established tests. Preserve a failing regression test when it captures a confirmed bug.
Then run the smallest relevant selection, followed by the related suite. Record failed, skipped and unexecuted tests separately. An agent's statement that it “verified the change” should be supported by the command and result, including the revision and environment needed to interpret them.
Retries need interpretation too. A test that passes on its third attempt may be unstable. A framework's automatic retry can make a useful interface test less disruptive while still requiring the team to investigate the underlying instability.
Share the test reasoning across the release
Testing work often outlives the chat in which the code was written. A reviewer needs the requirement, the important risks, the evidence produced and any remaining uncertainty. The next release needs to know why a particular regression case exists.
Swarm Spaces can keep that shared record alongside Tasks, Artifacts, discussion and agent Runs. For the importer, a Space can hold the agreed import behavior, a reviewed test plan, execution evidence and the product decisions that resolved ambiguity.
An implementation agent and a review agent can contribute different findings through their authorized tools. The reviewer can inspect the requirement and evidence before accepting the change. When a customer later reports a related failure, the team can continue from the saved record and add the confirmed regression case. The connection guide explains how agents work within selected Spaces.
Keep the repository as the place for executable tests and project conventions. Use the shared work record to retain decisions and findings that would otherwise disappear between tools or participants. Link the evidence to the relevant code revision so a passing result is not mistaken for proof about a later change.
A release decision needs more than a green count
Before accepting an AI assisted testing contribution, ask whether the tests check the agreed behavior, would catch representative defects and actually ran in the intended environment. Look for changes to old assertions, skipped cases and mocks that remove the important interaction.
The guide to LLM evaluation addresses evaluating model outputs and agent behavior. Application testing asks a related but distinct question: does the software behave as required? A feature using AI may need both forms of evidence.
For the import feature, a useful release record would show checked quantity boundaries, a preview that leaves orders unchanged, enforced account access and a verified submission path. It would also identify any remaining risks and who accepts them.
The benefit of AI is a testing process that can cover more of the reasoning and routine work while preserving the team's ability to judge the result. Good tests make an incorrect behavior visible. Good shared records help the next person understand why that visibility matters.