Validating a Generated Test
A generated test is a draft until something runs it. Validation executes one generated test through Cycle's normal execution tooling and reports how far it got, so you learn whether the conversion produced a working test before you invest in reviewing it line by line.
Validation is deliberately narrow. It checks one Feature File per run, and it is meant for a sandbox rather than a live environment.
Prerequisites
Validation executes a test, so it needs more than authoring did:
- A Cycle project to run the test in.
- An environment the test can actually reach, such as a non-production sandbox with the application under test available.
- A signed-in Cycle account, and a Cycle user profile if the test requires one.
Validating a test
Ask the agent to validate a specific generated test.
You say:
Validate
Test Cases/AdjustStorageLocation.featureinC:\CycleProjects\WarehouseSuite.
The agent:
- Checks the file's syntax with the CycleScript compiler.
- Runs the test with
cycle-cliand captures the results. - Reads the structured errors from the run and classifies any failure.
- Reports a staged verdict and, on failure, the specific failing steps.
Generating tests has a guided prompt.
Validating does not, so ask for it in plain language as shown above.
The agent calls formulation_validate-generated-test, the Cycle MCP tool that performs it.
Naming that tool is worth knowing: it is a single dedicated operation rather than a sequence the agent assembles itself, which is why the verdict comes back in defined stages.
Reading the staged verdict
The verdict reports each stage in order, so a failure tells you where the conversion fell short rather than only that it failed.
| Stage | Question it answers |
|---|---|
| Parsed | Is the CycleScript syntactically valid? |
| Steps resolved | Does every step correspond to a real Cycle step, with valid parameters? |
| Locators and variables bound | Did the locators find their elements, and did the variables resolve to values? |
| Executed | Did the test run to completion and pass? |
A test that clears all four stages is reported as validated.
Acting on the outcome
The outcome tells you whether to change the test or change the environment. This distinction matters, because refining a correct test against an environment problem wastes effort and can make a good test worse.
| Outcome | What it means | What to do |
|---|---|---|
| Validated | The test parsed, resolved, bound, and executed successfully. | Review it and move on. Finalize it against a live environment when you are ready. |
| Not yet runnable | The file is a stub, so there was nothing to execute. | Resolve the stub's open questions and convert the specification again. See Generating the tests. |
| Not validated, classified as a locator or data problem | The test content is wrong. A locator did not match, or a data value was missing or invalid. | Refine the test. This is the case where editing the Feature File is the right response. Use the debug-a-failed-run prompt, which inspects the live page and compares the failing locator against what is actually rendered. |
| Not validated, classified as an environment, timing, or popup condition | The environment interfered with the run. The test content is not implicated. | Do not refine the test. Fix the condition and validate again. |
| Not validated, unclassifiable | The failure could not be attributed to either the test's bindings or the environment. | Investigate before editing. Treat the cause as unknown rather than assuming the test is wrong. |
| Environment error | Cycle's tooling could not start the run at all. | Fix the local setup. See Troubleshooting. This is reported separately from test content precisely so it is not mistaken for a test failure. |
When a test is not validated, the agent reports the failing steps with their errors rather than the whole run log, so the relevant detail is in front of you.
From there, a generated test is an ordinary Cycle test.
The debug-a-failed-run prompt is the shortest path on a content failure, because it gathers the run evidence and then opens the page to check the locator against the live accessibility tree.
The same approach is described at greater length in Workflows.
When you edit or re-author a generated test, preserve the # Spec-Coverage: comment in its header.
That line is how a later conversion run knows the specification is already covered.
A test that loses it looks uncovered, and the specification is converted again.
A validated verdict means the test executed successfully in the environment you validated against. It is an execute-and-confirm signal from your sandbox, not approval against production data or a live WMS. Use finalizing for that.