Verification
Every Code turn tries to prove that the change didn't break what worked before. It runs a test command on the starting code first (the baseline), then again after the agent's change, and reports one of three results.
The three results
| Result | Meaning |
|---|---|
| "Validation passed" | The command passed before the change and passes after it. |
| "Validation failed" | The command passed before the change and fails after it, or it timed out after the change. The turn ends as Failed. |
| "Unverified" | No trustworthy comparison was possible. The reason is shown under the result. |
A turn is never reported as passed when nothing ran. Unverified is not a failure. The change may be fine, but nothing checked it.
How the command is chosen
If an administrator set an "Independent validation command" on the project in "Settings", that command runs. Otherwise Yi Code looks at the repository at the pinned commit, without installing anything or running repository scripts, and takes the first of these that applies and whose tool is in the runtime image:
| Found in the repository | Command |
|---|---|
package.json with a real test script | npm test |
pytest.ini, pyproject.toml, setup.cfg, conftest.py or a tests/ folder | Protected Python test runner |
go.mod | go test ./... |
Cargo.toml | cargo test --offline |
pom.xml | mvn -o test |
build.gradle or build.gradle.kts | gradle --offline test |
Makefile with a test target | make test |
A test script that is npm's placeholder ("no test specified") doesn't count.
For Python, the command runs the top-level tests/ folder from the starting commit, kept as a separate copy that the agent can't change. A Python project whose tests live somewhere else, or that has pyproject.toml or setup.cfg but no top-level tests/ folder, gets nothing to run, so its baseline fails. Other commands run in the working tree as the agent left it.
The default runtime image has Python and Node.js. For Go, Rust, Java or make projects the tool isn't there, so the turn is Unverified with "The execution environment lacks the required toolchain." unless the deployment uses a runtime image that includes it.
Python command
The Python runner imports the installed pytest before adding the project to its import path. It ignores PYTHONPATH, pins plugin discovery to installed packages and uses the protected acceptance copy for the test root and configuration. The command shown in the run's raw logs is:
python -I -c "import sys, importlib.metadata as metadata, pytest; installed = list(metadata.distributions()); metadata.distributions = lambda **_: iter(installed); sys.path.extend(['/workspace/repo', '/workspace/repo/src']); raise SystemExit(pytest.main(['-p', 'no:cacheprovider', '--rootdir=/workspace/acceptance', '--confcutdir=/workspace/acceptance', '/workspace/acceptance/tests']))"Time limits
Each run of the command, before and after the change, may take up to 180 seconds. A baseline that takes longer makes the turn Unverified. A check after the change that takes longer counts as failed.
Reasons for Unverified
| Shown as | Cause | What you can do |
|---|---|---|
| "No usable validation command was found." | None of the files above is in the repository, and the project has no command set. | Add a test script, or ask an administrator to set the project's validation command. |
| "The execution environment lacks the required toolchain." | The repository needs a tool the runtime image doesn't have. | Use a runtime image with that toolchain. On the cloud service the image is ours. |
| "The starting code failed validation, so regressions cannot be determined." | The command already fails on the pinned commit. Common causes: a dependency that isn't in the image, since the sandbox can't download anything, or a Python project with no top-level tests/ folder. | Look at tests.txt under "Raw logs and artifacts". Fix the tests on the branch, move Python tests to a top-level tests/ folder or ask an administrator to set the project's validation command, or ask for the change anyway and review it yourself. |
| "Validation of the starting code timed out." | The baseline ran longer than 180 seconds. | Make the default test command faster, or set a narrower command on the project. |
| "Validation detection is unavailable in this execution environment." | The detection step couldn't run in this sandbox. | Try again later. If it persists, tell your administrator. |
| "Ask is read-only; no change validation ran this turn." | Ask turns don't change code, so nothing is verified. | Nothing to do. |
| "Validation has not completed for this turn." | The turn ended before verification ran, for example because it failed or was cancelled. | Check the turn's failure message. |
When validation fails
"Validation failed after the change." means the agent broke something the tests covered. The turn ends as Failed, and you can't create or update a pull request from it.
- Open "Raw logs and artifacts" and read
tests.txt. It has the output of both runs. - Click "View diff" to see what changed.
- Ask a follow-up in Code mode, for example "The tests in test_dates.py fail after your change. Fix them without changing the expected values." The next turn starts from the latest saved changes.
No network in the sandbox
The sandbox has no outbound network. The command can't run npm install or pip install or reach a database on the internet. Anything the tests need must already be in the runtime image or in the repository. Tests that call outside services will make the baseline fail, and the turn will be Unverified.