YI EVAL DOCS
Bring testing into your workflow.
The Yi Eval docs live with the product: from connecting your first agent to datasets, scorers and experiments, fault and attack tests in a sandbox, the release gate, production logs and the API. Start from any section below; links open the docs in your current language.
Quickstart
Pick a way to connect; you can switch later, or use several in one project, with results in the same experiments, logs and release gate. With a GitHub repository, your first report takes about 30 minutes. The SDK, the AI Gateway, your own image and a chat AI by its API address are in development.
Repository sandbox
For agents that call tools and change data. The platform clones, installs and starts it in an isolated cloud sandbox, runs it under normal, fault and attack conditions, and deletes the sandbox afterwards.
02Local test runner
For agents that depend on internal systems, or code that must stay in-house: tests run on your own machine, and only results and evidence are uploaded.
03Logs
For apps already running with OpenTelemetry: every model and tool call in Logs within 2 minutes, then online scoring and alerts.
Guides
Datasets and scorers
Versioned rows from tables, logs, experiments or AI; built-in, LLM-judge, code, trajectory and human scorers.
Experiments
One version run over a dataset with its outputs and scores, compared row by row with the baseline: what got better and what got worse.
Release gate
Rules over scores, regressions and required tags give every version one verdict: “can ship” or “failed the release gate”.
Scenarios and test sets
Have AI draft tests, generate attack tests, import a test set and set your pass criteria.
Sandbox runs
Start a run, follow it, cancel it, and finish one that stopped early.
Full run reports
The verdict, the conversation, tool calls and data checks behind every problem, and fix notes for your coding AI.
Logs
Traces from production: filters, facets and the span tree, each with its cost and latency.
Online scoring and alerts
Score a sample of live traces; get mail, Slack or a webhook when a score, the error rate or the cost goes wrong.
Automations
Run on push, pull request or schedule, post a check on GitHub commits, or start runs from your own CI.
From a coding AI (MCP)
The coding AI on your dev machine connects over MCP or the CLI: start tests, read reports and fetch fix notes.
Organization & members
Signing in with your Yi Labs account, your organization, roles and limits.
Billing & usage
Your plan and its credit, upgrading, top-ups, and what you have used and paid.
Reference
review.yaml
How to start your agent, route its tool calls, reset and read your test backend, and judge the results.
CLI
reviewctl: sign in, run a local test runner, run tests on your machine, connect a coding AI.
API
Project API tokens: start runs from CI and read their results.
CI and the GitHub Action
Put the release gate in your pipeline: exit code 0 when it passes, 1 when it fails, and the verdict as a pull request comment.
Secrets
Model keys, test account tokens and registry sign-ins a cloud run needs.
Sandboxes
Sizes and architectures, what is installed, what it can reach and its limits.
Security & legal
Security overview
Isolation of cloud runs, encryption, data residency, retention and deletion.
Security questionnaire
Answers to what vendor-security reviews usually ask, as things stand today.
Data processing
How personal data is handled under the UK GDPR, and the sub-processors.
Legal (drafts)
Terms of Service, Privacy Policy, Acceptable Use Policy, Data Processing Agreement, Service Level Agreement.
Service status
Live status and incident history for the platform.
Common questions
Do I need an activation code?
No. Sign in to Yi Eval with your Yi Labs account and your personal workspace is created on first use. GitHub and Google sign-in are available. Contact admin@nice2.io if an existing account doesn't link up.
Where is my data?
Everything the platform stores is in London. Cloud runs happen in London by default, each on a fresh virtual machine deleted afterwards; a project set to run in the EU runs in Sweden (EU), and its run bundle is still stored in London. The platform's models (global deployments, which may process it in any region) receive the text they grade or answer; the security overview has the detail. Code that can't leave your company can use a local test runner.
What does “passed” mean?
Within that test's scope, every pass criterion you set was met. Nothing outside the scope was checked, so a pass doesn't mean the system is completely safe; anything that didn't finish or couldn't be decided is called out in the report and doesn't count as passed.
Next step