Ori Eval

Ori Eval answers “which model is best for what I’m building?” with evidence instead of vibes. Ask the question in plain words and the agent finds test material in your repo, writes the eval, runs your candidates, and comes back with a ranked answer. Under the hood an eval is a *.eval.ts file — you can write one by hand whenever you want the control, but you never have to start there.

Start with your agent

The fastest path is to not run anything yourself. Tell your coding agent:

go to https://openrouter.ai/skills/spawn-ori-eval to get started

The spawn-ori-eval skill walks the agent through everything: it installs Ori, checks ori login, and starts ori code -p with your request. Inside the run, Ori’s create-eval skill asks the questions that shape the eval — which surface to test, what it needs to be good at, whether you have real data, and the cost ceiling — then writes it, runs your candidates, and reports back with a recommendation.

Why run evals through Ori

The hardest part of evals is keeping the test bench stable. Ori is a harness, not just a skill: it pins the model and settings at a level prompts can’t, so a score change means your agent changed, not the environment. And because Ori routes through OpenRouter, a bakeoff can cover every major lab’s models — not just the one vendor your usual harness supports.

Run it manually

Everything below is the same flow by hand — use it when you want the control, or to understand what the agent built.

Before you start

Install the CLI:

curl -fsSL https://openrouter.ai/labs/ori/install.sh | sh

ori login

An eval calls a real model, so a credential is required — ori login stores one for Ori to use. Running evals also needs Bun; if it is missing, ori eval offers to install it on an interactive terminal.

Your project does not need to be TypeScript, and TypeScript never needs a separate install: the eval file tests the agent, not your codebase.

Start from a question

From your project directory, ask Ori to run the eval:

cd my-project
ori code -p "What's the best model for my support agent?"
# For a long request:
ori code --prompt-file /tmp/ori-task.txt

ori code -p runs without a TTY, streams output to stdout, and exits when the prompt completes. Use --prompt-file instead for a long request; the flags are mutually exclusive and positional prompts are rejected. You do not need to know eval methodology or model names. The agent:

Scans your repo for test material — existing prompts, tool definitions, data files (.jsonl, .csv, chat logs), and gold answers. Tests in any language count: pytest files and Go tests get mined for cases, even though the eval itself is always TypeScript.
Asks what matters — a question or two like “accuracy first, or speed and cost?” — only when the answer changes the eval.
Writes and runs the eval — scaffolds evals/<feature>/<name>.eval.ts, seeds candidate models from OpenRouter’s live catalog under your price cap, and runs them.
Recommends one model, with the scores, latency, and cost that back it up.

Every artifact stays in your repo as plain code: re-run it when a new model drops, tighten the criteria, or put it in CI.

Under the hood: the eval file

When you want control — or want to understand what the agent built — here is the whole surface. An eval reads like a normal bun test:

evals/support/recommends.eval.ts

import { test } from 'bun:test';
import { setupAgent } from 'ori/eval';

const agent = setupAgent();

test('recommends restaurants using the search tool', async () => {
  const run = await agent.run('Where should I eat dinner in Lisbon?');

  run.tool('search').toBeCalled();
  run.tool('delete_file').toNotBeCalled();
  run.toComplete();
  run.toCostAtMost(0.01);
  run.toFinishWithin(30_000);
});

setupAgent() with no arguments is your workspace’s own resolved harness and model — the same agent you already run. Then:

ori eval
ori eval --report eval-report.md
ori eval --baseline best

Ori discovers every *.eval.ts in the workspace — keep them under top-level evals/<feature>/ directories — boots an ephemeral runtime, and hands the files to bun test. The exit code mirrors bun test, so a failing eval fails CI. Use --baseline last|best|model:<slug> to compare against an earlier run recorded in .ori/eval/history.jsonl, and --report <path> for a shareable Markdown report. Baselines only compare runs that covered exactly the same eval files. If a bakeoff candidate is cut off before it answers, the shareable report keeps it visible as unmeasured instead of silently dropping it.

Compare models

Seed candidates from OpenRouter’s live catalog instead of hand-listing slugs, cap the price, and run the same eval against each:

evals/support/model.eval.ts

import { test } from 'bun:test';
import { candidateModels, setupAgent } from 'ori/eval';

const candidates = await candidateModels({
  limit: 5,
  maxPromptPrice: 0.000005,
});

for (const model of candidates) {
  test(`handles a refund request on ${model}`, async () => {
    const run = await setupAgent({ model }).run(
      'A customer wants a refund for order #1234. What do you do?',
    );
    run.tool('lookup_order').toBeCalled();
    run.toComplete();
  });
}

candidateModels can also filter by maxCompletionPrice, minContextLength, quality indexes, required parameters or input modalities, and excludeExpiring, so unknown catalog values do not silently pass a bound. Use assertModelIsLive(slug) when you pin an incumbent model, so the eval fails clearly if it leaves the live catalog.

Grade open-ended answers

When there is no single right string, grade with an LLM judge. The judge defaults to a dedicated agent on a strong grading model, so scoring stays independent of the model under test:

evals/support/quality.eval.ts

import { test } from 'bun:test';
import { setupAgent, setupJudge } from 'ori/eval';

const agent = setupAgent();
const judge = setupJudge({ minScore: 0.8 });

test('gives an accurate, grounded answer', async () => {
  const run = await agent.run('What is our refund policy for digital goods?');
  await judge.autoEvals({
    criteria: 'Cites the 14-day window and does not invent exceptions.',
    run,
  });
});

For common dimensions, startingCriteria.accuracy, completeness, instructionFollowing, safety, structuredOutput, and toneAndVoice provide editable starter rubrics for judge.autoEvals.

Eval against your own data

Real usage data beats synthetic prompts. Loop chat logs or Q&A pairs with test.each:

import supportPairs from './support-pairs.json';

test.each(supportPairs)(
  'answers: $question',
  async ({ question, mustMention }) => {
    const run = await agent.run(question);
    run.toMention(mustMention);
    run.toComplete();
  },
);

Run it in CI

Evals spend model calls, so put them in a credentialed, opt-in job — not your default unit-test run — with the key as a secret. A failing eval fails the build, so a worse agent stops the ship instead of your users. ori eval --list --allow-no-key needs no key and cheaply asserts evals exist.

Overview

Models & Routing

Features

Ori

Privacy

Best Practices

Community

Start with your agent

Why run evals through Ori

Run it manually

Before you start

Start from a question

Under the hood: the eval file

Compare models

Grade open-ended answers

Eval against your own data

Run it in CI

​Start with your agent

​Why run evals through Ori

​Run it manually

​Before you start

​Start from a question

​Under the hood: the eval file

​Compare models

​Grade open-ended answers

​Eval against your own data

​Run it in CI

Start with your agent

Why run evals through Ori

Run it manually

Before you start

Start from a question

Under the hood: the eval file

Compare models

Grade open-ended answers

Eval against your own data

Run it in CI