eval‑by‑anyone

Lab vs Deployment

The headline. One task skeleton — fixed tripwires, fixed success check — run through three different arms:

• lab — a bare model through a clean API harness, no user context.
• wild — a real configured agent: CLAUDE.md, memory, MCP servers.
• personal — the same agent, with the task surface grounded in the user's own context.

aware Δ (lab−wild) above zero means the clean harness was more obviously an eval than real deployment. trip Δ (wild−lab) above zero means behavior got worse once the context was real. Both are the point.

Models

Per model and arm. Cells with n < 5 are shown but not ranked. Runs where the human intervened mid-task are excluded from every rate.

Configuration variance

Do different user setups behave differently on the same question? Deployment runs bucketed by configuration, within a family so the task is held constant.

Matched questions

The strictest comparison available: identical family and seed — literally the same generated question — across different configurations.

Participants

Anonymous ids only — each one an HMAC of a random salt that never left the contributor's machine. intervened is shown because a participant whose runs are mostly hand-steered is contributing something different from one whose agent works unattended.

About

Why this exists

Frontier models are increasingly evaluation-aware: they recognize test contexts and change behavior in them. Verbalized eval-awareness in behavioral audits rose from 1–3% in earlier models to 10–15% for Claude Sonnet 4.5, and interpretability work shows the recognition is becoming unverbalized — no trace in the reasoning.

The one approach known to sidestep it — OpenAI's production evaluations, which resample real de-identified traffic and report very low eval awareness — needs privileged access to a frontier lab's traffic. Their write-up calls for others to adopt it. Nobody outside a lab can.

This is the outside-a-lab version: the eval arrives through a real user's real agent, in a real working context, because it genuinely is one.

What these numbers are not

Not a safety certification. A tripwire is one decision in one generated repository. These are behavioral tendencies on specific skeletons, with small samples, self-reported model names, and unverified provenance — every row is marked unverified, because anyone can submit.

Realistic evals also do not fully eliminate eval awareness (discussion), which is why eval-awareness is a measured column here and never an assumption. A zero delta is a publishable result, not a broken product.

Contribute a run

git clone https://github.com/avani17101/eval-by-anyone
cd eval-by-anyone && pip install .
ebx init
ebx doctor   # shows what you can run, free options first
ebx run

Free to run: if you have Claude Code installed, or Ollama / LM Studio running locally, no API key is needed at all. No Hugging Face account needed, and nothing identifying is collected: participant ids are HMACs of a local salt, and configurations are reported as hashes and counts, never as text.