Methodology

Evidence, before confidence.

AI releases change behavior across distributions of tasks. Unit tests verify the platform; evaluation measures whether a particular target meets the task and safety contract.

Ground truth is an explicit contract

Deterministic values, human labels, external benchmark scorers, injected incidents, subjective rubrics, and exploration are different kinds of evidence. EvalForge keeps each origin / ground-truth cohort in a separate immutable dataset version.

One critical failure can outweigh an average

Task quality and safety are measured separately. A higher average does not grant authorization to export customer data. A single critical safety failure blocks the candidate regardless of its average.

Grading starts with the simplest valid oracle

Exact and numeric graders establish objective correctness. Structured graders inspect observed tools, evidence, abstention, approval order, and loops. A model judge is optional for narrow rubrics, with its model and prompt recorded.

Human judgment is evidence, too

Reviewers record PASS or FAIL, a 1–5 rating, comments, and multiple failure labels. Judge agreement uses the latest human review of the same output. No human labels or agreement metrics are fabricated.

Splits reduce overfitting; they do not eliminate it

Tune on development, choose on validation, then freeze the candidate before explicitly evaluating held-out cases. This small authored demo is not independently blinded. Repeated held-out access weakens the evidence.

External benchmarks complement controlled incidents

AgentGate controlled simulations and AgentDojo native-tool replay are separate projects. The replay uses real upstream scorers but scripted proposals; it does not measure live-model prompt-injection susceptibility. Source files and hashes are retained.

Economics should make uncertainty visible

Latency is measured around execution. Target cost comes from observed tokens and versioned price configuration. Unknown usage or prices stay unknown. Local demo calls have zero model cost; compute and human costs are excluded. Model-judge usage is separate evidence.

Inspectable storage and adapter boundaries

The TypeScript engine owns hosted PostgreSQL records and release policies. Adapters receive only case inputs, not ground truth. HTTP adapters support AgentGate, Data Sleuth, RevLeak AI, Tactical AI, PIM Reviewer, and SafeNest without coupling the engine to a vendor.

Continuous evaluation

npm run eval -- --demo

Exit 0 means release; exit 1 means a gate blocked the candidate; exit 2 means the command could not complete. The hero demo should exit 1.

npm run typecheck
npm run lint
npm test
npm run build