A useful agent evaluation begins with observable acceptance criteria, runs deterministic checks on the exact submitted revision, and limits rubric scoring to qualities that tests cannot establish directly.
Translate the assignment into observable claims
List what must be present, what behavior must change, what must remain unchanged, and what evidence will establish each claim. A file existing is a structural check. A command returning the expected result is a behavioral check. Readability or explanatory quality may require a rubric. Keep these categories separate so a subjective score does not compensate for a failing executable requirement.
Run checks against the exact submitted revision in a recorded environment. Save the command, inputs, relevant fixture versions, result, and output excerpt. Include negative cases and unchanged behavior when the assignment could create regressions. If a test cannot run because a dependency is unavailable, report unavailable evidence. Do not convert the inability to test into a passing judgment.
Make rubric judgments cite their evidence
A rubric should define dimensions, anchors, disqualifying conditions, and the artifacts a grader may inspect. OpenAI's grader interface documents grader configurations for evaluating model outputs, while LangSmith describes evaluations over datasets and traces. Those systems still require a defined question and reference evidence. A score without a cited line, test, trace, or output is difficult to audit when graders disagree.
Calibrate the rubric on a small set of accepted and rejected examples. Compare automated scores with a human reviewer and record disagreement by dimension. If a grader rewards verbosity, misses a safety boundary, or varies materially across repeated runs, narrow the prompt or replace that dimension with a deterministic check. Keep the raw grader result so a later threshold change does not rewrite history.
Separate evaluation from acceptance authority
The packet should show passed checks, failed checks, rubric scores, cited evidence, reviewer notes, unresolved conflicts, and the exact revision. It can recommend accept, revise, retry, or stop under written rules. The designated buyer reviewer remains responsible for accepting work, especially where a change affects architecture, customer data, permissions, external actions, or another team's ownership.
Reality Contact, LLC installs the evaluation path for Fleet Completion Control. The buyer provides the criteria, examples, repositories, and final reviewer. The resulting scores describe the named artifacts under the accepted tests and rubric. They do not establish general code quality, eliminate review, or authorize a merge or production release.
Where the service stops
Reality Contact, LLC installs bounded completion and evaluation controls but does not manage the buyer's staff, declare disputed work correct, merge or deploy changes, delete worktrees, reassign tasks, or operate the fleet indefinitely. The buyer defines completion and review authority, approves rubrics and stopping rules, resolves disputed judgments, controls repositories and credentials, and authorizes every merge and deployment. This is technical implementation, not legal advice. Automated evaluation supports named criteria; final technical, security, legal, and production decisions remain with the buyer. We do not promise conflict-free parallel work, correct grader judgments, successful merges, or coverage beyond the accepted sources, criteria, and scenarios.
Sources: OpenAI API reference for graders; LangSmith documentation for evaluation.