Designing AI evaluation rubrics with domain experts
A rubric is only as good as the design process that produced it. This is how to run that process with domain experts, from eliciting tacit criteria to calibrating graders until they agree.
To design an AI evaluation rubric with domain experts, you run a structured process that pulls the tacit standard of correct work out of practitioners’ heads and turns it into named dimensions, explicit levels, and anchored examples that any qualified grader can apply the same way. The rubric is the instrument that makes evaluation repeatable, and the design process is what determines whether that instrument measures real correctness or just surface polish.
This post is a how-to for teams that buy verified human data to train and evaluate AI. It assumes you understand why rubrics matter and focuses on running the design engagement itself, step by step, with the experts in the room. If you need the concept first, our explainer on evaluation rubrics for AI models defines the term and makes the case for practitioner-defined criteria over generic scoring. This post is not professional advice in any field it references.
Why the design process, not just the rubric, decides the outcome
A rubric encodes a standard of correctness, and that standard only exists in the minds of people who do the work. This is why a rubric written by a generalist, however carefully, tends to reward answers that are clear and confident rather than answers that are right. The design process is where the real standard either gets captured or gets lost.
Where evaluation sits in the wider pipeline is covered in our overview of what AI training data is. The point specific to rubric design is that evaluation closes the loop on training. Demonstrations teach a behavior and preference data refines it, as our guide to preference data for AI explains, but none of it is trustworthy until you can grade outputs consistently. The design process is what earns that trust.
The same principle appears in how frontier labs specify desired behavior. Anthropic’s work on Constitutional AI shows a model being steered by an explicit written set of principles rather than case-by-case intuition, and a rubric plays the same role for human grading: it makes the standard explicit so it can be applied consistently and inspected later.
Step one: scope the task and its failure modes with experts
Rubric design starts before anyone writes a dimension. It starts by deciding, with the experts, exactly what the model is being graded on and what the dangerous failures look like. A rubric built without a clear picture of the failure modes will measure the easy things and miss the ones that matter.
In this first session, the experts should articulate:
- The specific task the output is meant to accomplish, in the terms a practitioner would use.
- The ways a plausible-looking output can still be wrong, which are usually invisible to non-experts.
- The relative stakes of each failure, since a safety error and a formatting error do not deserve equal weight.
This is also where you decide what the rubric will not cover, because a rubric that tries to grade everything grades nothing well.
Step two: elicit the tacit criteria
The hardest part of design is that experts often cannot state their standard in the abstract. They know a bad output when they see one, but the reasoning behind that judgment stays implicit until something forces it out. The reliable way to force it out is to have experts grade real model outputs and explain each decision.
The key facilitation move is to ask why, not whether. When an expert flags a response as wrong, the follow-up is always what specifically makes it wrong and how a correct version would differ. Those explanations are the raw material of the rubric, because each one is a criterion waiting to be written down.
Running these sessions at volume is where tooling earns its place. AI Interview Agents let you run structured grading-and-explanation sessions with many experts at scale, so the elicitation is not limited to whoever you can get into a live workshop. This is the same elicitation discipline we describe for collecting expert demonstrations for AI at quality, applied to grading instead of producing work.
Step three: translate criteria into dimensions, levels, and anchors
Once you have the raw judgments, you convert them into a structured rubric. A rubric that produces consistent scores has four parts, and the design process has to deliver all four.
| Rubric part | What it is | How experts shape it |
|---|---|---|
| Named dimensions | The distinct aspects of quality that matter | Experts name what they actually judge, such as accuracy, completeness, and safety |
| Explicit levels | A written description of what earns each score | Experts describe the difference between a top score and a near miss |
| Anchored examples | Real graded outputs at each level | Experts pick examples that illustrate each level unambiguously |
| Weighting | How dimensions combine into a result | Experts set weights to reflect real-world stakes |
The anchored examples are the part most rubrics lack, and their absence is why scores drift. When every level is illustrated by a real graded example, a grader is matching an output to a written standard rather than guessing, and two qualified reviewers converge on the same score. Getting experts to choose and annotate these anchors is the single highest-value hour of the design process.
Step four: calibrate and measure agreement
A draft rubric is a hypothesis, and calibration is the test. You take the draft, apply it to a fresh set of real model outputs, and have several experts grade the same outputs independently without conferring. Then you look at where they agree and where they diverge.
Agreement is measured, not eyeballed. A chance-corrected statistic such as Cohen’s kappa for two graders, or Fleiss’ kappa for more, tells you whether the rubric is reliable. When agreement is low on a dimension, the problem is almost always an ambiguous criterion rather than a bad grader, so you rewrite that dimension and test again. A rubric that produces different scores from different qualified graders is not measuring anything stable, and every downstream comparison built on it is noise.
This is the step teams skip and later regret. It is also the step that most resembles how large public evaluations are built. Efforts such as Stanford’s Holistic Evaluation of Language Models and standardized tests like MMLU are useful precisely because their scoring is defined tightly enough to reproduce, and a domain rubric has to clear the same bar to be worth running.
Step five: operationalize for graders at scale
A rubric that works in a design workshop still has to survive contact with a larger grading pool. Operationalizing means packaging the rubric so a grader who was not in the design sessions can apply it correctly: clear instructions, the anchored examples front and center, and a short onboarding where new graders score calibration items and are corrected against the standard before they grade production outputs.
This is where rubric design connects to the people who do the grading. Our guide to model evaluation data and hiring experts to grade AI outputs covers the grading operation itself, and the handoff between design and grading is where many rubrics quietly fail, because the standard that lived in the designers’ heads never made it into the written artifact the graders actually use.
Governance expectations reinforce the point. Documenting the rubric, its dimensions, and its calibration results, in line with the NIST AI Risk Management Framework, means that when someone asks how a model’s quality was judged, you have a defensible, inspectable answer rather than a recollection.
Common design pitfalls to avoid
A few failure patterns show up again and again, and all of them are process problems rather than expert problems:
- Writing dimensions before grading real outputs, which produces criteria that sound right but do not discriminate good from bad.
- Skipping anchored examples, which guarantees drift as the grading pool grows.
- Treating low agreement as a grader quality issue instead of a rubric clarity issue.
- Weighting every dimension equally, which lets a polished but unsafe answer score as well as a correct one.
- Freezing the rubric too early, before the first batch of real grading has exposed its ambiguities.
Avoiding these is less about expertise and more about discipline, which is why the design process deserves as much attention as the choice of experts.
Where the rubric fits next
A rubric defines how to grade, but it needs something to grade against. That is the evaluation set: the collection of questions and gold answers the model is tested on. Designing the rubric and building the set are two halves of the same evaluation program, and our companion piece on building domain-specific AI evaluation sets with verified professionals covers the other half. Together with expert demonstrations, they form a loop in which the same calibrated experts teach the model, test it, and score it to one consistent standard.
How CleverX supports rubric design with experts
CleverX is an on-demand verified-expert platform, not labeling software. It connects AI teams with more than 8 million verified professionals across 150-plus countries, each confirmed through a four-layer process that checks work email and LinkedIn identity alongside credentials and experience.
For rubric design, that means recruiting practicing specialists to surface the real success criteria, write anchored examples, and calibrate the rubric until qualified graders agree. Access is pay-as-you-go, panels typically assemble and begin producing data within roughly two to five days, and AI Interview Agents let you run structured grading-and-explanation sessions at scale when you need many experts involved without a live workshop for each one.
Train your AI with verified experts on CleverX
Frequently asked questions
How is designing a rubric with experts different from writing one yourself?
Writing a rubric yourself encodes your own assumptions about what good work looks like, which rewards fluency and surface polish on specialized tasks. Designing with domain experts elicits the real success criteria that live in practitioners’ heads, including the subtle failures only someone who does the work would catch, so the finished rubric scores whether an output would hold up in practice rather than whether it reads well.
How do you get tacit expert knowledge into a written rubric?
You elicit it through structured sessions where experts grade real model outputs and explain each judgment out loud, then you convert those explanations into named dimensions, explicit levels, and anchored examples. The key move is asking why a response is wrong rather than just whether it is, because the reasoning behind the judgment is what becomes the written criterion.
What does a good evaluation rubric contain?
A good rubric has named dimensions for the distinct aspects of quality that matter, explicit levels describing what earns each score, anchored examples at each level, and a weighting rule for how dimensions combine. The anchored examples are what most rubrics lack, and their absence is the main reason scores drift between otherwise qualified graders.
How do you know a rubric is reliable?
You measure inter-rater agreement by having several experts grade the same outputs independently and checking whether they reach the same scores, using a chance-corrected statistic such as Cohen’s kappa or Fleiss’ kappa. If agreement is low on a dimension, the criterion is ambiguous and needs rewriting, because a rubric that produces different scores from different qualified graders is not measuring anything stable.
How many experts should be involved in rubric design?
Design usually works best with a small core of senior practitioners who draft the dimensions and anchors, then a wider group who stress-test the draft by grading sample outputs and surfacing disagreement. The core keeps the standard coherent, while the wider group exposes ambiguity the core would miss, and both are needed before the rubric is handed to graders at scale.
How does CleverX support designing evaluation rubrics with experts?
CleverX is an on-demand platform that connects AI teams with verified domain experts across fields and geographies, with more than 8 million verified professionals across 150-plus countries. Teams can recruit specialists to define scoring dimensions, write anchored examples, and calibrate the rubric until qualified graders agree, with pay-as-you-go access, delivery in roughly two to five days, and structured sessions at scale through AI Interview Agents.