How to build an RL environment for a real domain
The practical steps to build an RL environment that trains agents on real work, and why expert judgment, not the simulation, sets the quality ceiling.
To build an RL environment for a real domain, you scope a specific task, build a sandbox that reproduces the tools and data behind it, get domain experts to perform the task so you have gold demonstrations, define a reward rubric that captures what correct means, grade agent attempts against it, and iterate. The engineering builds the sandbox, but the expert judgment inside it, the demonstrations, the rubric, and the grading, is what decides whether the environment actually trains a useful agent.
This is a practical guide to each step, aimed at teams who want to train or evaluate an agent on a real job rather than a game. The recurring theme is that the simulation is the easy part, and the expert work is the part that sets the ceiling on quality.
The build at a glance
| Step | What you do | Who leads it |
|---|---|---|
| 1. Scope the task | Pick a specific, bounded job the agent should do | Product plus domain expert |
| 2. Build the sandbox | Reproduce the tools, data, and systems of the task | Engineering |
| 3. Capture demonstrations | Have experts perform the task to create gold trajectories | Domain experts |
| 4. Define the reward rubric | Encode what correct outcome and process look like | Domain experts |
| 5. Grade agent runs | Score agent attempts against the rubric | Domain experts plus automated checks |
| 6. Iterate | Fix gaps where reward and real quality disagree | Whole team |
Step 1: Scope the task tightly
The most common mistake is scoping too broadly. “Build an agent that does accounting” is not a task, it is a department. A good environment targets a specific, bounded job with a clear beginning and end: reconcile a bank statement against the ledger, review a commercial lease for a defined set of risk clauses, triage an inbound insurance claim, or resolve a billing dispute within policy.
A well-scoped task has three properties. It has a definite end state so you can tell when it is finished. It has a notion of correct that a professional would recognize. And it is representative of real work, not a toy version. Spend real time here with someone who does the job, because a task scoped by someone who does not do the work tends to miss the judgment calls that make the job hard. For how this connects to the full environment loop, see what are RL environments.
Step 2: Build the sandbox and tools
Now reproduce the world the agent will act in. This is the engineering core of the environment: the tools the agent can call, the data it reads and writes, and the systems it interacts with, wrapped in the standard interface that lets an agent reset the task and step through it one action at a time. The widely used interface for this is Gymnasium, the maintained successor to OpenAI Gym.
Good public examples show the range. WebArena builds a realistic set of websites an agent must operate. SWE-bench wraps real code repositories so an agent can attempt real bug fixes. tau-bench reproduces customer-service workflows with tools and policy rules. For a real domain you are doing the same thing: standing up a replica of the role or company, the “fake company” approach, so the agent can practice the actual job.
The sandbox has to be realistic enough that success inside it means something outside it. If the simulated tools behave differently from the real ones, or the data is cleaner than reality, the agent will learn to handle a world that does not exist. But note what the sandbox cannot do on its own: it can present the task and record actions, but it cannot say whether the work was done well. That judgment has to come from people.
Step 3: Capture expert demonstrations
Before an agent can learn the job, you need examples of the job done right. This is where domain experts perform the task inside the sandbox and you record the full trajectory: the sequence of states, actions, and decisions that make up correct work.
These gold demonstrations do two things. They give the agent concrete examples to learn from, and they reveal the judgment calls that no manual captures, the moments where a professional deviates from the obvious step because experience tells them to. A demonstration is far richer than a label: it is an entire expert decision process, start to finish. We go deep on getting this right in collecting expert demonstrations for AI at quality.
The quality of the demonstrator matters enormously. A demonstration from someone who does not truly do the job will encode shortcuts and errors that the agent will faithfully reproduce. This is why demonstrations for regulated or technical work have to come from verified practitioners, which we cover in domain experts for AI training and evaluation.
Step 4: Define the reward rubric
The reward is the heart of the environment, and it is the thing most likely to go wrong. A reward signal tells the agent what to optimize for, and the agent will optimize for exactly what you reward, not what you intended. If you reward closing a support ticket quickly, you may get an agent that closes tickets fast and resolves nothing.
For real work, a single pass or fail reward is almost never enough. You need a rubric: a structured set of criteria that scores both the outcome and the process. A good rubric usually captures several things:
- Outcome. Did the agent reach the correct end result?
- Process. Did it follow the required steps, especially the non-negotiable ones?
- Compliance. Did it respect the rules, policies, and constraints of the domain?
- Cost of error. Did it avoid the mistakes that are expensive or dangerous, weighted by how serious they are?
Only an expert can write this well, because only an expert knows which steps are mandatory, which errors are catastrophic versus cosmetic, and where a plausible-looking answer is actually wrong. The rubric is where professional standards get encoded into the environment, and it is the reason expert judgment, not engineering, sets the quality ceiling. The reward-design challenge is a long-standing theme in reinforcement learning, well summarized in the Sutton and Barto textbook, Reinforcement Learning: An Introduction.
Step 5: Grade the agent runs
With a sandbox, demonstrations, and a rubric in place, you let the agent attempt the task and grade what it produces. Some of this grading can be automated, the parts with checkable answers, but the parts that require professional judgment need an expert in the loop, at least to calibrate and audit the automated grader.
Expert grading catches the failure that automated metrics miss: the answer that looks right, reads fluently, and is wrong in a way only a practitioner would notice. An agent that drafts a confident but legally flawed clause, or an underwriting decision that ignores a subtle risk, will pass a shallow check and fail a real one. Catching that is the whole point. We cover this discipline in evaluating AI agents with domain experts, and the sourcing side in data for AI agents.
Grading is also how you detect reward hacking, cases where the agent games the rubric to earn reward without doing the job. When expert review of a high-scoring run disagrees with the score, you have found a hole in the rubric.
Step 6: Iterate until reward matches reality
An RL environment is never right on the first pass. You build it, run agents against it, and watch for the gap between what scores well and what a domain expert considers good work. Every disagreement is a signal: the rubric missed a criterion, the grading was too lenient, the sandbox was unrealistic, or the demonstrations were thin.
You close those gaps and run again. Over several cycles the reward signal converges on real-world quality, and the environment becomes a reliable place to train and measure the agent. The test of a finished environment is simple: agents that score well in it should also do the real job well when an expert reviews their output. If they do not, the environment is not done.
This iteration is also where environments stay fresh as the real job changes. Update the rubric and demonstrations, and the environment keeps teaching the current version of the work rather than a frozen snapshot, an advantage we contrast with fixed data in RL environments vs static datasets.
Why expert judgment sets the ceiling
Step back and look at who touches each stage. Scoping needs a practitioner. Demonstrations are produced by experts. The rubric is written by experts. Grading is done or calibrated by experts. The sandbox, the only purely technical piece, is necessary but not sufficient.
That is the real lesson of building RL environments for real domains: you can engineer a perfect simulation and still train a useless or dangerous agent if the human judgment inside it is weak. A realistic sandbox with a bad rubric just teaches the agent to be confidently wrong. The environment is a vessel. The expertise is the content, and it decides how good the trained agent can ever be. For the wider frontier view, see the hub guide, reinforcement learning environments: how experts train AI agents.
Frequently asked questions
How do you build an RL environment for a real domain?
You scope a specific task, build a sandbox that reproduces the tools and data of that task, get domain experts to perform it to create gold demonstrations, define a reward or rubric that captures what correct means, grade agent attempts against it, and iterate. The engineering builds the sandbox, but expert judgment produces the demonstrations, the rubric, and the grading that make the environment teach the right behavior.
What is the hardest part of building an RL environment?
The hardest part is not the simulation, it is the expert judgment. Defining what a successful outcome looks like, demonstrating how the work is really done, and grading agent attempts all require a practitioner who does the job. A realistic sandbox with a bad rubric simply trains the agent to earn reward for the wrong thing, so expert quality sets the ceiling.
What is a reward rubric in an RL environment?
A reward rubric is the set of criteria that decides how much reward an agent earns for an attempt. Instead of a single pass or fail, it breaks a task into the things that matter: did the agent reach the right outcome, follow the required process, respect the rules, and avoid costly mistakes. Experts write the rubric because they know which steps are non-negotiable and which errors are serious.
Why do you need domain experts to build an RL environment?
An environment can only reward behavior that someone has defined as correct, and for real professional work that definition requires someone who does the job. Experts perform the task to create demonstrations, encode professional standards into the rubric, and judge whether an agent’s plausible-looking answer is actually right. Generalists cannot reliably do this for regulated or technical domains.
What is the difference between a demonstration and a rubric?
A demonstration is an expert performing the task end to end, producing a gold example trajectory the agent can learn from. A rubric is the scoring guide that defines what a good outcome and process look like, which becomes the reward signal. Demonstrations show how to do the work, while the rubric decides how well any given attempt did it.
How do you know if an RL environment is working?
You check whether agents that score well in the environment also do the real job well when a domain expert reviews their output. If high-reward agents still make mistakes an expert catches, the rubric or grading is wrong and needs revision. The environment is working when its reward signal lines up with expert judgment of real-world quality.
Train your AI with verified experts
CleverX does not build environments. It supplies the verified domain experts that every step above depends on: real practising professionals who scope the task, perform it to create gold demonstrations, define the rubric, build benchmarks from real work, and grade agent runs. Experts are verified through government ID, professional licence confirmation, LinkedIn experience matching, and recorded interviews, across finance and accounting, healthcare, law, engineering and manufacturing, insurance, and operations, from a pool of 10M+ verified professionals.