Is the AI steward accurate? Why it watches first and judges second

Someone asked us a fair question recently: is the system actually accurate? There isn't one honest number that answers it, because "accurate" covers several different things. Describing what happened on track, applying your rulebook, and deciding the outcome are three different skills, and a model is much better at some than others.

So the first thing we did was stop asking one model to do everything at once. This post covers how that works and why it matters.

The problem with "watch this and tell me who's at fault"

If you hand a video to an AI and ask for a verdict in one go, you get a fluent answer where the facts, the rules and the opinion are all mixed together. You can't tell which part is wrong. If the model misjudged the gap between the cars, that mistake is quietly baked into the penalty, and nothing on the page shows it.

Human stewards are trained to avoid exactly this. You establish the facts first, then look at the rules, then decide. We built the same order into the system.

Step 1: Establish the facts

The first stage is the only one that actually looks at the footage. Its job is to write down what happened, and its instructions are blunt about what it must not do: no assigning fault, no quoting regulations, no recommending a penalty, no guessing what the driver intended.

The output is a short structured report with fixed headings: which drivers are visible, where on track it happened, a numbered sequence of events, the point of contact, the outcome, any track limits or rejoin issues, and a list of things that can't be determined from the footage. It's kept short on purpose, and it avoids loaded words like "blame" or "divebomb", because those imply a judgement the model hasn't earned yet.

Because the model has to commit to a sequence of events in plain language, you can read it and check it against the clip in a few seconds. If it says the car on the inside had its front axle alongside and you can see it didn't, you've found the problem before any penalty exists.

Step 2: Apply the rules

The second stage never sees the video. It only gets the written fact report and your league's rulebook. Its job is to find which rules are relevant, then list aggravating and mitigating factors, for example "loss of control caused a position change" or "the affected car rejoined safely".

Keeping the video out of this step is deliberate. The model can't re-interpret the footage to fit a rule it likes. Every claim has to trace back to either the fact report or the rulebook, and anything it can't trace is left out.

This also makes your ruleset the source of truth. A model that knows your actual rules will give you a very different result from one reasoning from a vague idea of "racing etiquette". If you want better results, a clear ruleset is the single biggest lever you have. We wrote about how to write one.

Step 3: Decide

The final stage plays the chief steward. It reads the facts and the rule analysis and has to land on exactly one of four outcomes: penalty, warning, no further action, or needs more info. It explains its reasoning, considers alternative interpretations, and only suggests a penalty size when it has actually decided on a penalty.

That fourth option is important. A system that has to pick a penalty every time will invent certainty. One that is allowed to say "I can't tell from this footage, and here's what would settle it" is far more trustworthy, and we cover why in the next post.

Why the split makes it more accurate

Errors are visible. A wrong fact shows up in the fact report, not hidden inside a verdict.
Each stage has one job. Models do better with a narrow task and clear instructions than with a broad one.
Low randomness. These calls run at a low temperature, so the same clip and the same rulebook give very similar answers on a re-run.
Your rules, not ours. The decision is made against the rulebook you supplied, not a generic one.
A human reviews the result. The output is a draft with its reasoning shown. The steward watches the clip and records the final decision.

What this doesn't fix

Being honest about it: splitting the work doesn't give the model better eyes. If a flat video can't show how close two cars really were, the first stage can only report that it's unclear, and that uncertainty has to travel forward. It also can't know about lag or what happened before the clip started. We cover those limits in can AI steward sim racing incidents? and netcode and incidents.

How to check it for yourself

The best test isn't a statistic from us. Take ten incidents your stewards already decided, run them through, and compare. Look at the fact report first: is it right about what happened? Then look at where it disagrees with your panel, and ask which stage caused the difference. That tells you far more than a percentage would.

You can try Sim Racing Steward free and run your own cases.