Why reviewers disagree about 'alongside'

In the first two parts of this series we found that reviewers agree about half the time, and that half of all threads turn on who had space. This part asks which topics make agreement worse. If we know where disagreement comes from, we know what a rulebook should pin down.

How we measured it

For each thread we measured how often two random verdicts match. Separately, we measured how much of the rest of the conversation, the comments that don't state a verdict, was about each topic: overlap, track limits, intent, multiclass etiquette, the quality of the clip, and several others. Keeping the two apart avoids a trap, which is that the words used to detect a verdict are also words about the topic.

Then we asked: when a topic is a bigger share of the conversation, does agreement go down? We controlled for how many comments the thread had, which sim it was, and the year.

The result: two topics clearly hurt agreement

Dot chart of the change in agreement per topic. Overlap and the clip itself reduce agreement by about 2 percentage points; the other topics are inconclusive Change in agreement (percentage points) when a topic is a larger share of a thread, with 95% ranges. Red topics remain clearly below zero after correcting for the number of comparisons.

"Alongside", overlap and right of way. In the 35% of threads where this topic makes up at least 15% of the non-verdict comments, two reviewers agree 48% of the time, against 53% elsewhere. That's a 4-point drop, and the clearest single signal in the data.

The clip itself. Threads where commenters talk about the camera, the lag, missing context or "I can't tell from this" are also harder to agree on, by a similar or slightly larger amount. When the evidence is poor, reviewers fill the gap with their own assumptions.

Weaker signs, which could be noise:

  • Track limits and rejoining (about 1.4 points lower per standard deviation) and intent (about 1.1 lower) point the same way but don't hold up once we account for testing ten topics.
  • Multiclass etiquette makes no difference. People may disagree about who should have yielded, but not more often than elsewhere.
  • Quoting the rulebook doesn't improve agreement either, though only 87 threads leaned on it heavily, so we can't say much.

How much do topics explain?

Very little. Everything in our model together (all ten topics, plus the sim, the year and the size of the thread) explains under 3% of the variation in agreement from thread to thread. Whether a thread produces consensus depends mostly on the specific clip, not on which rule is involved. That is an uncomfortable finding if you were hoping one clarified rule would fix everything. It won't, but it can still move a lot of cases at the edges.

The direction of each argument

When comments bring up a dive-bomb, 45% of the verdicts blame the poster, against 36% otherwise. Comments about defending or blocking lean the other way: 31% blame the poster, against 37%. Commenters treat "I went for the gap" as the riskier act, and treat "I held my line" as the safer one. "Alongside" leans slightly towards blame as well (40% against 36%), possibly because most disputes about it are the attacker claiming room.

What a league can write down

You can't remove disagreement, but you can stop it from being avoidable.

  • Give "alongside" a number, and a moment. Say what counts as alongside (for example, the attacker's front wheels level with the defender's rear wheels) and when it's measured (at turn-in, not at the point of contact). Most arguments are about different moments, not different rules.
  • Set an evidence standard. Decide how many angles you need before a penalty stands, and what happens when the clip can't show the key moment. A rule like "we can only penalise what a neutral view shows" prevents a lot of guessing. See more angles, better calls.
  • Say what the system can't know. Lag, a clip that starts late, and a camera that hides the second car all belong in a "can't determine" box, not in a verdict. Our post on why the AI steward says it doesn't know covers the same idea.

A small note about us

Sim Racing Steward builds the evidence standard in. Each clip is read for what its camera can show, the report lists what couldn't be determined, and the rules it cites come from the ruleset you wrote, including your own definition of "alongside". Sign up now, put your own definition in a ruleset, and see how it reads a real incident.

Data: 6,563 first-person threads from r/Simracingstewards; 2,694 with at least three explicit verdicts. Topics are measured from keywords. These are associations in public threads, not proof of cause.