How we build Bayan

How we grade Arabic legal answers in Bayan Bench

Five criteria, and a confident error scores below "I don't know".

[date] · [author] · 6 min read

Bayan Bench is a public test of how AI models answer questions of Egyptian law. We write the questions, collect the models' answers, grade them and publish the results, Bayan's included. This article explains how we grade, and why we designed the grading this way.

The questions

Practising Egyptian lawyers write the questions from real issues with known answers, and write a model answer with its sources alongside each one. [Sherif confirms the number of questions and their areas.]

The five criteria

  • Conclusion: is the answer right on the question actually asked?
  • Authority: does the cited article exist, in the right law, in force on the relevant date, and does it say what the answer claims? Does the cited ruling exist, is it cited correctly, and is its principle what the answer says?
  • Completeness: does the answer state the conditions and exceptions? An answer on a penalty clause that does not say the debtor carries the burden of proof is incomplete.
  • Language: does it use the right legal term, rescission and not dissolution, in sound formal Arabic rather than literal translation?
  • Calibration: does the answer say the sources are not enough when they are not?

A confident error scores below abstaining

This is the heart of the design. Most common tests grade an answer right or wrong, so "I don't know" earns nothing and a guess sometimes earns points, and the model learns that guessing always pays. OpenAI's researchers showed in 2025 that this kind of scoring is one of the causes of hallucination itself, and proposed penalising confident errors more than expressions of uncertainty. That is what we do: a confident wrong answer scores lower than an answer that says clearly the sources are not enough. [Sherif confirms the weights.]

How we grade

[Sherif confirms before publication.]

  • Two lawyers grade each answer independently; if they disagree, a third decides.
  • Graders do not know which model wrote the answer.
  • Every citation is checked against the text it claims.
  • We publish a sample of questions with their answers and keep the rest unpublished, because published questions can end up in the training data of future models and stop measuring anything.

Why general models struggle on this test

Everything this series has explained shows up here at once. A general model predicts the most likely word, so it is weakest where text is rare, and Egyptian law in Arabic is rare in its training text. It is tuned toward answers people like, so it drops qualifications. Its knowledge stops on a date, so it misses what is new. And common tests reward it for guessing, so it does not say "I don't know". A benchmark that rewards fluency measures fluency, so we built one that rewards accuracy.

Sources. Kalai et al., "Why Language Models Hallucinate", OpenAI, September 2025.

Share

The month's key legal updates and what's new in Bayan, once a month.

Ask Bayan your first question.

Or book a team demo and bring a real contract: thirty minutes is enough to know whether Bayan fits your practice.

Book a team demo