AI in practice

Choosing legal AI for an Egyptian firm

Eight questions to ask before you buy.

[date] · [author] · 7 min read

Every demo of a legal AI tool looks convincing. The vendor types a question and a well organised answer appears in seconds, in confident language. What you are watching is fluency, and fluency is the cheapest thing these tools have. What you are actually buying is accuracy, and accuracy only shows when you ask the right questions.

Why the demo is not enough

General language models are trained to predict the next word, then tuned toward the answers people prefer. The result is a model that writes with the same confidence when it is right and when it is wrong. OpenAI's researchers showed in 2025 that common ways of scoring models reward a guess more than an admission of not knowing. So do not judge a tool by its style. Judge it by what it can show you.

The eight questions

  1. Where do the answers come from? Can you open the source of every answer, the article or the ruling, in full?
  2. Is Egyptian law in it as text? Which laws, which courts, which years? How long before a new law is added?
  3. Does it show the date of the text it quotes, and can it tell the text in force today from the text in force on the date of the facts?
  4. How does it handle Arabic: different spellings of the same word, old Official Gazette texts, questions written in Egyptian Arabic?
  5. What does it do when it cannot find an answer? In the demo, ask something you know its sources do not cover, and see whether it says so or invents an answer.
  6. What happens to your documents? Are they used to train any model? How long are they kept, who can reach them, and how are they deleted?
  7. Where is the data processed, and what does the data protection law require of you? Moving data out of Egypt needs a licence, a permit or another legal ground, and the firm answers for whom it chooses to process its clients' data. Ask for a written data processing agreement.
  8. How is it tested, by whom, and are the results published? Is it tested on real Egyptian questions with known answers, and are its weak spots published as well as its strengths?

A test to run in the demo

  • Three questions from your old files whose answers you know well.
  • One question about a recent change in the law, such as the new Labour Law.
  • One question Egyptian law does not answer.
  • One question written in Egyptian Arabic.

Then open every source and read it.

Why a general model gets this wrong

Question 5 matters most. A model that is not built on defined sources has no way to know the limits of what it knows. To the model, every question is text to be continued, and every continuation has a most likely wording, so it always answers. It says "I don't know" only if it was trained to and given a source to measure its knowledge against. A good legal tool starts from its sources, and when it finds nothing there, it says so.

In Bayan

We answer the eight questions in public. Every answer in Bayan shows its source for you to open, and Bayan does not claim what its sources do not hold. Bayan Index lists what the library holds [Sherif confirms]. Your clients' documents never train any model and are deleted on request, and we sign a data processing agreement with teams and organisations. Bayan Bench publishes our test results, Bayan's included. Book a demo for your team and ask us all eight.

Sources. Kalai et al., "Why Language Models Hallucinate", OpenAI, September 2025.

Share

The month's key legal updates and what's new in Bayan, once a month.

Ask Bayan your first question.

Or book a team demo and bring a real contract: thirty minutes is enough to know whether Bayan fits your practice.

Book a team demo