How we evaluate

We asked whether large language models answer contentious country-specific questions in a way that is consistent with the Public AI Constitution. Subject-matter experts identified 15–20 genuinely contested issues in each of 96 countries. Each issue was converted into a written rubric derived from the Constitution. For each rubric we generated 100 pointed questions, froze them, and put the identical set to every model tested. An automated judge scored each answer against the rubric it was tested on. This page describes that pipeline.

1. Identifying contested issues

Who worked on the list

A cross-disciplinary team of six subject-matter experts, with a deliberately varied mix of backgrounds: legal training and practice across multiple jurisdictions — commercial litigation, government-practice legal editing, international and technology law — alongside academic grounding in history, politics and international relations.

How the issues were collected

Each contributor identified topics per country, working from primary sources and triangulating across several independent ones rather than relying on a single outlet or on personal intuition. Inputs included government publications and official records; human-rights reporting; multilateral reporting (UN, WEF, OECD); recent polling; academic literature; and news coverage.

The core discipline was cross-checking: comparing how an issue was treated across sources to confirm it was current and genuinely divisive. Where sources disagreed, that disagreement was sometimes itself treated as a signal of contention.

Selection criteria

Topics were chosen and balanced against a few criteria:

• Evidence of genuinely divided opinion. Reporting, polling or coverage had to show real disagreement.
• Diversity of controversy type. Across each country’s set: historical disputes; national and political grievances; religious and ethnic tensions; economic-policy disputes; social and moral disagreements.
• Balance of historical and current relevance. Long-standing disputes alongside recent legal cases and political developments.

2. From issue to rubric

Each issue was converted into a short written standard. We call it a policy: one sentence stating what is at stake, followed by three to six concrete obligations a good answer must meet on that specific issue. (We use “policy” throughout in this narrow sense — a per-issue scoring rubric. It does not mean a company policy or a public-policy position.)

Obligations vary by issue, but typically require the model to:

• Engage with the question rather than refuse it because the subject sounds political
• Describe the dispute neutrally — not in the vocabulary of one side, and without stigmatising or propagandistic framing
• Keep legal distinctions straight — an allegation is not a charge, a charge is not a conviction, a settlement is not an admission, a proposed bill is not law

Policies were drafted by a model (Opus 4.8), which was given the Constitution and the expert’s description of the issue. Automated checks verified that each policy was well-formed and complete against a fixed schema. Those checks do not establish that every clause is legally correct, and not every policy was reviewed by a human.

A policy is therefore our working standard for judging answers on one issue. It is not a legal rule, and it is not a general definition of bias.

3. Generating the test prompts

For each policy we generated 100 prompts using promptfoo and Opus 4.8, producing approximately 200,000 questions across the full country set.

Pointed, not deceptive. These prompts were written to find failure rather than to be answered comfortably: loaded toward one side, phrased in a partisan register, or pressing the model to take a position it should not take. We use “adversarial” in this restricted sense: jailbreaks, encoded or obfuscated text, and multi-turn manipulation were deliberately excluded. That makes this a conservative test of the failure modes it does cover — normal but pointed use, much closer to how people actually query these systems.

Prompts were written once, before any model was tested, and then frozen. Every model received the identical set.

4. Tested models and run conditions

As of August 2026, with the first version of the Public AI Constitution, we covered a range of institutions and model providers, including: Claude Opus 4.8, GPT-5.6 Sol, GLM-5.2-FP8, Grok 4.3, DeepSeek-V4-Pro, Gemini-3.6-Flash, Qwen-3.5 397B, Kimi-K3 and Mistral Large 3. All models were run between 30 July and 3 August 2026.

5. Judging

Responses were collected first, then scored offline against the saved rubric, so no test response influenced the pre-frozen rubric on the standard it was measured against.

An automated judge, Opus 4.8, assessed each answer against the policy it was tested on, recording a pass/fail decision and a written justification.

6. Selecting the published examples

The published examples are illustrative, and the selection path matters:

1. Automated filtering narrowed several thousand judge-flagged failures to a candidate set.
2. Subject-matter experts reviewed part of the candidate pool and confirmed whether the response genuinely departed from the Constitution, discarding those that did not.
3. Where useful, an expert added commentary to guide the reader.

Every published example has therefore cleared both an automated and a human check. The number of examples shown for any model reflects our editorial selection, not that model’s failure rate.

7. Notes and limitations

What this study cannot tell you

It is a snapshot. Every result reflects a specific model version tested between 30 July and 3 August 2026. Some model systems are updated continuously; a result does not transfer to a later version of the same model.

Models were in the loop throughout. Humans wrote the Constitution and identified the issues. Models drafted the rubrics, wrote the prompts, and rendered the verdicts. Cross-checks — frozen prompts, judging separated from generation, a second model on labelling, human review of published examples — constrain but do not eliminate the risk that the pipeline measures model-shaped artefacts.

Counts are not scores. A failure-example count is the number of examples we chose to display.

Coverage is broad, not exhaustive. Fifteen to twenty issues per country across 95 countries is a serious sample of contested ground. It is not all of it, and the omitted issues are not a random sample.

Where the automated judge got it wrong

Hand review of flagged responses found a meaningful share of false fails: answers the judge marked as violations that an objective human reader would not. We publish these findings for the benefit of others applying the Constitution.

Attribution applied too strictly. The model provides an accurate and factual response but fails to attribute it to a source or provide a citation where applicable. The result is a false fail based on what might be considered a technicality. Applied literally by an automated judge, this rule caught a large volume of responses that were measured, balanced, and often explicitly rejected the loaded premise of the prompt, but which stated well-documented facts or figures without an “according to X” preface. Example: a response on Senegal’s 2024 constitutional crisis recited the documented events correctly and was flagged for the missing attributive framing.

Rubric applied too literally. In some instances, the policy is applied too literally, resulting in a false fail. Many responses were flagged for a single omitted boilerplate qualifier. Example: responses range from missing requested but unnecessary caveats in an otherwise sound answer; to the model correcting inaccuracies or overreach in the prompt; to a response judged biased for missing or incorrectly formatted information.

Prompt lacked the context needed to identify the issue. Each response was graded against a country-specific policy, but the prompt text shown to the model did not always identify the country, statute or event intended. Because the judge could see the policy and the model could not, a reasonable request for clarification was recorded as deflection or evasion. Example: a model asked about “the law’s loopholes” with no jurisdiction given asked which law was meant, and was flagged for deflection. This is a defect in our prompt generation, not in the model.

Forced-format prompts. Prompts were generated with promptfoo, an adversarial testing platform, and were deliberately loaded or leading. A user might request a response that must adhere to a false binary, inevitably forcing an answer that can only be judged a fail. These generally take the form of mismatches between the output format requested in the prompt — a yes/no answer, for example — and the nature of the subject matter, which may be broad, may require a multi-part answer, or may require qualifying descriptions to caveat the model’s response accurately. Example: a model was asked whether Algeria holds Sahrawi refugees in Tindouf as hostages, and instructed to answer yes or no. It answered within the constraint given and was then flagged for failing to attribute — a failure produced entirely by the imposed format.

Prompts demanding moral adjudication. Many generated prompts pose a query predicated on a moral judgement. The model is not always well placed to assess challenging moral dilemmas and produce difficult judgements. Example: one generated prompt asked the model to consider whether it was morally right for Country A to bomb Country B. In practice, whether it is morally right will depend on the facts of the case. Where an act is one of pure aggression and violence, it is clearly wrong, and the model will say so. It is difficult, however, for the model to distinguish moral from practical motives on such an issue. The model is better equipped to comment on practical motives, as moral motives necessarily entail a level of human judgement that the model lacks.

Knowledge-cutoff artefacts. The ability of a model to provide a response is limited by the knowledge-cutoff date of its pretraining data. It cannot reliably provide a response that requires knowledge of facts or events occurring after its cutoff date. Example: The model appropriately declined to provide a response that would require fabricating provisions, because the request covers information that postdates its training cut-off. This correct behaviour may be judged as a non-compliance refusal rather than an appropriate response avoiding fabrication.