Hack Apertus 2026 · Track 1B Swiss Voices

Inkraft

The law is the answer key.

A trilingual benchmark (DE · FR · IT) for Swiss social and health insurance law. Every gold answer rests on evidence quoted word for word from fedlex, valid on 1 October 2026.

Team Apertool · solo builder · Core Task Intelligence

The problem

Same question. Two confident answers.

ASKED TO AN AI ASSISTANT How much is the minimum full AHV old-age pension per month?
PREVIOUS RULE
CHF 1,225

Correct until 31 Dec 2024.

IN FORCE
CHF 1,260

Since 1 Jan 2025 · Art. 34 para. 5 AHVG

Social insurance law changes on fixed dates.
A model doesn't know which version it learned.

Who asks

  • ResidentsLiving in Switzerland means compulsory KVG and AHV cover.
  • Medical practice assistantsFranchise, billing, deadlines at the front desk.
  • Municipal AHV branch officesGeneralists who answer pension and benefit questions.

The insight

The answer is already written down. In three languages.

DE AHVG Art. 34 Abs. 5
«Der Mindestbetrag der vollen Altersrente von 1260 Franken entspricht dem Rentenindex von 229,1 Punkten.»
FR LAVS art. 34 al. 5
«Le montant minimal de la rente de vieillesse complète de 1260 francs correspond à un indice des rentes de 229,1 points.»
IT LAVS art. 34 cpv. 5
«L’importo minimo della rendita completa di vecchiaia di 1260 franchi corrisponde a un indice delle rendite di 229,1 punti.»
Art. 14 para. 1
PublG
«Bei Erlassen sind die drei Fassungen in gleicher Weise verbindlich.»

All three language versions of an act are equally binding. So each language gets its gold answer from its own authentic text, never from a translation.

What Inkraft measures

Not every wrong answer is a hallucination.

CORRECT
CHF 1,260

The rule in force on the as‑of date, 1 Oct 2026.

gold, quoted from fedlex
STALE
CHF 1,225

Was the law until the last amendment. Inkraft stores the previous rule and its fedlex version.

outdated_answers
OTHER WRONG
CHF 1,300

Example: an amount in no stored version of the law. Often a hallucination.

any other answer
NOT ATTEMPTED
“I'm not sure.”

Honest. Scored apart from wrong answers, as in SimpleQA.

abstention

A stale answer was once right. Inkraft tells it apart.

The dataset

Built from the statute book.

326
core questions in German
948
rows across DE · FR · IT
311
cores in all three languages
52
cores with the previous rule stored as stale answer

Core questions per federal law

KVG54
AHVG49
KVV33
UVG29
EOG28
ATSG27
IVG25
BVG19
OR17
ELG13
AVIG10
8 more22

Cited sentence last amended

248
70
8
before 2024
in 2024 or 2025
in 2026
948 / 948 rows pass the source check: the evidence appears word for word in the cited fedlex paragraph, in the row's own language.
v1.0.0 · LLM drafts (Codex 98, Claude 97, Gemini 86, Nemotron 45), checked by validator, second model and a human (35 cores, 1 removed).

The finding

30 %

When a rule changed, Apertus 70B gives the old rule.

On the 154 rows whose rule changed since 2024 and whose earlier answer is stored, the 70B states the earlier law in 30 % of answers and the rule in force in only 7 %.

7 %correct, rule changed, no paragraph given
99 %correct with the cited paragraph in the prompt
−0.3 ptseffect of today's date in the prompt, German rows (p = 1.0)
A knowledge gap, not a reading gap. Main results: majority of 3 open-weight judges; date ablation (German rows): first judge only. Release v1.0.0.

Evaluation

Graded like SimpleQA. Checked by a human.

1 · RULES

Numbers and dates first

Amounts, deadlines and dates are extracted and matched against gold and the stale answer.

2 · JUDGE PANEL

3 open-weight judges, majority

Qwen3-235B-A22B-2507 · GLM-5.3 · DeepSeek V3.2, each calibrated on 61 cases. Grades: correct · stale · near miss · other · not attempted.

3 · HUMAN

22 % blind sample

Graded without seeing the judges. Cohen's κ 0.91 vs the first judge, 0.92 vs the panel majority.

Apertus 1.5 Correct DE · FR · IT Rule changed, old answer stored: correct · old rule given Not attempted (all rows) Effect of the date in the prompt
8B 34 · 34 · 31 % 8 / 154 · 22 / 154 11.5 % −1.5 pts (p = 0.27)
70B 53 · 50 · 49 % 11 / 154 · 47 / 154 1.2 % −0.3 pts (p = 1.0)

Release v1.0.0, 948 rows, majority of 3 judges.
Overall correct: 8B 33.1 %, 70B 50.8 %. Open book (law paragraph in the prompt): 70B right in 99.4 % of the 154 stale-pair rows. Date effect on the 326 German rows, graded by the first judge.

Contamination check

Did Apertus see this law? We check its legal training corpus.

Apertus publishes its training catalogue. For v1.5 it lists Swiss legislation:

Multi_Legal_Pile · switzerland_lexfind

For every item we look up the cited paragraph in that corpus and check which version of the rule it contains.

Rows: 564 + 81 + 277 in the three buckets, 26 with evidence too short to search (948 in total).

Leak check: 55 of 948 questions share a 13-gram with the corpus (statutory phrasing; the questions are new).

Current rule in that corpusseen and still valid · 564 rows
70B correct64 %
Previous rule in that corpusseen, then amended (not the same set as the stale-pair rows) · 81 rows
70B correct11 %
Not in that corpusnot found there · 277 rows
70B correct38 %

We can now relate accuracy to the exposure of the wording in the training corpus.

Reproducibility

One command. Scoring replayed.

$ make run # Docker, no keys, no GPU
recomputes every result block from 5,096 stored answers and grades
PASS: all 8 result blocks match expected_results.json
$ make test # unit tests, same image
$ make run MODE=full
# any OpenAI-compatible model: LLM_BASE_URL, LLM_API_KEY, LLM_NAME
test.jsonl questions and gold
metadata.jsonl fedlex ELI with version date
responses/ every model answer · 5,096
judgements/ every grade · 3 judges + majority
human_labels/ 22 % sample · κ
No personal data: vignettes use roles, the validator scans for PII. Consent form for reviewers, data card (Datasheets, Data Cards).
Licences: data CDLA-Permissive-2.0 · code Apache-2.0 · docs CC-BY-4.0 · statute text public domain (Art. 5 URG)
Related: SwitzerlandQA, Swiss-Bench and LEXam test Swiss law and civics; Prior et al. (2026) test temporal knowledge of German law. Inkraft adds stored earlier answers, scored abstention and DE · FR · IT.

Inkraft · Track 1B Swiss Voices

The law is the answer key.

Our ask: run Inkraft before Apertus answers citizens on AHV or KVG, and refresh it every 1 January, when the amounts change.

Team Apertool
Niklas Wichter · solo builder
inkraft.apertool.ch
all slides