Hack Apertus 2026 · Track 1B Swiss Voices
The law is the answer key.
A trilingual benchmark (DE · FR · IT) for Swiss social and health insurance law. Every gold answer rests on evidence quoted word for word from fedlex, valid on 1 October 2026.
The problem
Correct until 31 Dec 2024.
Since 1 Jan 2025 · Art. 34 para. 5 AHVG
Social insurance law changes on fixed dates.
A model doesn't
know which version it learned.
The insight
«Der Mindestbetrag der vollen Altersrente von 1260 Franken entspricht dem Rentenindex von 229,1 Punkten.»
«Le montant minimal de la rente de vieillesse complète de 1260 francs correspond à un indice des rentes de 229,1 points.»
«L’importo minimo della rendita completa di vecchiaia di 1260 franchi corrisponde a un indice delle rendite di 229,1 punti.»
«Bei Erlassen sind die drei Fassungen in gleicher Weise verbindlich.»
All three language versions of an act are equally binding. So each language gets its gold answer from its own authentic text, never from a translation.
What Inkraft measures
The rule in force on the as‑of date, 1 Oct 2026.
Was the law until the last amendment. Inkraft stores the previous rule and its fedlex version.
Example: an amount in no stored version of the law. Often a hallucination.
Honest. Scored apart from wrong answers, as in SimpleQA.
A stale answer was once right. Inkraft tells it apart.
The dataset
The finding
On the 154 rows whose rule changed since 2024 and whose earlier answer is stored, the 70B states the earlier law in 30 % of answers and the rule in force in only 7 %.
Evaluation
Amounts, deadlines and dates are extracted and matched against gold and the stale answer.
Qwen3-235B-A22B-2507 · GLM-5.3 · DeepSeek V3.2, each calibrated on 61 cases. Grades: correct · stale · near miss · other · not attempted.
Graded without seeing the judges. Cohen's κ 0.91 vs the first judge, 0.92 vs the panel majority.
| Apertus 1.5 | Correct DE · FR · IT | Rule changed, old answer stored: correct · old rule given | Not attempted (all rows) | Effect of the date in the prompt |
|---|---|---|---|---|
| 8B | 34 · 34 · 31 % | 8 / 154 · 22 / 154 | 11.5 % | −1.5 pts (p = 0.27) |
| 70B | 53 · 50 · 49 % | 11 / 154 · 47 / 154 | 1.2 % | −0.3 pts (p = 1.0) |
Release v1.0.0, 948 rows, majority of 3 judges.
Overall correct: 8B 33.1 %, 70B 50.8 %. Open book (law paragraph in the prompt):
70B right in 99.4 % of the 154 stale-pair rows. Date effect on the 326 German rows, graded by the first judge.
Contamination check
Apertus publishes its training catalogue. For v1.5 it lists Swiss legislation:
Multi_Legal_Pile · switzerland_lexfind
For every item we look up the cited paragraph in that corpus and check which version of the rule it contains.
Rows: 564 + 81 + 277 in the three buckets, 26 with evidence too short to search (948 in total).
Leak check: 55 of 948 questions share a 13-gram with the corpus (statutory phrasing; the questions are new).
We can now relate accuracy to the exposure of the wording in the training corpus.
Reproducibility
Inkraft · Track 1B Swiss Voices
Our ask: run Inkraft before Apertus answers citizens on AHV or KVG, and refresh it every 1 January, when the amounts change.