Gold from the statute
Every answer is anchored to one paragraph of federal law: fedlex ELI with version date and the verbatim quote. The answer stands in that quote or follows from it by a recorded calculation.
Hack Apertus 2026 · Track 1B Swiss Voices
Inkraft asks a language model short questions on Swiss social and health insurance law, in German, French and Italian, as of 1 October 2026. Every gold answer rests on a verbatim quote from the federal statute. Every wrong answer is sorted: was it the rule that applied before an amendment, or does it match no stored version?
Swiss social insurance law changes on fixed dates. A model trained on last year's web answers with last year's rule, confidently. For a municipal AHV office, a medical practice or a social service, that is worse than “I don't know”.
Art. 34 para. 5 AHVG (SR 831.10), consolidated versions of 1 January 2024 and 1 January 2025 on fedlex.
Every answer is anchored to one paragraph of federal law: fedlex ELI with version date and the verbatim quote. The answer stands in that quote or follows from it by a recorded calculation.
German, French and Italian versions of federal acts are equally binding. The French and Italian gold answers are taken from those texts, not translated.
For 52 provisions amended since 2024 the item also stores the earlier rule, quoted from the earlier fedlex version. A wrong answer that matches it counts as stale, not as another error.
SimpleQA-style grading: correct, incorrect or not attempted. A penalised score rewards abstaining over guessing.
make run replays the scoring.
It recomputes every result block from the stored answers and
grades and checks them against the expected results.
Dataset release v1.0.0, evaluated with Apertus 1.5 8B and 70B.
Package with command line, Docker make run that
verifies its own output, and unit tests.
326 German core questions, 948 question–answer rows in German, French and Italian, from 19 federal acts and ordinances. 52 questions store the previous rule word for word from the earlier fedlex version. All cores are LLM drafts (98 GPT-5.5 via Codex, 97 Claude, 86 Gemini, 45 the open model Nemotron 3 Ultra). Every row passes the check against fedlex, and every core was checked by a second model (15 flagged: 8 dropped, 7 calculation cores kept because the validator recomputes them); a human checks a random sample of 35 cores and removed 1.
The Swiss legislation in MultiLegalPile is part of the Apertus 1.5 training mix. Of the 948 rows, the corpus holds the current wording for 564, only the old wording for 81 and neither for 277 (26 evidences are too short to search). Of the 132 rows of temporal questions it holds the current wording for 1 and the old one for 73.
Graded by the judge panel: the 70B answers 50.8 % of the rows correctly, the 8B 33.1 %. On the 154 rows whose rule changed and whose old answer is stored, the 70B is right in 7.1 % and gives the old rule in 30.5 %. With the paragraph in the prompt it is right in 99.4 % of the same rows. Adding the as-of date to the prompt does not change accuracy significantly (−0.3 points, p = 1.0; German rows, graded by the first judge).
A two-minute walkthrough of Inkraft.
Ten slides on the problem, the dataset, the finding and how it is graded.
Open the pitch →A two-minute walkthrough.
Watch the video →Six pages: dataset, judge panel, human check and results for Apertus 1.5 8B and 70B.
Download PDF →
Code and data:
GitHub repository
(data card, make run) ·
dataset on Hugging Face.
Inkraft is one of four tools built by Team Apertool on Apertus, Switzerland's fully open language model. Meet the team →
Questions, ideas or feedback on Inkraft? Write to Team Apertool and we will answer by e-mail.
Your name, e-mail address and message are forwarded by e-mail to Team Apertool, solely to answer you. The form does not store them and does not pass them on. This site uses no analytics and sets no cookies. Fonts are served from this site.