Hack Apertus 2026 · Track 1B Swiss Voices

The law is the answer key.

Inkraft asks a language model short questions on Swiss social and health insurance law, in German, French and Italian, as of 1 October 2026. Every gold answer rests on a verbatim quote from the federal statute. Every wrong answer is sorted: was it the rule that applied before an amendment, or does it match no stored version?

One question, two amounts

Swiss social insurance law changes on fixed dates. A model trained on last year's web answers with last year's rule, confidently. For a municipal AHV office, a medical practice or a social service, that is worse than “I don't know”.

“What is the minimum full AHV old-age pension per month?”
In force CHF 1,260 since 1 January 2025
Stale CHF 1,225 until 31 December 2024

Art. 34 para. 5 AHVG (SR 831.10), consolidated versions of 1 January 2024 and 1 January 2025 on fedlex.

  • AHV branch offices in the municipalities, often run by generalists
  • Medical practice staff answering insurance and billing questions
  • Social services and the insured themselves

How Inkraft measures

Gold from the statute

Every answer is anchored to one paragraph of federal law: fedlex ELI with version date and the verbatim quote. The answer stands in that quote or follows from it by a recorded calculation.

Three authentic languages

German, French and Italian versions of federal acts are equally binding. The French and Italian gold answers are taken from those texts, not translated.

An as-of date

For 52 provisions amended since 2024 the item also stores the earlier rule, quoted from the earlier fedlex version. A wrong answer that matches it counts as stale, not as another error.

Credit for “I don't know”

SimpleQA-style grading: correct, incorrect or not attempted. A penalised score rewards abstaining over guessing.

Four outcomes per answer

  • Correct matches the law in force on 1 October 2026
  • Stale matches the rule before an amendment
  • Other error matches neither the current nor a stored earlier answer
  • Not attempted the model declines or refers to an office

Grading you can check

  1. Rules first. Numbers, units and refusals are extracted in all three languages.
  2. Then a panel of 3 open-weight judges from other model families than Apertus (Qwen3-235B-A22B-2507 (Alibaba, open weights), GLM-5.3 (Z.ai, open weights), DeepSeek V3.2 (DeepSeek, open weights)); each graded 61 constructed answers correctly before use (error subtype 57 to 60 of 61), and the majority counts.
  3. Then a human grades a blind 22 % sample of the 948 judgements of the 70B main run: Cohen's κ 0.91 against the first judge, 0.92 against the panel majority.
  4. Then make run replays the scoring. It recomputes every result block from the stored answers and grades and checks them against the expected results.

Status

Dataset release v1.0.0, evaluated with Apertus 1.5 8B and 70B.

Done

Evaluation harness

Package with command line, Docker make run that verifies its own output, and unit tests.

Done

Dataset

326 German core questions, 948 question–answer rows in German, French and Italian, from 19 federal acts and ordinances. 52 questions store the previous rule word for word from the earlier fedlex version. All cores are LLM drafts (98 GPT-5.5 via Codex, 97 Claude, 86 Gemini, 45 the open model Nemotron 3 Ultra). Every row passes the check against fedlex, and every core was checked by a second model (15 flagged: 8 dropped, 7 calculation cores kept because the validator recomputes them); a human checks a random sample of 35 cores and removed 1.

Done

Exposure check against Apertus training data

The Swiss legislation in MultiLegalPile is part of the Apertus 1.5 training mix. Of the 948 rows, the corpus holds the current wording for 564, only the old wording for 81 and neither for 277 (26 evidences are too short to search). Of the 132 rows of temporal questions it holds the current wording for 1 and the old one for 73.

Done

Results for Apertus 1.5 8B and 70B

Graded by the judge panel: the 70B answers 50.8 % of the rows correctly, the 8B 33.1 %. On the 154 rows whose rule changed and whose old answer is stored, the 70B is right in 7.1 % and gives the old rule in 30.5 %. With the paragraph in the prompt it is right in 99.4 % of the same rows. Adding the as-of date to the prompt does not change accuracy significantly (−0.3 points, p = 1.0; German rows, graded by the first judge).

Demo video

A two-minute walkthrough of Inkraft.

Inkraft is one of four tools built by Team Apertool on Apertus, Switzerland's fully open language model. Meet the team →

Contact

Questions, ideas or feedback on Inkraft? Write to Team Apertool and we will answer by e-mail.

Your name, e-mail address and message are forwarded by e-mail to Team Apertool, solely to answer you. The form does not store them and does not pass them on. This site uses no analytics and sets no cookies. Fonts are served from this site.