mirror of
https://github.com/data-privacy-stack/presidio.git
synced 2026-07-23 11:20:55 -05:00
* feat(analyzer): add German PII recognizers (DE_*) Adds 9 new predefined recognizers for German personally identifiable information, all disabled by default (enabled: false) and scoped to supported_language: de. Recognizers with checksum validation: - DE_TAX_ID: Steueridentifikationsnummer (§§ 139a-e AO) ISO 7064 Mod 11,10 checksum (Bundeszentralamt für Steuern) - DE_SOCIAL_SECURITY: Rentenversicherungsnummer / RVNR (§ 147 SGB VI) Deutsche Rentenversicherung Bund checksum algorithm - DE_HEALTH_INSURANCE: Krankenversicherungsnummer KVNR (§ 290 SGB V) GKV-Spitzenverband checksum; DSGVO Art. 9 (health data) Pattern-based recognizers: - DE_TAX_NUMBER: Steuernummer (§ 139a AO) – ELSTER 13-digit and state-specific slash-separated formats - DE_PASSPORT: Reisepassnummer (PassG § 4, ICAO Doc 9303) - DE_ID_CARD: Personalausweisnummer (PAuswG) – nPA and legacy format - DE_KFZ: KFZ-Kennzeichen (FZV § 8, ECJ C-582/14) - DE_HANDELSREGISTER: Handelsregisternummer HRA/HRB (§§ 9, 14 HGB) - DE_PLZ: Postleitzahl (DSGVO Art. 4 Nr. 1) – very low base confidence (0.05), context words required for actionable results Also updates: - predefined_recognizers/__init__.py: imports and __all__ - conf/default_recognizers.yaml: all 9 recognizers registered - docs/supported_entities.md: new Germany section - CHANGELOG.md: entry under [unreleased] 133 new tests pass (9 test files, one per recognizer). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: enforce LF line endings for shell scripts in containers Added .gitattributes to force LF for *.sh files. CRLF endings in entrypoint.sh caused 'no such file or directory' when running the presidio-analyzer Docker container on Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(config): enable German recognizers and add 'de' language support Enabled all 9 DE_* recognizers in default_recognizers.yaml and added 'de' to supported_languages so they are loaded and visible to clients (e.g. LiteLLM) without requiring ad-hoc configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(config): add 'de' to analyzer engine supported_languages The registry and engine supported_languages must match. Missing 'de' in default_analyzer.yaml caused a startup crash. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(config): restore default enabled=false and single-language defaults Reverts the deployment-specific changes made for local testing: - German recognizers back to enabled=false (upstream contribution convention) - supported_languages back to [en] only in both config files Users who want German PII detection should enable the DE_* recognizers explicitly in their own deployment configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix ruff linting issues in German recognizers - D205: add blank line between summary and description (de_health_insurance_recognizer) - E501: shorten long lines in de_health_insurance_recognizer, de_kfz_recognizer, de_social_security_recognizer, de_tax_id_recognizer - D214/D406: fix Examples section formatting in de_kfz_recognizer (auto-fixed) All 133 tests still pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(config): enable German language support and fix DE recognizer patterns Enable German (de) as a supported language alongside English: - default_analyzer.yaml: add 'de' to supported_languages - default_recognizers.yaml: add 'de' to supported_languages, enable all nine German PII recognizers (DeTaxIdRecognizer, DeTaxNumberRecognizer, DePassportRecognizer, DeIdCardRecognizer, DeSocialSecurityRecognizer, DeHealthInsuranceRecognizer, DeKfzRecognizer, DeHandelsregisterRecognizer, DePlzRecognizer) - spacy_en_de.yaml: add dedicated en+de NLP config (en_core_web_lg + de_core_news_md) for deployments that only need English and German Fix DeKfzRecognizer: - Replace \b word-boundary anchors with (?<![\w-]) / (?!\w) lookarounds to prevent false matches starting mid-plate after a hyphen separator (e.g. 'M-AB 123' was matching as 'AB 123' instead of 'M-AB 123') - Add two missing patterns for the common Bindestrich+Leerzeichen format '[District]-[Letters] [Digits]' (e.g. M-AB 123, HH-XY 999) Fix DeTaxNumberRecognizer: - Replace \b anchors with (?<!\w) / (?!\w) lookarounds on slash-format patterns to prevent boundary bleed into surrounding words - Raise Bayern/BW (3/3/5) pattern confidence from 0.3 to 0.4 NOTE: After this PR is merged a corresponding update to the LiteLLM Presidio guardrail configuration is needed to map the newly active German entity types (DE_KFZ, DE_TAX_ID, DE_TAX_NUMBER, DE_ID_CARD, DE_PASSPORT, DE_SOCIAL_SECURITY, DE_HEALTH_INSURANCE, DE_HANDELSREGISTER, DE_PLZ) to the desired masking actions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(config): restore English-only defaults and fix import ordering - Remove 'de' from supported_languages in default_analyzer.yaml and default_recognizers.yaml so defaults remain ["en"] as tests expect - Add enabled: false to all German recognizers in default_recognizers.yaml (opt-in instead of opt-out, consistent with other disabled recognizers) - Move Germany imports to correct alphabetical position in __init__.py (after Finland, before India) to fix ruff I001 import-ordering error Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(de-recognizers): address Copilot review feedback on German recognizers - de_id_card_recognizer: fix docstring example T220001292 (10 chars) -> T22000129 (9 chars) - de_plz_recognizer: tighten regex to exclude boundary values 01000 and 99999 via negative lookahead; update docstring accordingly - test_de_plz_recognizer: add negative test cases for 01000 and 99999 - de_social_security_recognizer: fix day regex [4-7]\d (40-79) -> 5[1-9]|[67]\d so supplemental range is correctly 51-81 per spec, not 41-81 - de_tax_number_recognizer: remove undocumented 10-digit plain format from docstring (no corresponding pattern exists) - de_passport_recognizer: remove unused _ICAO_CHARS constant; fix docstring example F204004812 (10 chars) -> F20400481 (9 chars) - de_health_insurance_recognizer: fix docstring example A123456780 (invalid checksum) -> A123456787 (valid checksum) - de_tax_id_recognizer: remove invalid example 02476291358 (leading zero); remove unimplemented digit-frequency constraint from docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(de-recognizers): add DE_LANR, DE_BSNR, DE_VAT_ID, DE_FUEHRERSCHEIN recognizers Adds four new German PII recognizers with tests: - DE_LANR (Lebenslange Arztnummer): 9-digit physician number with KBV check digit validation (weights [4,9,2,10,5,3], cross-sum, mod 10). Legal basis: § 75 Abs. 7 SGB V. - DE_BSNR (Betriebsstättennummer): 9-digit practice/site-of-care number, no public checksum, relies on context words for high-confidence matches. Legal basis: § 75 Abs. 7 SGB V. - DE_VAT_ID (Umsatzsteuer-Identifikationsnummer): fixed-prefix pattern "DE" + 9 digits per § 27a UStG / BZSt format. - DE_FUEHRERSCHEIN (Führerscheinnummer): post-2013 EU-harmonized format [A-Z]{2}\d{8}[A-Z0-9] (11 chars) per FeV Anlage 8 / EU Directive 2006/126/EC. No checksum (KBA algorithm not published). Pre-2013 card formats are explicitly out of scope. Also moves spacy_en_de.yaml from presidio_analyzer/conf/ to docs/recipes/german-language-support/ per reviewer feedback, and reverts the docker-compose.yml NLP_CONF_FILE build arg. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(recipes): align german-language-support README with recipe template Rewrites the README to follow the template.md structure: Overview, Quick Start, Approach (with entity/checksum table), Results (TBD pending formal evaluation), Key Findings, and Tips sections. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
91 lines
3.0 KiB
Python
91 lines
3.0 KiB
Python
"""
|
||
Tests for DeHealthInsuranceRecognizer (Krankenversicherungsnummer / KVNR).
|
||
|
||
Format: 10 characters – 1 uppercase letter (birth surname initial) +
|
||
8 digits (birth date + serial) + 1 check digit.
|
||
|
||
Valid numbers are generated with the GKV-Spitzenverband checksum algorithm
|
||
(letter expanded to 2-digit ordinal, weights [2,9,8,7,6,5,4,3,2,1],
|
||
products digit-summed if ≥ 10, sum mod 10).
|
||
|
||
Legal basis: § 290 SGB V. DSGVO Art. 9 (Gesundheitsdaten).
|
||
|
||
Pre-calculated valid examples (fictitious):
|
||
A123456787 – letter A (=01), data 12345678, check = 7
|
||
M123456789 – letter M (=13), data 12345678, check = 9
|
||
B123456787 – letter B (=02), data 12345678, check = 7
|
||
"""
|
||
import pytest
|
||
|
||
from tests import assert_result
|
||
from presidio_analyzer.predefined_recognizers import DeHealthInsuranceRecognizer
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def recognizer():
|
||
return DeHealthInsuranceRecognizer()
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def entities():
|
||
return ["DE_HEALTH_INSURANCE"]
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"text, expected_len, expected_positions",
|
||
[
|
||
# fmt: off
|
||
# Valid KVNR – checksum passes → result at MAX_SCORE
|
||
("A123456787", 1, ((0, 10),)),
|
||
("M123456789", 1, ((0, 10),)),
|
||
("B123456787", 1, ((0, 10),)),
|
||
("Krankenkasse KVNR: A123456787", 1, ((19, 29),)),
|
||
("eGK-Nummer M123456789 bitte angeben.", 1, ((11, 21),)),
|
||
# Invalid: wrong check digit
|
||
("A123456780", 0, ()),
|
||
("M123456781", 0, ()),
|
||
# Invalid: starts with digit instead of letter
|
||
("1123456787", 0, ()),
|
||
# Lowercase: Presidio uses global IGNORECASE; validate_result calls .upper()
|
||
# so a valid lowercase KVNR is matched and validated correctly
|
||
("a123456787", 1, ((0, 10),)),
|
||
# Too short (9 chars)
|
||
("A12345678", 0, ()),
|
||
# Too long (11 chars)
|
||
("A1234567890", 0, ()),
|
||
# fmt: on
|
||
],
|
||
)
|
||
def test_when_all_de_health_insurance_numbers_then_succeed(
|
||
text, expected_len, expected_positions, recognizer, entities, max_score
|
||
):
|
||
results = recognizer.analyze(text, entities)
|
||
assert len(results) == expected_len
|
||
for res, (st_pos, fn_pos) in zip(results, expected_positions):
|
||
assert_result(res, entities[0], st_pos, fn_pos, max_score)
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"number, expected",
|
||
[
|
||
# Valid
|
||
("A123456787", True),
|
||
("M123456789", True),
|
||
("B123456787", True),
|
||
# Wrong check digit
|
||
("A123456780", False),
|
||
("M123456781", False),
|
||
# Starts with digit (after .upper() it's still a digit → re.match fails)
|
||
("1123456787", False),
|
||
# Wrong length
|
||
("A12345678", False),
|
||
("A1234567890", False),
|
||
# Lowercase: .upper() converts to valid → True (IGNORECASE is global)
|
||
("a123456787", True),
|
||
],
|
||
)
|
||
def test_when_de_health_insurance_validated_then_checksum_result_is_correct(
|
||
number, expected, recognizer
|
||
):
|
||
assert recognizer.validate_result(number) == expected
|