mirror of
https://github.com/data-privacy-stack/presidio.git
synced 2026-07-23 11:20:55 -05:00
98f79b9c64
* feat(analyzer): add German PII recognizers (DE_*) Adds 9 new predefined recognizers for German personally identifiable information, all disabled by default (enabled: false) and scoped to supported_language: de. Recognizers with checksum validation: - DE_TAX_ID: Steueridentifikationsnummer (§§ 139a-e AO) ISO 7064 Mod 11,10 checksum (Bundeszentralamt für Steuern) - DE_SOCIAL_SECURITY: Rentenversicherungsnummer / RVNR (§ 147 SGB VI) Deutsche Rentenversicherung Bund checksum algorithm - DE_HEALTH_INSURANCE: Krankenversicherungsnummer KVNR (§ 290 SGB V) GKV-Spitzenverband checksum; DSGVO Art. 9 (health data) Pattern-based recognizers: - DE_TAX_NUMBER: Steuernummer (§ 139a AO) – ELSTER 13-digit and state-specific slash-separated formats - DE_PASSPORT: Reisepassnummer (PassG § 4, ICAO Doc 9303) - DE_ID_CARD: Personalausweisnummer (PAuswG) – nPA and legacy format - DE_KFZ: KFZ-Kennzeichen (FZV § 8, ECJ C-582/14) - DE_HANDELSREGISTER: Handelsregisternummer HRA/HRB (§§ 9, 14 HGB) - DE_PLZ: Postleitzahl (DSGVO Art. 4 Nr. 1) – very low base confidence (0.05), context words required for actionable results Also updates: - predefined_recognizers/__init__.py: imports and __all__ - conf/default_recognizers.yaml: all 9 recognizers registered - docs/supported_entities.md: new Germany section - CHANGELOG.md: entry under [unreleased] 133 new tests pass (9 test files, one per recognizer). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: enforce LF line endings for shell scripts in containers Added .gitattributes to force LF for *.sh files. CRLF endings in entrypoint.sh caused 'no such file or directory' when running the presidio-analyzer Docker container on Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(config): enable German recognizers and add 'de' language support Enabled all 9 DE_* recognizers in default_recognizers.yaml and added 'de' to supported_languages so they are loaded and visible to clients (e.g. LiteLLM) without requiring ad-hoc configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(config): add 'de' to analyzer engine supported_languages The registry and engine supported_languages must match. Missing 'de' in default_analyzer.yaml caused a startup crash. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(config): restore default enabled=false and single-language defaults Reverts the deployment-specific changes made for local testing: - German recognizers back to enabled=false (upstream contribution convention) - supported_languages back to [en] only in both config files Users who want German PII detection should enable the DE_* recognizers explicitly in their own deployment configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix ruff linting issues in German recognizers - D205: add blank line between summary and description (de_health_insurance_recognizer) - E501: shorten long lines in de_health_insurance_recognizer, de_kfz_recognizer, de_social_security_recognizer, de_tax_id_recognizer - D214/D406: fix Examples section formatting in de_kfz_recognizer (auto-fixed) All 133 tests still pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(config): enable German language support and fix DE recognizer patterns Enable German (de) as a supported language alongside English: - default_analyzer.yaml: add 'de' to supported_languages - default_recognizers.yaml: add 'de' to supported_languages, enable all nine German PII recognizers (DeTaxIdRecognizer, DeTaxNumberRecognizer, DePassportRecognizer, DeIdCardRecognizer, DeSocialSecurityRecognizer, DeHealthInsuranceRecognizer, DeKfzRecognizer, DeHandelsregisterRecognizer, DePlzRecognizer) - spacy_en_de.yaml: add dedicated en+de NLP config (en_core_web_lg + de_core_news_md) for deployments that only need English and German Fix DeKfzRecognizer: - Replace \b word-boundary anchors with (?<![\w-]) / (?!\w) lookarounds to prevent false matches starting mid-plate after a hyphen separator (e.g. 'M-AB 123' was matching as 'AB 123' instead of 'M-AB 123') - Add two missing patterns for the common Bindestrich+Leerzeichen format '[District]-[Letters] [Digits]' (e.g. M-AB 123, HH-XY 999) Fix DeTaxNumberRecognizer: - Replace \b anchors with (?<!\w) / (?!\w) lookarounds on slash-format patterns to prevent boundary bleed into surrounding words - Raise Bayern/BW (3/3/5) pattern confidence from 0.3 to 0.4 NOTE: After this PR is merged a corresponding update to the LiteLLM Presidio guardrail configuration is needed to map the newly active German entity types (DE_KFZ, DE_TAX_ID, DE_TAX_NUMBER, DE_ID_CARD, DE_PASSPORT, DE_SOCIAL_SECURITY, DE_HEALTH_INSURANCE, DE_HANDELSREGISTER, DE_PLZ) to the desired masking actions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(config): restore English-only defaults and fix import ordering - Remove 'de' from supported_languages in default_analyzer.yaml and default_recognizers.yaml so defaults remain ["en"] as tests expect - Add enabled: false to all German recognizers in default_recognizers.yaml (opt-in instead of opt-out, consistent with other disabled recognizers) - Move Germany imports to correct alphabetical position in __init__.py (after Finland, before India) to fix ruff I001 import-ordering error Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(de-recognizers): address Copilot review feedback on German recognizers - de_id_card_recognizer: fix docstring example T220001292 (10 chars) -> T22000129 (9 chars) - de_plz_recognizer: tighten regex to exclude boundary values 01000 and 99999 via negative lookahead; update docstring accordingly - test_de_plz_recognizer: add negative test cases for 01000 and 99999 - de_social_security_recognizer: fix day regex [4-7]\d (40-79) -> 5[1-9]|[67]\d so supplemental range is correctly 51-81 per spec, not 41-81 - de_tax_number_recognizer: remove undocumented 10-digit plain format from docstring (no corresponding pattern exists) - de_passport_recognizer: remove unused _ICAO_CHARS constant; fix docstring example F204004812 (10 chars) -> F20400481 (9 chars) - de_health_insurance_recognizer: fix docstring example A123456780 (invalid checksum) -> A123456787 (valid checksum) - de_tax_id_recognizer: remove invalid example 02476291358 (leading zero); remove unimplemented digit-frequency constraint from docstring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(de-recognizers): add DE_LANR, DE_BSNR, DE_VAT_ID, DE_FUEHRERSCHEIN recognizers Adds four new German PII recognizers with tests: - DE_LANR (Lebenslange Arztnummer): 9-digit physician number with KBV check digit validation (weights [4,9,2,10,5,3], cross-sum, mod 10). Legal basis: § 75 Abs. 7 SGB V. - DE_BSNR (Betriebsstättennummer): 9-digit practice/site-of-care number, no public checksum, relies on context words for high-confidence matches. Legal basis: § 75 Abs. 7 SGB V. - DE_VAT_ID (Umsatzsteuer-Identifikationsnummer): fixed-prefix pattern "DE" + 9 digits per § 27a UStG / BZSt format. - DE_FUEHRERSCHEIN (Führerscheinnummer): post-2013 EU-harmonized format [A-Z]{2}\d{8}[A-Z0-9] (11 chars) per FeV Anlage 8 / EU Directive 2006/126/EC. No checksum (KBA algorithm not published). Pre-2013 card formats are explicitly out of scope. Also moves spacy_en_de.yaml from presidio_analyzer/conf/ to docs/recipes/german-language-support/ per reviewer feedback, and reverts the docker-compose.yml NLP_CONF_FILE build arg. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(recipes): align german-language-support README with recipe template Rewrites the README to follow the template.md structure: Overview, Quick Start, Approach (with entity/checksum table), Results (TBD pending formal evaluation), Key Findings, and Tips sections. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
3 lines
91 B
Plaintext
3 lines
91 B
Plaintext
# Force LF line endings for shell scripts (required for Linux containers)
|
|
*.sh text eol=lf
|