* feat(analyzer): add nine South African predefined recognizers
Extend ZA coverage beyond ZA_ID_NUMBER with passport, tax, VAT, CIPC
registration, eNaTIS driver's licence and traffic register numbers,
licence plates, and mobile/telephone numbers split by line type.
All recognizers are disabled by default in default_recognizers.yaml.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(analyzer): update year validation logic in South African recognizers
Refactor the year validation logic in the ZaCompanyRegistrationRecognizer and ZaDriverLicenseRecognizer to ensure the current year is accurately checked without allowing for the next year. Remove unnecessary dependency on ZaIdNumberRecognizer in ZaDriverLicenseRecognizer to streamline the code. Update the validate_result method in ZaLicensePlateRecognizer to return a boolean type for consistency.
* fix(analyzer): address Copilot review feedback for ZA recognizers
Correct 08x NSN fallback classification, tighten driver licence validation,
and replace the unstable passport docstring reference.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(analyzer): align ZA driver licence docs and address Copilot round 3
Update driver licence length documentation to 10-14 characters to match
validation constraints, and use PhoneNumberMatcher.number directly instead
of re-parsing matched substrings.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update CHANGELOG.md
* Update CHANGELOG.md
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
`PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS` contained "UK", which is not a
valid region code in the phonenumbers library (Google libphonenumber). Region
codes are ISO 3166-1 alpha-2, where the United Kingdom is "GB"
(country_code_for_region("UK") == 0, and "UK" is not in
phonenumbers.SUPPORTED_REGIONS).
As a result the "UK" entry was a no-op: UK numbers written in national/local
format (e.g. "020 7946 0958") were never detected with the default
configuration. Only international-format "+44 ..." numbers matched, because
they carry the country code and match under any region.
Replaced "UK" with "GB" and added a regression test for a national-format
UK number.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Match batch deanonymization with existing traversal
* Keep README examples exercised by CI
* Align batch deanonymize dict API with documented usage
* Keep batch deanonymize docs in contract order
* Keep batch deanonymization from truncating or over-forwarding
---------
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* fix(analyzer): honour language_model_params in BasicLangExtractRecognizer
Fixes#1942. Merge language_model_params into provider_kwargs so timeout,
num_ctx and other LLM provider params configured in YAML actually reach
the provider (e.g. Ollama) instead of being silently dropped.
Also fixes TypeError when kwargs: or language_model_params: is null in YAML.
- Merge language_model_params into provider_kwargs via setdefault (so
explicit kwargs: entries still win for backwards compatibility)
- Guard against null YAML values with 'or {}'
- Strengthen regression tests to assert params reach ModelConfig.provider_kwargs
- Add test for kwargs: null edge case
- Document fix in CHANGELOG
Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped)
* fix(analyzer): honour config_path for LangExtract recognizers in YAML registry
LM recognizers (BasicLangExtractRecognizer, AzureOpenAILangExtractRecognizer)
configured via a recognizer registry YAML silently ignored config_path: the
strict PredefinedRecognizerConfig schema has no config_path field and forbids
extras, so Pydantic dropped it and the recognizer fell back to its bundled
default model configuration.
Add a LangExtractRecognizerConfig model (extra=allow, explicit config_path
field) mirroring the existing HuggingFace/GLiNER configs, and register both
LM recognizer class names in CONFIG_MODEL_MAP so config_path (and other
recognizer-specific kwargs) survive validation and reach the constructor.
Adds regression tests covering config_path preservation via both the model
and the full ConfigurationValidator registry path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Initial plan
* feat: add Python 3.14 compatibility to all remaining Presidio packages
- Update requires-python to <3.15 in anonymizer, image-redactor, cli,
structured, and umbrella packages
- Add Python 3.14 PyPI classifier to all affected packages
- Split spacy dependency markers in image-redactor to exclude 3.8.14
on Python 3.14 (no compatible wheel), matching analyzer pattern
- Extend CI matrix to test all components on Python 3.14
- Document Python 3.14 as a supported version in installation docs
- Add CHANGELOG entry under [unreleased]
Closes#2096
* ci: move Python 3.14 into main python-version matrix
All packages now support 3.14, so add it directly to the matrix array
instead of duplicating each component in `include` entries.
* build: regenerate uv.lock files to match updated requires-python bounds
* fix: merge duplicate Changed headings in CHANGELOG.md Unreleased section
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Fixes#1942. Merge language_model_params into provider_kwargs so timeout,
num_ctx and other LLM provider params configured in YAML actually reach
the provider (e.g. Ollama) instead of being silently dropped.
Also fixes TypeError when kwargs: or language_model_params: is null in YAML.
- Merge language_model_params into provider_kwargs via setdefault (so
explicit kwargs: entries still win for backwards compatibility)
- Guard against null YAML values with 'or {}'
- Strengthen regression tests to assert params reach ModelConfig.provider_kwargs
- Add test for kwargs: null edge case
- Document fix in CHANGELOG
Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped)
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* ci: use uv for test dependency installation + add timeout safety net
The analyzer test job runs `poetry install --all-extras` with no committed
lock, so Poetry performs a universal resolve of the full optional-dependencies
graph on every run. That resolve backtracks for many minutes — effectively
hanging — driven by the langextract extra's deep, loosely-bounded transitive
tree. Reproduced locally: Poetry >6 min (never completed); uv ~2s for the same
158-package resolution.
Switch the test jobs to uv:
- Install uv via astral-sh/setup-uv.
- Create a per-component venv with `uv venv --seed` (seed keeps pip for the
wheel-build step) and expose it via $GITHUB_PATH / $VIRTUAL_ENV.
- `uv pip install -e '.<extras>'` plus the test tooling that previously came
from the Poetry dev group (uv does not read that group).
- Add a job-level `timeout-minutes: 60` (and 30 on the install step) so a bad
resolve fails fast instead of hanging.
No package metadata changed — pyproject.toml is untouched; this only swaps the
CI install tooling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: run tests via uv run inside the uv-managed venv
Use "uv run --no-sync" for spaCy model downloads, pytest, and diff-cover
instead of exposing the venv on $GITHUB_PATH. "uv sync" builds the
project's .venv (resolving the analyzer --all-extras graph in seconds
where Poetry's lockless universal resolve hangs), and the subsequent
steps execute inside it idiomatically. The --all-extras selection stays
in the job matrix, unchanged from main. Wheel-build keeps using the
system interpreter, matching the previous Poetry behaviour.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: install uv via pip instead of the setup-uv action
The astral-sh/setup-uv action is not on the org's allowed-actions list,
so adding it made the CI workflow fail at startup (startup_failure)
before any job ran. Install a pinned uv from PyPI with the already
present setup-python instead; this needs no new third-party action and
keeps the resolver benefits that motivated the Poetry -> uv switch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add pip to the uv venv for spaCy model downloads
spaCy's `download` command shells out to `python -m pip install`, but
uv-created virtualenvs ship without pip, so runtime/first-use model
downloads (e.g. the presidio-cli conftest fetching en_core_web_lg)
failed with SystemExit: 1. Install pip alongside the test tooling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: pre-install tomli for the wheel build on Python < 3.11
`python -m build` imports `tomli` on Python < 3.11 to read pyproject.toml,
but it is not in the hashed requirements-build.txt (pip-compiled on 3.11+
where tomllib is stdlib). Poetry previously pulled tomli onto the runner
as one of its own dependencies, so `pip install --require-hashes` found it
already satisfied. uv has no Python dependencies, so the 3.10 wheel-build
jobs began failing with "requirements must be pinned with ==". Pre-install
a marker-guarded tomli to restore that condition.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: build service Docker images with uv to fix the E2E resolve hang
The local-build-and-E2E job builds the analyzer/anonymizer/image-redactor
images via `docker compose build`. Each Dockerfile ran
`poetry install --no-root --only=main -E server`, which triggered the same
lockless Poetry backtracking that hangs the test jobs: the analyzer image
sat on "Resolving dependencies" for ~40 minutes until the job was
cancelled. Switch the dependency install to `uv pip install --system
-r pyproject.toml --extra server`, which resolves the same graph in
seconds (verified locally: the analyzer install layer completes in ~9s and
imports spacy/flask/gunicorn while correctly omitting the project itself,
matching --no-root). Drop the now-unused POETRY_VIRTUALENVS_CREATE env and
switch `poetry run python install_nlp_models.py` to a direct call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: run gunicorn directly in container entrypoints (drop poetry run)
The Docker images no longer install Poetry (deps come from uv), so the
`exec poetry run gunicorn ...` entrypoints crashed at startup with
"exec: poetry: not found". The analyzer/anonymizer/image-redactor
containers never came up, so every E2E test failed with connection
refused on ports 5002/5003. uv installs gunicorn system-wide, so invoke
it directly. Verified locally: the anonymizer container now starts
healthy and /health returns 200.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: keep uv's wheel cache on the /mnt ephemeral disk
Restore the intent of the old POETRY_CACHE_DIR=/mnt/poetry_cache setup for
uv: point UV_CACHE_DIR at the runner's large ephemeral disk (/mnt, ~65 GB)
so the heavy analyzer --all-extras wheel cache does not fill the smaller
root volume. /mnt is root-owned, so a prep step creates the directory and
hands it to the runner user. UV_LINK_MODE=copy silences the cross-filesystem
hardlink warning, since the project venv lives on the root volume under the
checkout while the cache is on /mnt.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: broaden CHANGELOG/CI comment to cover the Docker image changes
Address review feedback that the PR scope grew beyond the test jobs: the
CHANGELOG now notes the analyzer/anonymizer/image-redactor image builds
and entrypoints also moved from Poetry to uv, and clarifies pyproject.toml
metadata is unchanged. Also clarify the ci.yml install-step comment that
the wheel-build step deliberately uses the system interpreter (not uv run).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* build: adopt PEP 735 dependency groups and commit uv.lock files
Migrate each CI-tested package's development tooling from Poetry's
[tool.poetry.group.dev.dependencies] to the standard PEP 735
[dependency-groups] table, and commit a uv.lock per package. This makes
pyproject.toml the single source of truth for dev tooling and lets CI and
Docker install a pinned, reproducible dependency graph instead of resolving
on every run. The [project] metadata and the poetry-core build backend are
unchanged, so wheels/sdists are byte-for-byte identical.
Covers presidio-analyzer, presidio-anonymizer, presidio-cli,
presidio-image-redactor, presidio-structured.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: install from the committed uv.lock in CI and Docker builds
CI now runs `uv sync --locked --group dev`, which installs exactly the
locked graph and fails if pyproject.toml and uv.lock disagree rather than
silently resolving new versions. The dev group provides the test tooling
(and pip, which spaCy's model download shells out to), so the manual
`uv pip install pytest ...` line is gone. The uv cache is persisted across
runs via actions/cache keyed on uv.lock and trimmed with `uv cache prune
--ci`.
The analyzer/anonymizer/image-redactor images consume the same lockfile
with `uv sync --locked --no-default-groups --extra server
--no-install-project` into UV_PROJECT_ENVIRONMENT=/usr/local, guaranteeing
images use the same dependency graph as CI. Splitting the dependency layer
from the source COPY keeps it cached until pyproject/uv.lock change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: allow pre-existing transformers advisories in dependency review
The newly committed uv.lock files pin exact versions, so dependency-review
now sees two transformers advisories that already exist on main (capped at
transformers <5 by spacy-huggingface-pipelines and tracked as Dependabot
alerts). They are unrelated to the Poetry->uv migration and will be
addressed in separate security PRs, so allow-list the two GHSAs here to
keep this PR's dependency review unblocked.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: require uv.lock review and commits alongside pyproject changes
Address review feedback from @SharonHart:
- CODEOWNERS: require presidio-administrators approval for **/uv.lock, the
same as **/pyproject.toml, so lockfile changes get dependency review.
- copilot-instructions: switch the local dev commands to uv and add an
explicit rule that editing pyproject.toml dependencies requires
regenerating and committing that package's uv.lock in the same change,
since CI installs with `uv sync --locked` and fails on drift.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: reference UV_CACHE_DIR in cache path; align dev-docs to uv
Address review feedback:
- ci.yml: use ${{ env.UV_CACHE_DIR }} for the cache action path instead of
hardcoding /mnt/uv-cache, so the cache and uv stay in sync if the dir
changes.
- copilot-instructions: update the "Technology Stack" bullet to name uv as
the dependency/install tool (poetry-core retained only as build backend),
matching the uv-based dev commands already documented.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* ci: shorten workflow comments; move transformers allow-ghsas out
Trim the verbose explanatory comments in the test job to 1-3 lines each,
and drop the dependency-review `allow-ghsas` entry for the transformers
advisories. That suppression is a security concern rather than part of the
Poetry->uv migration, so it moves to the stacked security PR alongside the
.trivyignore.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: drop stale pip cache; trim CHANGELOG entry
Remove `cache: 'pip'` (keyed on pyproject.toml) from the test job's
setup-python: it is leftover Poetry-era config. Project dependencies now use
the uv cache, and the only remaining pip installs (uv, tomli, build tools)
are tiny and unrelated to pyproject.toml. Also condense the unreleased
CHANGELOG entry to a single concise line; the implementation detail lives in
the PR description.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* small commit to trigger the pipeline again.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The "Creating an image redactor engine in Python" snippet defined the OCR
engine as `diOCR` but passed `ocr=di_ocr` to ImageAnalyzerEngine, raising
NameError when copied verbatim. Use `diOCR` consistently.
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* feat(analyzer): add South African ID number recognizer (ZA_ID_NUMBER)
- Introduced a new recognizer for South African ID numbers, utilizing regex pattern matching, context words, and Luhn checksum validation. The recognizer is disabled by default.
- Updated CHANGELOG to reflect the addition of the ZA_ID_NUMBER recognizer.
- Added documentation for the new recognizer in supported_entities.md.
- Included tests to validate the functionality of the ZA_ID_NUMBER recognizer.
* refactor(za_id_number_recognizer): improve regex documentation and initialization logic
- Updated the documentation for South African ID numbers to correct the format description.
- Refactored the initialization logic to use more explicit conditional expressions for setting patterns and context, enhancing readability and maintainability.
- Renamed the test function to better reflect its purpose, adding a docstring for clarity on testing valid and invalid ID numbers.
* feat(analyzer): reolve ruff linting I001 check - reorder improrts so that South Africa appears in the right order
* refactor(za_id_number_recognizer): enhance validation logic and documentation
- Updated the validation logic for South African ID numbers to use constants for better readability and maintainability.
- Improved documentation for citizenship and age indicator requirements, specifying allowed values.
- Introduced constants for ID length and index positions to streamline future modifications.
* refactor(za_id_number_recognizer): enhance validation and testing for South African ID numbers
- Improved the validation logic for birth dates to ensure they are not in the future and clarified the year mapping.
- Updated documentation to provide detailed explanations of the ID number structure, including citizenship and legacy race classification.
- Renamed constants for better clarity and added new tests to validate the birth date logic and recognize refugee ID numbers.
* feat: add Philippine mobile number (PH_MOBILE_NUMBER) recognizer tests
- Add PhoneRecognizer with supported_regions=["PH"] and supported_entity="PH_MOBILE_NUMBER"
- Add Filipino and English context words for better detection accuracy
- Cover international (+63), national (09XX) formats and valid PH mobile prefixes
- Add invalid number and edge case tests
- Reference: ITU-T E.164
* docs: add PH_MOBILE_NUMBER to CHANGELOG
- Add PH_MOBILE_NUMBER entry under Unreleased section
- Disabled by default; enabled programmatically via PhoneRecognizer
* docs: add PH_MOBILE_NUMBER to supported_entities.md
- Add Philippines section with PH_MOBILE_NUMBER entity
- Supports international (+63 9XX XXX XXXX) and national (09XX XXX XXXX) formats
- Detected via phonenumbers library, disabled by default
* fix: improve comments for clarity in PH mobile number recognizer tests
* refactor: simplify comment for clarity in PH mobile number recognizer tests
* docs: move programmatic phone note to PHONE_NUMBER entity per maintainer feedback
- Remove PH_MOBILE_NUMBER row from supported_entities.md
- Remove TR_PHONE_NUMBER row from Turkey section
- Add programmatic usage note under PHONE_NUMBER with PH and TR examples
* feat: enhance TR_PHONE_NUMBER recognizer with comprehensive validation
- Add Turkey (TR) support to generic PhoneRecognizer
- Extend TR_PHONE_NUMBER to support geographic numbers (2/3/4 prefix)
- Implement ITU-T E.164 compliant validation with MNP awareness
- Add Turkish context words for better detection accuracy
- Update tests and documentation for enhanced coverage
- Legal basis: KTK Madde 23, ITU-T E.164 compliance
Addresses SharonHart's feedback on country-specific checks
* refactor: remove generic PhoneRecognizer TR change from this PR
Generic PhoneRecognizer region changes are out of scope for TR_PHONE_NUMBER.
Focus only on the country-specific recognizer.
* refactor(tr-phone): fix PhoneRecognizer bug and use programmatic config instead of subclass
- Fix PhoneRecognizer._get_recognizer_result to use self.supported_entities[0]
instead of hardcoded 'PHONE_NUMBER', making the supported_entity parameter
from PR #2014 fully functional
- Delete TrPhoneNumberRecognizer subclass; TR phone detection now uses
PhoneRecognizer(supported_regions=['TR'], supported_entity='TR_PHONE_NUMBER',
context=[...]) programmatically per maintainer guidance
- Remove TrPhoneNumberRecognizer from __init__.py, __all__, and
default_recognizers.yaml
- Rewrite tests to use PhoneRecognizer with TR config (40 test cases)
- Update CHANGELOG.md and docs/supported_entities.md
* fix: add trailing newline to fix ruff W292 lint error
---------
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* feat(analyzer): add optional country filter to load_predefined_recognizers
Allows callers to narrow the predefined recognizers loaded by
RecognizerRegistry.load_predefined_recognizers() to a subset of
countries, alongside the existing language filter.
Country is inferred from the recognizer's module path (the segment
directly under `country_specific/`), so no changes to individual
recognizer classes are required. Locale-agnostic recognizers are
always preserved.
Fixes microsoft/presidio#1328
* feat(analyzer): make country_code a first-class recognizer attribute
Adopts plan A from PR #2000 review: rather than inferring country from
module path, country information is now a first-class attribute on
EntityRecognizer, declared once and resolved consistently everywhere.
Changes:
- EntityRecognizer
- New optional `country_code` constructor argument and instance attribute
(lower-cased ISO-3166 alpha-2 by convention; None means generic).
- New `COUNTRY_CODE` ClassVar so subclasses can declare their country
once at the class level without overriding `__init__`.
- `to_dict()` now serializes `country_code` when set.
- PatternRecognizer
- Forwards a new `country_code` constructor argument to the base class so
YAML-defined custom recognizers can declare their country.
- All 60 predefined country-specific recognizers now declare
`COUNTRY_CODE` as a class-level attribute (au, ca, de, es, fi, in, it,
kr, ng, pl, se, sg, th, tr, uk, us). Generic recognizers (credit card,
email, phone, IP, IBAN, dates, etc.) intentionally remain unset.
- RecognizerRegistry
- `_get_recognizer_country` now prefers `recognizer.country_code`, with
legacy module-path inference kept as a fallback for any user-defined
recognizer that has not yet adopted the attribute.
- `_prepare_recognizer_kwargs` drops `country_code` from kwargs when the
target class's `__init__` does not accept it, so existing predefined
recognizers that have not yet been migrated to accept the kwarg keep
working unchanged.
- `filter_by_countries` now keeps generic (None) recognizers and filters
only country-tagged ones; emits a WARNING when a requested country
matches zero recognizers, to aid discoverability.
- New `RecognizerRegistry.get_country_codes()` helper returns the set of
country codes currently loaded.
- Configuration
- `example_recognizers.yaml` documents the `country_code` field and adds
a `BR CPF` example recognizer that declares `country_code: "br"`.
- Documentation
- New `docs/analyzer/filtering_by_country.md` page covering the country
filter, how to declare `country_code` on custom recognizers, when to
leave it unset, backwards compatibility, and debugging tips.
- `docs/analyzer/index.md` and `mkdocs.yml` link to the new page.
- `docs/analyzer/adding_recognizers.md` documents the `COUNTRY_CODE`
convention for predefined recognizers.
- `CONTRIBUTING.md` adds a "Contributing a New Predefined Recognizer"
rule requiring `COUNTRY_CODE` for country-specific recognizers.
- Tests
- Extended `test_recognizer_registry.py` with coverage for attribute-
based filtering, generic (untagged) recognizers, the warning path,
and `get_country_codes()`.
- CHANGELOG: entry under [unreleased] / Analyzer / Added.
Behavior:
- `load_predefined_recognizers(countries=None)` -> all recognizers
(country-tagged + generic), unchanged from before.
- `load_predefined_recognizers(countries=["us"])` -> US-tagged
recognizers + all generic (None) recognizers.
- Backwards compatible: any recognizer that does not set country_code
(custom or otherwise) is treated as generic and is always returned.
Closes#1328 (analyzer side).
Signed-off-by: Nachiket Torwekar <nachiket.torwekar@gmail.com>
Made-with: Cursor
* refactor(analyzer): make COUNTRY_CODE the only source of truth, push country filter into RecognizerListLoader
Addresses both review comments on PR #2000:
1. Discoverability — country tag is now a fact about the recognizer
*class*, not an instance attribute. ``EntityRecognizer`` exposes the
``COUNTRY_CODE`` ClassVar as the canonical declaration, and reads it
back through two new classmethods so callers / new contributors land
on a named API:
- ``EntityRecognizer.country_code() -> Optional[str]`` — lower-cased
ISO code or ``None``.
- ``EntityRecognizer.is_country_specific() -> bool`` — named
predicate; the filter logic uses this so the field is hard to miss
when reading the registry / docs.
The ``country_code`` constructor kwarg, the ``self.country_code``
instance attribute, the ``country_code`` field in ``to_dict()``, and
the temporary ``_prepare_recognizer_kwargs`` shim that swallowed the
YAML kwarg are all removed. Setting ``COUNTRY_CODE`` on a subclass is
now the only supported way to tag a recognizer; instance-level
overrides are intentionally a no-op so the country tag is a single
source of truth at the type level.
2. Threading the filter through ``RecognizerListLoader`` — instead of
running the country filter post-hoc in
``RecognizerRegistry.load_predefined_recognizers``, the filter is now
applied inline inside ``RecognizerListLoader.get(...)``, exactly
alongside ``_is_language_supported_globally``. Concretely:
- ``RecognizerListLoader.get`` gains a
``supported_countries: Optional[Iterable[str]] = None`` kwarg.
- Threaded through ``RecognizerConfigurationLoader`` automatically
(no schema changes needed — the configuration dict simply carries
the new key when set).
- ``RecognizerRegistry.load_predefined_recognizers(countries=...)``
forwards into ``registry_configuration["supported_countries"]``.
The post-hoc ``filter_by_countries`` call site in
``recognizer_registry.py`` is removed.
- The filter is now also driveable from a top-level
``supported_countries: ["us", "uk"]`` field in the registry YAML,
mirroring how ``supported_languages`` works. Documented in
``docs/analyzer/filtering_by_country.md`` and
``conf/example_recognizers.yaml``.
Other follow-ups in this commit:
- ``filter_by_countries`` survives as the helper called internally by
``RecognizerListLoader.get``; it now reads country via the
classmethod (``recognizer.country_code()``) rather than poking at an
instance attribute. Module-path inference is kept as a transitional
fallback for any custom recognizer that follows the
``country_specific/<iso>/`` directory convention but hasn't adopted
the class attribute.
- ``RecognizerRegistry.get_country_codes()`` reads via the classmethod.
- ``example_recognizers.yaml`` drops the BR CPF custom example that
relied on the now-removed YAML ``country_code:`` field; that field
was only ever a thin wrapper over the constructor kwarg, and the
per-recognizer YAML route is intentionally left out of this commit
pending follow-up review feedback.
- Docs (``filtering_by_country.md``, ``adding_recognizers.md``,
``CONTRIBUTING.md``) updated to point exclusively at the ClassVar +
classmethod API.
- Tests rewritten against the classmethod API; tests for the removed
constructor kwarg / instance attribute are dropped, and a new test
exercises ``RecognizerListLoader.get(supported_countries=...)``
directly to lock in the loader-level filter contract.
Behavior is unchanged from the prior commit on this branch:
``countries=None`` → all recognizers; ``countries=["us"]`` → US-tagged
+ all locale-agnostic recognizers; untagged recognizers are always
kept.
Signed-off-by: Nachiket Torwekar <nachiket.torwekar@gmail.com>
Made-with: Cursor
* review(analyzer): YAML country tags, input validation, missing COUNTRY_CODEs
Addresses three review comments on PR microsoft#2000:
1. ``default_recognizers.yaml`` now declares ``country_code: <iso>`` on
every predefined country-specific entry. The field is advisory
metadata for no-code YAML users; the class-level ``COUNTRY_CODE``
ClassVar remains the single source of truth. The loader strips
``country_code`` from instantiation kwargs and cross-checks it
against the class attribute via a new
``RecognizerListLoader._validate_yaml_country_code`` helper —
YAML/class disagreement raises ``ValueError`` at load time naming
both values, so the misconfiguration is fixable from the error
message alone. Locale-agnostic predefined recognizers omit the
field; YAML-on-locale-agnostic-class is also rejected as a
misconfiguration.
2. Input validation for ``countries=`` / ``supported_countries=``.
``RecognizerListLoader.filter_by_countries`` (and, transitively,
``RecognizerRegistry.load_predefined_recognizers(countries=...)``
and ``RecognizerListLoader.get(supported_countries=...)``) now
reject the common footguns up front through
``_normalize_countries``:
- Bare ``str`` (e.g. ``countries="us"``) raises ``TypeError``
instead of silently iterating over characters.
- Non-iterable scalars raise ``TypeError``.
- Non-string elements raise ``TypeError`` naming the offender.
- Blank / whitespace-only codes raise ``ValueError``.
- Whitespace is stripped and codes are lower-cased.
3. ``COUNTRY_CODE`` declarations added to two recognizers that were
missing them after the merge from main:
- ``ItVatCodeRecognizer`` (``it``) — flagged in review.
- ``EsPassportRecognizer`` (``es``) — newly added in main during
this PR's review cycle, so it never went through the original
migration sweep.
Docs / changelog / contributing:
- ``docs/analyzer/filtering_by_country.md``: new section "Annotating
predefined recognizers in YAML" walking through the per-recognizer
``country_code:`` field, the cross-check, and the resulting error.
- ``CONTRIBUTING.md``: the predefined-recognizer checklist now
includes the YAML ``country_code:`` step alongside ``enabled:
false``.
- ``CHANGELOG.md``: extended the unreleased entry to mention the
per-recognizer YAML field, the cross-check, and the input
validation contract.
Tests:
- 7 new tests in ``tests/test_recognizer_registry.py`` covering input
validation (bare str, non-iterable, non-string element, blank
element, normalization, end-to-end through
``load_predefined_recognizers``) and YAML cross-validation (every
YAML ``country_code`` in the shipped default config matches its
class, mismatch raises, locale-agnostic-class with a YAML value
raises, blank YAML value raises). Existing 35 tests in the file
continue to pass.
Local verification: 169 tests pass across recognizer_registry,
recognizer_registry_provider, analyzer_engine, pattern_recognizer,
it_vat_code_recognizer, and es_passport_recognizer. ``ruff check``
is net-improved vs. baseline (44 → 32 pre-existing errors only;
all newly-introduced lint issues fixed). ``ruff format --check`` is
clean on all touched production files.
Signed-off-by: Nachiket Torwekar <nachiket.torwekar@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(analyzer): country_code constructor kwarg unlocks YAML type:custom tagging
Addresses the final review thread on PR microsoft#2000: custom recognizers
defined entirely in YAML (``type: custom``) couldn't previously declare
a country tag — the only path was subclassing ``PatternRecognizer`` and
setting the class-level ``COUNTRY_CODE``. This commit adds the
constructor route so YAML-only users get the same capability.
EntityRecognizer:
- New optional ``country_code: Optional[str] = None`` constructor kwarg.
Reconciled at construction time against the class-level
``COUNTRY_CODE`` ClassVar via a new ``_resolve_country_code`` helper:
| ClassVar | kwarg | result
| -------------- | ---------------- | -----------------------
| None | None | self._country_code=None
| None | "am" | self._country_code="am" (NEW)
| "us" | None | self._country_code="us"
| "us" | "us" (matches) | self._country_code="us"
| "us" | "uk" (conflict) | ValueError
The conflict path means a predefined country recognizer can never be
silently re-tagged via the constructor — misconfiguration raises with
both values in the message.
- ``country_code()`` and ``is_country_specific()`` flip from classmethods
to instance methods reading ``self._country_code``. The class-level
attribute remains accessible as ``cls.COUNTRY_CODE`` for callers that
need to introspect without instantiating.
- ``to_dict()`` now emits ``country_code`` whenever the resolved tag is
non-None, so ``from_dict(to_dict(rec))`` round-trips losslessly for
both class-tagged and constructor-tagged recognizers.
PatternRecognizer:
- Threads ``country_code`` through to ``super().__init__``. This is what
makes ``PatternRecognizer.from_dict({"country_code": "am", ...})``
just work — and by extension, the YAML ``type: custom`` route.
Loader (``RecognizerListLoader``):
- ``custom_to_exclude`` no longer strips ``country_code``: the field
flows into ``__init__`` via the kwargs path. Predefined entries still
strip it (their ``__init__`` signatures don't accept the kwarg) and
cross-check against the class attribute via the existing
``_validate_yaml_country_code`` helper.
- ``_validate_yaml_country_code`` switches to reading
``recognizer_cls.COUNTRY_CODE`` directly (the classmethod is gone).
- ``_get_recognizer_country`` docstring updated to reflect that the
instance method now covers both tagging paths uniformly.
Tests:
- Replaced ``test_country_code_classmethod_lowercases_class_attribute``
(renamed; now exercises the instance method) and the
``test_predefined_country_specific_recognizer_carries_class_country_code``
classmethod-on-class assertion (now reads ``cls.COUNTRY_CODE`` for
the no-instantiation path).
- Replaced ``test_country_code_is_a_class_level_fact_not_instance``
(asserted instance overrides were a no-op — that contract is gone)
with ``test_country_code_class_attribute_is_authoritative_over_instance_attr``,
which locks in the new "resolved once at construction time, cached
on the instance" contract.
- Replaced ``test_to_dict_does_not_carry_country_code`` with two tests:
``to_dict`` omits ``country_code`` for untagged recognizers and emits
it for class-tagged ones.
- New construction-matrix coverage:
``test_constructor_country_code_tags_locale_agnostic_class``,
``test_constructor_country_code_normalizes_case_and_whitespace``,
``test_constructor_country_code_rejects_blank``,
``test_constructor_country_code_rejects_non_string``,
``test_constructor_country_code_matches_class_level_is_fine``,
``test_constructor_country_code_conflicts_with_class_level_raises``,
``test_constructor_country_code_match_is_case_insensitive``.
- New round-trip and end-to-end coverage:
``test_pattern_recognizer_from_dict_roundtrip_preserves_country_code``,
``test_custom_yaml_country_code_flows_through_to_filter`` (loads a
``type: custom`` YAML with ``country_code: am`` through the real
``RecognizerConfigurationLoader`` / ``RecognizerListLoader.get`` path
and asserts it survives the country filter when included and is
dropped when excluded).
Docs:
- ``docs/analyzer/filtering_by_country.md`` "How the filter works"
section rewritten to cover both tagging paths and the reconciliation
rules. "Declaring a country on your own recognizers" now has three
numbered subsections — subclass + ClassVar (Brazilian CPF), Python
constructor kwarg (Armenian one-off), and YAML ``type: custom``
(Armenian no-code) — plus a "Combining paths" note covering the
conflict-raises invariant.
- CHANGELOG entry extended to mention the constructor kwarg, the
reconciliation rules, and the ``to_dict`` / ``from_dict`` round-trip.
Behavior summary (unchanged from prior commit on this branch):
- Predefined recognizers stay tagged at the class level; their YAML
entries still carry advisory ``country_code:`` cross-checked at load.
- Custom recognizers can now declare ``country_code`` via constructor
kwarg, ``from_dict``, or YAML ``type: custom`` — all three end up at
the same ``self._country_code`` and are filtered identically.
- ``countries=None`` → all recognizers; ``countries=["us"]`` → US-tagged
+ locale-agnostic + untagged custom recognizers; conflicts raise.
Local verification: 234 tests pass across recognizer_registry,
recognizer_registry_provider, analyzer_engine, pattern_recognizer,
pattern, it_vat_code, es_passport, us_ssn, pl_pesel, uk_nino, es_nif.
``ruff check`` is clean on all touched production files; ``ruff format
--check`` clean on all touched production files. Test-file lint stats
are net-improved vs. prior commit (32 → 29 pre-existing only).
Signed-off-by: Nachiket Torwekar <nachiket.torwekar@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address country filter review feedback
* Add custom YAML country_code validation tests
---------
Signed-off-by: Nachiket Torwekar <nachiket.torwekar@gmail.com>
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
Co-authored-by: ntorwe_nike <nachiket.torwekar@nike.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: Add UK driving licence number recognizer
Add UkDrivingLicenceRecognizer for detecting DVLA driving licence numbers.
The 16-character alphanumeric format encodes surname, date of birth, and
initials. Uses regex with embedded month/day validation and surname padding
checks. Disabled by default per country-specific convention.
* Update default_recognizers.yaml
---------
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* feat(analyzer): add German PII recognizers (DE_*)
Adds 9 new predefined recognizers for German personally identifiable
information, all disabled by default (enabled: false) and scoped to
supported_language: de.
Recognizers with checksum validation:
- DE_TAX_ID: Steueridentifikationsnummer (§§ 139a-e AO)
ISO 7064 Mod 11,10 checksum (Bundeszentralamt für Steuern)
- DE_SOCIAL_SECURITY: Rentenversicherungsnummer / RVNR (§ 147 SGB VI)
Deutsche Rentenversicherung Bund checksum algorithm
- DE_HEALTH_INSURANCE: Krankenversicherungsnummer KVNR (§ 290 SGB V)
GKV-Spitzenverband checksum; DSGVO Art. 9 (health data)
Pattern-based recognizers:
- DE_TAX_NUMBER: Steuernummer (§ 139a AO) – ELSTER 13-digit and
state-specific slash-separated formats
- DE_PASSPORT: Reisepassnummer (PassG § 4, ICAO Doc 9303)
- DE_ID_CARD: Personalausweisnummer (PAuswG) – nPA and legacy format
- DE_KFZ: KFZ-Kennzeichen (FZV § 8, ECJ C-582/14)
- DE_HANDELSREGISTER: Handelsregisternummer HRA/HRB (§§ 9, 14 HGB)
- DE_PLZ: Postleitzahl (DSGVO Art. 4 Nr. 1) – very low base confidence
(0.05), context words required for actionable results
Also updates:
- predefined_recognizers/__init__.py: imports and __all__
- conf/default_recognizers.yaml: all 9 recognizers registered
- docs/supported_entities.md: new Germany section
- CHANGELOG.md: entry under [unreleased]
133 new tests pass (9 test files, one per recognizer).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: enforce LF line endings for shell scripts in containers
Added .gitattributes to force LF for *.sh files.
CRLF endings in entrypoint.sh caused 'no such file or directory'
when running the presidio-analyzer Docker container on Linux.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(config): enable German recognizers and add 'de' language support
Enabled all 9 DE_* recognizers in default_recognizers.yaml and added
'de' to supported_languages so they are loaded and visible to clients
(e.g. LiteLLM) without requiring ad-hoc configuration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(config): add 'de' to analyzer engine supported_languages
The registry and engine supported_languages must match.
Missing 'de' in default_analyzer.yaml caused a startup crash.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert(config): restore default enabled=false and single-language defaults
Reverts the deployment-specific changes made for local testing:
- German recognizers back to enabled=false (upstream contribution convention)
- supported_languages back to [en] only in both config files
Users who want German PII detection should enable the DE_* recognizers
explicitly in their own deployment configuration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: fix ruff linting issues in German recognizers
- D205: add blank line between summary and description (de_health_insurance_recognizer)
- E501: shorten long lines in de_health_insurance_recognizer, de_kfz_recognizer,
de_social_security_recognizer, de_tax_id_recognizer
- D214/D406: fix Examples section formatting in de_kfz_recognizer (auto-fixed)
All 133 tests still pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(config): enable German language support and fix DE recognizer patterns
Enable German (de) as a supported language alongside English:
- default_analyzer.yaml: add 'de' to supported_languages
- default_recognizers.yaml: add 'de' to supported_languages, enable all
nine German PII recognizers (DeTaxIdRecognizer, DeTaxNumberRecognizer,
DePassportRecognizer, DeIdCardRecognizer, DeSocialSecurityRecognizer,
DeHealthInsuranceRecognizer, DeKfzRecognizer, DeHandelsregisterRecognizer,
DePlzRecognizer)
- spacy_en_de.yaml: add dedicated en+de NLP config (en_core_web_lg +
de_core_news_md) for deployments that only need English and German
Fix DeKfzRecognizer:
- Replace \b word-boundary anchors with (?<![\w-]) / (?!\w) lookarounds
to prevent false matches starting mid-plate after a hyphen separator
(e.g. 'M-AB 123' was matching as 'AB 123' instead of 'M-AB 123')
- Add two missing patterns for the common Bindestrich+Leerzeichen format
'[District]-[Letters] [Digits]' (e.g. M-AB 123, HH-XY 999)
Fix DeTaxNumberRecognizer:
- Replace \b anchors with (?<!\w) / (?!\w) lookarounds on slash-format
patterns to prevent boundary bleed into surrounding words
- Raise Bayern/BW (3/3/5) pattern confidence from 0.3 to 0.4
NOTE: After this PR is merged a corresponding update to the LiteLLM
Presidio guardrail configuration is needed to map the newly active
German entity types (DE_KFZ, DE_TAX_ID, DE_TAX_NUMBER, DE_ID_CARD,
DE_PASSPORT, DE_SOCIAL_SECURITY, DE_HEALTH_INSURANCE,
DE_HANDELSREGISTER, DE_PLZ) to the desired masking actions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(config): restore English-only defaults and fix import ordering
- Remove 'de' from supported_languages in default_analyzer.yaml and
default_recognizers.yaml so defaults remain ["en"] as tests expect
- Add enabled: false to all German recognizers in default_recognizers.yaml
(opt-in instead of opt-out, consistent with other disabled recognizers)
- Move Germany imports to correct alphabetical position in __init__.py
(after Finland, before India) to fix ruff I001 import-ordering error
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(de-recognizers): address Copilot review feedback on German recognizers
- de_id_card_recognizer: fix docstring example T220001292 (10 chars) -> T22000129 (9 chars)
- de_plz_recognizer: tighten regex to exclude boundary values 01000 and 99999
via negative lookahead; update docstring accordingly
- test_de_plz_recognizer: add negative test cases for 01000 and 99999
- de_social_security_recognizer: fix day regex [4-7]\d (40-79) -> 5[1-9]|[67]\d
so supplemental range is correctly 51-81 per spec, not 41-81
- de_tax_number_recognizer: remove undocumented 10-digit plain format from
docstring (no corresponding pattern exists)
- de_passport_recognizer: remove unused _ICAO_CHARS constant; fix docstring
example F204004812 (10 chars) -> F20400481 (9 chars)
- de_health_insurance_recognizer: fix docstring example A123456780 (invalid
checksum) -> A123456787 (valid checksum)
- de_tax_id_recognizer: remove invalid example 02476291358 (leading zero);
remove unimplemented digit-frequency constraint from docstring
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(de-recognizers): add DE_LANR, DE_BSNR, DE_VAT_ID, DE_FUEHRERSCHEIN recognizers
Adds four new German PII recognizers with tests:
- DE_LANR (Lebenslange Arztnummer): 9-digit physician number with KBV
check digit validation (weights [4,9,2,10,5,3], cross-sum, mod 10).
Legal basis: § 75 Abs. 7 SGB V.
- DE_BSNR (Betriebsstättennummer): 9-digit practice/site-of-care number,
no public checksum, relies on context words for high-confidence matches.
Legal basis: § 75 Abs. 7 SGB V.
- DE_VAT_ID (Umsatzsteuer-Identifikationsnummer): fixed-prefix pattern
"DE" + 9 digits per § 27a UStG / BZSt format.
- DE_FUEHRERSCHEIN (Führerscheinnummer): post-2013 EU-harmonized format
[A-Z]{2}\d{8}[A-Z0-9] (11 chars) per FeV Anlage 8 / EU Directive
2006/126/EC. No checksum (KBA algorithm not published). Pre-2013
card formats are explicitly out of scope.
Also moves spacy_en_de.yaml from presidio_analyzer/conf/ to
docs/recipes/german-language-support/ per reviewer feedback, and
reverts the docker-compose.yml NLP_CONF_FILE build arg.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* docs(recipes): align german-language-support README with recipe template
Rewrites the README to follow the template.md structure: Overview,
Quick Start, Approach (with entity/checksum table), Results (TBD
pending formal evaluation), Key Findings, and Tips sections.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* feat: Add UK passport and vehicle registration recognizers
Add two new UK-specific recognizers:
- UK_PASSPORT: detects 2-letter + 7-digit passport numbers (2015+ format)
- UK_VEHICLE_REGISTRATION: detects current (2001+), prefix (1983-2001),
and suffix (1963-1983) number plate formats with age identifier validation
* style: fix ruff formatting in UK recognizer tests
---------
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Add a pattern-based recognizer for UK postcodes covering all six
standard formats (A9, A99, A9A, AA9, AA99, AA9A) plus GIR 0AA.
The regex enforces position-specific letter restrictions per Royal Mail
rules. Base score is 0.1 due to the short length of postcodes, with
context words (postcode, address, delivery, etc.) boosting confidence.
Disabled by default per country-specific convention.
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
