Extract the release build/push and manifest jobs into a reusable
build-images.yml workflow, extend it with a distroless variant, and add
a weekly rebuild-images.yml so digest-pinned bases pick up security
updates without a manual re-pin.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Deleting uv, pip and setuptools in a runtime RUN removed them from the
final filesystem but not from the image: the COPY layer that carried
them is still shipped. The analyzer's /usr/local/bin layer measured
48.9MB while the directory itself is 148KB in the final image.
Stage uv in /usr/bin, outside the /usr/local tree the runtime stage
copies from, and strip packaging tooling in the builder before the copy.
The runtime removal stays, because it also has to clear the copies that
ship in the runtime base image.
anonymizer 332MB -> 252MB, analyzer 1.59GB -> 1.50GB. Both are now
smaller than the currently published images (355MB / 1.53GB). CVE counts
are unchanged at 111, health checks pass, the encrypt/decrypt round-trip
returns the original text and the analyzer still detects PERSON,
EMAIL_ADDRESS, PHONE_NUMBER and CREDIT_CARD.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* ci: auto-label pull requests by component and area
Adds a path-based labeler so every pull request arrives routed to the
package it changes and the analyzer subsystem it touches, instead of
being triaged by hand.
Uses pull_request_target for the same reason label-external.yml does:
fork pull requests get a read-only token on pull_request, so the label
call would fail silently. The action never checks out or runs pull
request code, and both the workflow and the config are read from the
base branch, so a pull request cannot alter its own label rules.
sync-labels is left off so hand-applied triage labels are never removed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183ij1wiTEcd8wA58xY7e6C
* ci: label pull requests touching docs/samples
Samples are documentation but also code that has to keep working, so they
need to be findable on their own rather than only under documentation.
Reuses the existing `samples` label.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183ij1wiTEcd8wA58xY7e6C
* ci: correct facet documentation in labeler config
The header described two facets while the file defines three, and
`samples` sat under the component heading rather than type.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183ij1wiTEcd8wA58xY7e6C
* ci: name the labeler step for all three facets
The workflow and step names described component and area only, while the
config also applies type labels.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183ij1wiTEcd8wA58xY7e6C
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(anonymizer): bump cryptography to >=50.0.0 for GHSA-g6cj-pr64-35w5
GHSA-g6cj-pr64-35w5 (CVE-2026-69247) affects cryptography >=44.0.0,<50.0.0
and is first patched in 50.0.0. The <49.0.0 ceiling made presidio-anonymizer
and the patched release mutually exclusive, so downstream projects could not
take the fix without dropping presidio-anonymizer.
Same shape as #2143, resolved by #2144.
Fixes: #2229
* fix(anonymizer): synchronize cryptography 50 lockfile
Regenerate the lock with uv 0.11.6 for cryptography 50.0.1 while preserving the >=50.0.0,<51.0.0 constraint and all other package versions. Canonicalize mirror artifact locations against authoritative public metadata and retain the resolved exceptiongroup typing-extensions marker.
Remove only the PR-added changelog entry, preserving upstream release notes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 827bfc85-5ab8-4ca3-b15c-21a897b0c0dd
---------
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Co-authored-by: Sharon Hart <shhart@microsoft.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 827bfc85-5ab8-4ca3-b15c-21a897b0c0dd
* fix(analyzer): register KrPassportRecognizer and align its default language
KrPassportRecognizer was the only Korean recognizer absent from
default_recognizers.yaml, for two reasons that also kept it out of the
predefined registry.
RecognizerListLoader.get instantiates every predefined recognizer with a
name keyword argument, taken from the YAML entry (or from name when
class_name supplies the class). KrPassportRecognizer.__init__ did not
accept it, so listing the recognizer in default_recognizers.yaml made the
registry raise TypeError on load. Adding the argument is what makes the
entry possible.
Its default supported_language was kr, while every other Korean
recognizer defaults to ko, the ISO 639-1 code. #1742 migrated the Korean
recognizers from kr to ko; #1814 added this one afterwards and
reintroduced kr, so an AnalyzerEngine running ko silently skipped it. The
default is now ko. This only affects direct instantiation: when an entry
omits supported_languages the loader passes the registry's language
explicitly, so the class default never applies on the YAML path.
Registers the recognizer with enabled: false and country_code: kr,
matching its siblings, and covers all three points with tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(analyzer): make every listed recognizer loadable from the registry YAML
Three recognizers ship in default_recognizers.yaml with enabled: false but
cannot be turned on. KrBrnRecognizer, KrDriverLicenseRecognizer and
UsMbiRecognizer do not accept the name keyword argument that
RecognizerListLoader passes to every predefined recognizer, so flipping
enabled to true raises
TypeError: __init__() got an unexpected keyword argument 'name'
before the registry finishes loading. enabled: false is an opt-in switch,
not a disclaimer: an entry that cannot be enabled should not be listed.
The failure also reads as a user configuration error even though the YAML
is correct, unlike the optional-dependency entries, which refuse to load
with an actionable ImportError.
Adds the argument to the three constructors, and adds contract tests so
the next one is caught by CI.
Why this survived: the constructor signature is part of a contract that
nothing enforced. Each recognizer's own tests instantiate the class
directly, where no name is passed, so they all pass. The registry-level
tests build the default configuration, in which roughly 60 entries are
disabled and therefore never constructed. Nothing in between ever looked.
The new tests close that gap from both sides:
- Every predefined PatternRecognizer subclass must accept the kwargs the
loader passes. This fires when the class is added, before it reaches the
YAML at all, which is the point at which KrPassportRecognizer went
wrong in #1814.
- Every entry in default_recognizers.yaml must resolve to a class and must
load once enabled, exercised entry by entry so a failure names the
recognizer.
- A class_name plus name entry must produce an instance with the
configured name, which is the documented reason the loader passes name
and what makes the kwarg contract load-bearing.
Verified by reverting the three constructors: the signature test and the
load test each fail for exactly those three, with no other failures.
One entry is excluded from the load test by name, with its reason recorded
next to the exclusion, and a further test asserts each exclusion still
matches a shipped entry so the list cannot rot:
- HuggingFaceNerRecognizer cannot load from its shipped entry even with
its dependencies installed. EntityRecognizer.__init__ calls load()
unconditionally, load() requires model_name, and the entry does not
supply one, so it raises ValueError rather than the ImportError it
raises when transformers is absent. Supplying model_name here would make
the test download a model. The entry stays covered by the resolve test.
The load test skips only BasicLangExtractRecognizer, and only on
ImportError, because refusing to load without the langextract extra is that
recognizer's intended behavior. The skip is scoped to that name rather than
to the exception type, so an ImportError from any other entry is a failure
instead of a green skip.
BasicLangExtractRecognizer also carries a config_path that the recognizer
resolves against the working directory, so it raises FileNotFoundError when
pytest runs from the repository root. The load test sets the working
directory to the component root, the same one CI uses, instead of catching
that error: catching FileNotFoundError would turn a deleted or renamed
shipped config file into a passing skip. A separate test asserts every
config_path in the shipped configuration resolves to a file that exists,
which holds even in an environment that cannot construct the recognizer at
all.
The load test deliberately covers non-pattern entries too. Narrowing it to
PatternRecognizer subclasses would silently drop PhoneRecognizer,
ZaMobileNumberRecognizer and ZaTelephoneNumberRecognizer, which are not
PatternRecognizers and do load.
The signature test is scoped to PatternRecognizer subclasses.
AzureAILanguageRecognizer is the one remaining class that does not accept
name; it is absent from default_recognizers.yaml and fixes its own display
name, so it is out of scope here, but it is reachable from a user config by
class name and fails the same way. That is pre-existing and tracked
separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(analyzer): tighten the recognizer contract tests
Addresses the review on #2170.
The KR passport YAML load test wrote the entry to a `tempfile.mkdtemp()`
directory that nothing removed. Rather than wrap it in a context manager,
the temporary file is gone: `RecognizerRegistryProvider` accepts the
parsed mapping via `registry_configuration`, which is what the sibling
contract test in this PR already does, so the round trip through the
filesystem was never needed.
Its assertion also only checked that `KR_PASSPORT` was among the loaded
entities. Both tests now assert on the loaded recognizer *class*: the
loader drops a recognizer whose language the registry does not support
with a log warning and no exception, so a non-empty registry does not on
its own prove the entry under test is what loaded.
Also in this commit:
- Removed a leftover debug `print` from the parametrized passport test.
It predates this PR -- stripping trailing whitespace on the line above
pulled it into the diff -- but it is adjacent to code this PR touches.
- `default_recognizers.yaml` is parsed once at import instead of once per
parametrized load test, which re-read it for the same two keys.
- A `**kwargs` constructor now reports `pytest.skip` with a reason rather
than returning as a silent pass, so the gap stays visible. No shipped
`PatternRecognizer` subclass takes `**kwargs` today.
- `test_yaml_entry_class_resolves` asserts the resolved object is an
`EntityRecognizer` subclass instead of relying on the lookup raising.
- Dropped the entry count from the module docstring. It was accurate but
drifts every time a recognizer is added.
* test(analyzer): fold the contract tests into the loader test file
Review feedback: tests in this repo are arranged by the file they test, so
a standalone test_predefined_recognizer_contract.py is the wrong shape.
Moved into tests/test_recognizers_loader_utils.py, which covers
recognizers_loader_utils.py -- the module whose contract these assert.
RecognizerListLoader is what passes `name`/`supported_language` to every
predefined recognizer, what resolves an entry's class, and what consumes
default_recognizers.yaml. That file already hosts the sibling shipped-YAML
check (test_default_recognizers_yaml_country_code_matches_class), so the
two now sit together instead of in separate files.
The tests themselves are unchanged; the module docstring became a section
header. test_default_recognizers_yaml_country_code_matches_class now reads
the shipped YAML through the module-level constant the move introduced,
rather than re-opening the same file a second time in one module.
Test count is unchanged at 295 (294 pass, 1 skip -- BasicLangExtract, no
optional dependency installed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(analyzer): check loader kwargs on YAML-listed non-pattern recognizers too
Follow-up to review feedback on #2170 suggesting a signature-level check
driven from default_recognizers.yaml.
The signature sweep was package-driven and scoped to PatternRecognizer
subclasses, which left five shipped entries unreached: PhoneRecognizer, the
two Za* ones, and the NER/LLM wrappers. Three of those the load test already
constructs outright, but HuggingFaceNerRecognizer is excluded from it (its
shipped entry supplies no model_name), so nothing in the suite touched its
constructor at all.
Parametrize over the union of both sources instead. Neither subsumes the
other: the package sweep reaches a class before it is listed anywhere, which
is how KrPassportRecognizer failed -- it could not be added to the yaml at
all, so a yaml-driven check could never have named it -- while the yaml
sweep reaches a listed class the package sweep skips by base class.
Renamed to test_recognizer_accepts_loader_kwargs, since it is no longer
pattern-only. Adds 5 params: 3 assertions and 2 skips, the skips being the
**kwargs constructors whose signature cannot show which kwargs they honor.
HuggingFaceNerRecognizer now surfaces as a named skip explaining why it is
uncovered, rather than being silently absent.
297 passed, 3 skipped (was 294 / 1).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* feat: Add CaPostalCodeRecognizer for Canadian postal codes (CA_POSTAL_CODE)
* fix: add fr to CaPostalCodeRecognizer supported_languages per Copilot review
---------
Co-authored-by: Steve Watson <steve@disciplinedactions.com>
* feat(analyzer): add nine South African predefined recognizers
Extend ZA coverage beyond ZA_ID_NUMBER with passport, tax, VAT, CIPC
registration, eNaTIS driver's licence and traffic register numbers,
licence plates, and mobile/telephone numbers split by line type.
All recognizers are disabled by default in default_recognizers.yaml.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(analyzer): update year validation logic in South African recognizers
Refactor the year validation logic in the ZaCompanyRegistrationRecognizer and ZaDriverLicenseRecognizer to ensure the current year is accurately checked without allowing for the next year. Remove unnecessary dependency on ZaIdNumberRecognizer in ZaDriverLicenseRecognizer to streamline the code. Update the validate_result method in ZaLicensePlateRecognizer to return a boolean type for consistency.
* fix(analyzer): address Copilot review feedback for ZA recognizers
Correct 08x NSN fallback classification, tighten driver licence validation,
and replace the unstable passport docstring reference.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(analyzer): align ZA driver licence docs and address Copilot round 3
Update driver licence length documentation to 10-14 characters to match
validation constraints, and use PhoneNumberMatcher.number directly instead
of re-parsing matched substrings.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update CHANGELOG.md
* Update CHANGELOG.md
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Initial plan
* chore: bump presidio-anonymizer to >=2.2.364 for CVE fix in presidio-structured
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
`PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS` contained "UK", which is not a
valid region code in the phonenumbers library (Google libphonenumber). Region
codes are ISO 3166-1 alpha-2, where the United Kingdom is "GB"
(country_code_for_region("UK") == 0, and "UK" is not in
phonenumbers.SUPPORTED_REGIONS).
As a result the "UK" entry was a no-op: UK numbers written in national/local
format (e.g. "020 7946 0958") were never detected with the default
configuration. Only international-format "+44 ..." numbers matched, because
they carry the country code and match under any region.
Replaced "UK" with "GB" and added a regression test for a national-format
UK number.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add Philippine UMID (PH_UMID) recognizer
- Add PhUmidRecognizer using PatternRecognizer
- Support dashed (0111-1234567-8) and plain 12-digit formats
- Add Filipino context words and COUNTRY_CODE = "ph"
* feat: add PH_UMID recognizer tests
- Cover valid/invalid formats, context words, and edge cases
- Include multiple UMID detection in single text
* feat: register PhUmidRecognizer in config and exports
- Add import and __all__ entry in predefined_recognizers
- Register in default_recognizers.yaml with enabled: false
* docs: add PH_UMID to CHANGELOG and supported_entities.md
- Add entry under Unreleased section
- Add Philippines section between Turkey and Germany
* fix: correct test score ranges for plain 12-digit UMID format
* feat: export predefined recognizers in a new package initialization file
* chore: remove CHANGELOG.md modifications per reviewer request
---------
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Match batch deanonymization with existing traversal
* Keep README examples exercised by CI
* Align batch deanonymize dict API with documented usage
* Keep batch deanonymize docs in contract order
* Keep batch deanonymization from truncating or over-forwarding
---------
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* fix(analyzer): honour language_model_params in BasicLangExtractRecognizer
Fixes#1942. Merge language_model_params into provider_kwargs so timeout,
num_ctx and other LLM provider params configured in YAML actually reach
the provider (e.g. Ollama) instead of being silently dropped.
Also fixes TypeError when kwargs: or language_model_params: is null in YAML.
- Merge language_model_params into provider_kwargs via setdefault (so
explicit kwargs: entries still win for backwards compatibility)
- Guard against null YAML values with 'or {}'
- Strengthen regression tests to assert params reach ModelConfig.provider_kwargs
- Add test for kwargs: null edge case
- Document fix in CHANGELOG
Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped)
* fix(analyzer): honour config_path for LangExtract recognizers in YAML registry
LM recognizers (BasicLangExtractRecognizer, AzureOpenAILangExtractRecognizer)
configured via a recognizer registry YAML silently ignored config_path: the
strict PredefinedRecognizerConfig schema has no config_path field and forbids
extras, so Pydantic dropped it and the recognizer fell back to its bundled
default model configuration.
Add a LangExtractRecognizerConfig model (extra=allow, explicit config_path
field) mirroring the existing HuggingFace/GLiNER configs, and register both
LM recognizer class names in CONFIG_MODEL_MAP so config_path (and other
recognizer-specific kwargs) survive validation and reach the constructor.
Adds regression tests covering config_path preservation via both the model
and the full ConfigurationValidator registry path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
The invalidate_result() sample-SSN blocklist was matched with
str.startswith(), and the literal "98765432" is only 8 digits (a
truncation of the canonical fake "987654320"). The prefix match
therefore invalidated the entire 987-65-4320 .. 987-65-4329 family --
ten distinct SSN-shaped values, including the widely-printed 987-65-4321
-- so real SSNs were dropped to no result and left unredacted.
Split the two concerns: keep 000/666 as a never-issued area-number
(first-group) check, and match the full 9-digit sample SSNs
(123456789 / 987654320 / 078051120) by exact equality so they no longer
over-block neighbours. NeMo Guardrails' PII rail inherits this
recognizer, so the leak reached that deployed guardrail too.
Co-authored-by: John Kearney <johndanielkearney@gmail.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
validate_result checked the date with datetime.strptime(date_part, "%d%m%y"),
which only sees the 2-digit year and maps it to the 2000s. The century is
actually carried by the 7th character (the separator), which strptime never
receives. So 29 Feb of a non-leap century is wrongly accepted: "290200+311B"
denotes 29 February 1800 ("+" = 1800s), but 1800 is not a leap year, yet
strptime validates it as 29 Feb 2000 (which is a leap year).
Resolve the century from the separator and validate the full date. Unknown
separators fall back to the previous strptime check, so this is independent of
the separator character-class handling.
Adds a regression case (290200+311B) that was wrongly accepted at score 1.0.
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Initial plan
* feat: add Python 3.14 compatibility to all remaining Presidio packages
- Update requires-python to <3.15 in anonymizer, image-redactor, cli,
structured, and umbrella packages
- Add Python 3.14 PyPI classifier to all affected packages
- Split spacy dependency markers in image-redactor to exclude 3.8.14
on Python 3.14 (no compatible wheel), matching analyzer pattern
- Extend CI matrix to test all components on Python 3.14
- Document Python 3.14 as a supported version in installation docs
- Add CHANGELOG entry under [unreleased]
Closes#2096
* ci: move Python 3.14 into main python-version matrix
All packages now support 3.14, so add it directly to the matrix array
instead of duplicating each component in `include` entries.
* build: regenerate uv.lock files to match updated requires-python bounds
* fix: merge duplicate Changed headings in CHANGELOG.md Unreleased section
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Fixes#1942. Merge language_model_params into provider_kwargs so timeout,
num_ctx and other LLM provider params configured in YAML actually reach
the provider (e.g. Ollama) instead of being silently dropped.
Also fixes TypeError when kwargs: or language_model_params: is null in YAML.
- Merge language_model_params into provider_kwargs via setdefault (so
explicit kwargs: entries still win for backwards compatibility)
- Guard against null YAML values with 'or {}'
- Strengthen regression tests to assert params reach ModelConfig.provider_kwargs
- Add test for kwargs: null edge case
- Document fix in CHANGELOG
Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped)
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* ci: use uv for test dependency installation + add timeout safety net
The analyzer test job runs `poetry install --all-extras` with no committed
lock, so Poetry performs a universal resolve of the full optional-dependencies
graph on every run. That resolve backtracks for many minutes — effectively
hanging — driven by the langextract extra's deep, loosely-bounded transitive
tree. Reproduced locally: Poetry >6 min (never completed); uv ~2s for the same
158-package resolution.
Switch the test jobs to uv:
- Install uv via astral-sh/setup-uv.
- Create a per-component venv with `uv venv --seed` (seed keeps pip for the
wheel-build step) and expose it via $GITHUB_PATH / $VIRTUAL_ENV.
- `uv pip install -e '.<extras>'` plus the test tooling that previously came
from the Poetry dev group (uv does not read that group).
- Add a job-level `timeout-minutes: 60` (and 30 on the install step) so a bad
resolve fails fast instead of hanging.
No package metadata changed — pyproject.toml is untouched; this only swaps the
CI install tooling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: run tests via uv run inside the uv-managed venv
Use "uv run --no-sync" for spaCy model downloads, pytest, and diff-cover
instead of exposing the venv on $GITHUB_PATH. "uv sync" builds the
project's .venv (resolving the analyzer --all-extras graph in seconds
where Poetry's lockless universal resolve hangs), and the subsequent
steps execute inside it idiomatically. The --all-extras selection stays
in the job matrix, unchanged from main. Wheel-build keeps using the
system interpreter, matching the previous Poetry behaviour.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: install uv via pip instead of the setup-uv action
The astral-sh/setup-uv action is not on the org's allowed-actions list,
so adding it made the CI workflow fail at startup (startup_failure)
before any job ran. Install a pinned uv from PyPI with the already
present setup-python instead; this needs no new third-party action and
keeps the resolver benefits that motivated the Poetry -> uv switch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: add pip to the uv venv for spaCy model downloads
spaCy's `download` command shells out to `python -m pip install`, but
uv-created virtualenvs ship without pip, so runtime/first-use model
downloads (e.g. the presidio-cli conftest fetching en_core_web_lg)
failed with SystemExit: 1. Install pip alongside the test tooling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: pre-install tomli for the wheel build on Python < 3.11
`python -m build` imports `tomli` on Python < 3.11 to read pyproject.toml,
but it is not in the hashed requirements-build.txt (pip-compiled on 3.11+
where tomllib is stdlib). Poetry previously pulled tomli onto the runner
as one of its own dependencies, so `pip install --require-hashes` found it
already satisfied. uv has no Python dependencies, so the 3.10 wheel-build
jobs began failing with "requirements must be pinned with ==". Pre-install
a marker-guarded tomli to restore that condition.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: build service Docker images with uv to fix the E2E resolve hang
The local-build-and-E2E job builds the analyzer/anonymizer/image-redactor
images via `docker compose build`. Each Dockerfile ran
`poetry install --no-root --only=main -E server`, which triggered the same
lockless Poetry backtracking that hangs the test jobs: the analyzer image
sat on "Resolving dependencies" for ~40 minutes until the job was
cancelled. Switch the dependency install to `uv pip install --system
-r pyproject.toml --extra server`, which resolves the same graph in
seconds (verified locally: the analyzer install layer completes in ~9s and
imports spacy/flask/gunicorn while correctly omitting the project itself,
matching --no-root). Drop the now-unused POETRY_VIRTUALENVS_CREATE env and
switch `poetry run python install_nlp_models.py` to a direct call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: run gunicorn directly in container entrypoints (drop poetry run)
The Docker images no longer install Poetry (deps come from uv), so the
`exec poetry run gunicorn ...` entrypoints crashed at startup with
"exec: poetry: not found". The analyzer/anonymizer/image-redactor
containers never came up, so every E2E test failed with connection
refused on ports 5002/5003. uv installs gunicorn system-wide, so invoke
it directly. Verified locally: the anonymizer container now starts
healthy and /health returns 200.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: keep uv's wheel cache on the /mnt ephemeral disk
Restore the intent of the old POETRY_CACHE_DIR=/mnt/poetry_cache setup for
uv: point UV_CACHE_DIR at the runner's large ephemeral disk (/mnt, ~65 GB)
so the heavy analyzer --all-extras wheel cache does not fill the smaller
root volume. /mnt is root-owned, so a prep step creates the directory and
hands it to the runner user. UV_LINK_MODE=copy silences the cross-filesystem
hardlink warning, since the project venv lives on the root volume under the
checkout while the cache is on /mnt.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: broaden CHANGELOG/CI comment to cover the Docker image changes
Address review feedback that the PR scope grew beyond the test jobs: the
CHANGELOG now notes the analyzer/anonymizer/image-redactor image builds
and entrypoints also moved from Poetry to uv, and clarifies pyproject.toml
metadata is unchanged. Also clarify the ci.yml install-step comment that
the wheel-build step deliberately uses the system interpreter (not uv run).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* build: adopt PEP 735 dependency groups and commit uv.lock files
Migrate each CI-tested package's development tooling from Poetry's
[tool.poetry.group.dev.dependencies] to the standard PEP 735
[dependency-groups] table, and commit a uv.lock per package. This makes
pyproject.toml the single source of truth for dev tooling and lets CI and
Docker install a pinned, reproducible dependency graph instead of resolving
on every run. The [project] metadata and the poetry-core build backend are
unchanged, so wheels/sdists are byte-for-byte identical.
Covers presidio-analyzer, presidio-anonymizer, presidio-cli,
presidio-image-redactor, presidio-structured.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: install from the committed uv.lock in CI and Docker builds
CI now runs `uv sync --locked --group dev`, which installs exactly the
locked graph and fails if pyproject.toml and uv.lock disagree rather than
silently resolving new versions. The dev group provides the test tooling
(and pip, which spaCy's model download shells out to), so the manual
`uv pip install pytest ...` line is gone. The uv cache is persisted across
runs via actions/cache keyed on uv.lock and trimmed with `uv cache prune
--ci`.
The analyzer/anonymizer/image-redactor images consume the same lockfile
with `uv sync --locked --no-default-groups --extra server
--no-install-project` into UV_PROJECT_ENVIRONMENT=/usr/local, guaranteeing
images use the same dependency graph as CI. Splitting the dependency layer
from the source COPY keeps it cached until pyproject/uv.lock change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: allow pre-existing transformers advisories in dependency review
The newly committed uv.lock files pin exact versions, so dependency-review
now sees two transformers advisories that already exist on main (capped at
transformers <5 by spacy-huggingface-pipelines and tracked as Dependabot
alerts). They are unrelated to the Poetry->uv migration and will be
addressed in separate security PRs, so allow-list the two GHSAs here to
keep this PR's dependency review unblocked.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: require uv.lock review and commits alongside pyproject changes
Address review feedback from @SharonHart:
- CODEOWNERS: require presidio-administrators approval for **/uv.lock, the
same as **/pyproject.toml, so lockfile changes get dependency review.
- copilot-instructions: switch the local dev commands to uv and add an
explicit rule that editing pyproject.toml dependencies requires
regenerating and committing that package's uv.lock in the same change,
since CI installs with `uv sync --locked` and fails on drift.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: reference UV_CACHE_DIR in cache path; align dev-docs to uv
Address review feedback:
- ci.yml: use ${{ env.UV_CACHE_DIR }} for the cache action path instead of
hardcoding /mnt/uv-cache, so the cache and uv stay in sync if the dir
changes.
- copilot-instructions: update the "Technology Stack" bullet to name uv as
the dependency/install tool (poetry-core retained only as build backend),
matching the uv-based dev commands already documented.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* ci: shorten workflow comments; move transformers allow-ghsas out
Trim the verbose explanatory comments in the test job to 1-3 lines each,
and drop the dependency-review `allow-ghsas` entry for the transformers
advisories. That suppression is a security concern rather than part of the
Poetry->uv migration, so it moves to the stacked security PR alongside the
.trivyignore.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* ci: drop stale pip cache; trim CHANGELOG entry
Remove `cache: 'pip'` (keyed on pyproject.toml) from the test job's
setup-python: it is leftover Poetry-era config. Project dependencies now use
the uv cache, and the only remaining pip installs (uv, tomli, build tools)
are tiny and unrelated to pyproject.toml. Also condense the unreleased
CHANGELOG entry to a single concise line; the implementation detail lives in
the PR description.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* small commit to trigger the pipeline again.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
isdigit was referenced without being called, so the digit guard was
always truthy and never actually checked anything. Under the
recognizer's default pattern this stays hidden since the middle
characters are already constrained to digits, but a custom patterns
argument (which the constructor explicitly supports) can let non-digit
text reach the checksum line further down and raise a ValueError
instead of returning False.
Fixes#2145
* Fix ISO 8601 date pattern accepting impossible month/day values
The "ISO 8601 datetime" pattern in DateRecognizer used `[01]\d` for the
month and `[0-3]\d` for the day. These ranges admit impossible values:
month `00` and `13`-`19`, and day `00` and `32`-`39`. As a result
strings such as `2024-13-15T14:30:00Z` and `2024-12-32T14:30Z` were
detected as DATE_TIME.
Every other date pattern in this same file already constrains the
month to `01`-`12` and the day to `01`-`31`; only the ISO 8601 pattern
was loose. Tighten the ISO month/day fields to match (using
non-capturing groups so existing capture-group positions are
unaffected). No valid ISO 8601 datetime is lost, since those values are
not valid dates to begin with.
Adds parametrized cases for invalid month (00, 13) and day (00, 32).
* Address review: apply word boundary across all alternatives, rename pattern
Two review points from #2113:
1. `|` has lower precedence than concatenation, so the pattern
`\b A | B | C \b` was parsed as `(\b A) | B | (C \b)`. The leading
`\b` only guarded the first alternative (the full-fractional form)
and the trailing `\b` only guarded the last (the minutes-only form).
The seconds-only alternative in the middle had no word-boundary
anchor at all, so a valid seconds/minutes datetime could match
mid-word (e.g. `Today is2024-03-15T14:30:00+02:00`). Wrap the
alternation in a non-capturing group so both `\b` anchors apply to
every alternative.
2. Rename the pattern from "ISO 8601 datetime" to
"Datetime (yyyy-mm-ddThh:mm[:ss[.f]] with timezone)" — the pattern
doesn't fully validate ISO 8601 (e.g. the hour field admits 24–29).
The new name honestly describes the shape it accepts.
---------
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
The class docstring gives "B0" (Berlin) as an example of a
Behördenkürzel and lists `B012345678A` in the Examples line, but the
regex `[A-Z]{2}\d{8}[A-Z0-9]` requires both leading characters to be
letters, so `B012345678A` never matches. The test file's docstring is
the authoritative one — it explicitly notes that single-letter Kfz
codes are used in 2-letter authority forms (e.g. `BO`, `KN`) and that
`B0`-style single-letter-plus-digit combinations are out of scope, and
`test_when_all_de_fuehrerschein_numbers_then_succeed` asserts
`B12345678A` is rejected.
Update the class docstring to match the tests: replace the `"B0"
Berlin` example with `"BO" Bochum`, add a short clarification about
single-letter Kfz codes, and change the misleading `B012345678A`
Examples entry to `BO12345678A`.
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
The ABN checksum remapped a leading-zero first digit to 9
(`abn_list[0] = 9 if abn_list[0] == 0 else abn_list[0] - 1`). The official ABR
algorithm simply subtracts 1 from the first (left-most) digit, so a leading 0
becomes -1. Remapping it to 9 instead shifts the weighted sum by 100 (= 11 mod
89) and admits some invalid 11-digit numbers beginning with 0 as valid ABNs —
a false positive (e.g. 00000000560). Valid ABNs never start with 0.
Use the literal subtract-1. This is identical for first digits 1-9, so no
valid ABN is affected; only invalid leading-zero numbers, which the official
algorithm rejects, are now correctly rejected.
Adds a regression case (00000000560) that was wrongly accepted before.
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
