mirror of
https://github.com/data-privacy-stack/presidio.git
synced 2026-09-21 13:38:05 -05:00
sharonhart-presidio-image-hardening
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ron Shakutai
|
4d6b108f06 | ci: restrict release workflow to main (#2187) | ||
|
|
b2680be771 |
fix(analyzer): honour config_path for LangExtract recognizers in YAML registry (#2157)
* fix(analyzer): honour language_model_params in BasicLangExtractRecognizer Fixes #1942. Merge language_model_params into provider_kwargs so timeout, num_ctx and other LLM provider params configured in YAML actually reach the provider (e.g. Ollama) instead of being silently dropped. Also fixes TypeError when kwargs: or language_model_params: is null in YAML. - Merge language_model_params into provider_kwargs via setdefault (so explicit kwargs: entries still win for backwards compatibility) - Guard against null YAML values with 'or {}' - Strengthen regression tests to assert params reach ModelConfig.provider_kwargs - Add test for kwargs: null edge case - Document fix in CHANGELOG Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped) * fix(analyzer): honour config_path for LangExtract recognizers in YAML registry LM recognizers (BasicLangExtractRecognizer, AzureOpenAILangExtractRecognizer) configured via a recognizer registry YAML silently ignored config_path: the strict PredefinedRecognizerConfig schema has no config_path field and forbids extras, so Pydantic dropped it and the recognizer fell back to its bundled default model configuration. Add a LangExtractRecognizerConfig model (extra=allow, explicit config_path field) mirroring the existing HuggingFace/GLiNER configs, and register both LM recognizer class names in CONFIG_MODEL_MAP so config_path (and other recognizer-specific kwargs) survive validation and reach the constructor. Adds regression tests covering config_path preservation via both the model and the full ConfigurationValidator registry path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> |
||
|
Ron ShakutaiandSharon Hart
|
08d5f18b8c |
fix(analyzer): honour language_model_params in BasicLangExtractRecognizer (#2150)
Fixes #1942. Merge language_model_params into provider_kwargs so timeout, num_ctx and other LLM provider params configured in YAML actually reach the provider (e.g. Ollama) instead of being silently dropped. Also fixes TypeError when kwargs: or language_model_params: is null in YAML. - Merge language_model_params into provider_kwargs via setdefault (so explicit kwargs: entries still win for backwards compatibility) - Guard against null YAML values with 'or {}' - Strengthen regression tests to assert params reach ModelConfig.provider_kwargs - Add test for kwargs: null edge case - Document fix in CHANGELOG Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped) Co-authored-by: Sharon Hart <sharonh.dev@gmail.com> |
||
|
|
8b7877ccd0 |
build: migrate CI and service images from Poetry to uv (#2152)
* ci: use uv for test dependency installation + add timeout safety net The analyzer test job runs `poetry install --all-extras` with no committed lock, so Poetry performs a universal resolve of the full optional-dependencies graph on every run. That resolve backtracks for many minutes — effectively hanging — driven by the langextract extra's deep, loosely-bounded transitive tree. Reproduced locally: Poetry >6 min (never completed); uv ~2s for the same 158-package resolution. Switch the test jobs to uv: - Install uv via astral-sh/setup-uv. - Create a per-component venv with `uv venv --seed` (seed keeps pip for the wheel-build step) and expose it via $GITHUB_PATH / $VIRTUAL_ENV. - `uv pip install -e '.<extras>'` plus the test tooling that previously came from the Poetry dev group (uv does not read that group). - Add a job-level `timeout-minutes: 60` (and 30 on the install step) so a bad resolve fails fast instead of hanging. No package metadata changed — pyproject.toml is untouched; this only swaps the CI install tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: run tests via uv run inside the uv-managed venv Use "uv run --no-sync" for spaCy model downloads, pytest, and diff-cover instead of exposing the venv on $GITHUB_PATH. "uv sync" builds the project's .venv (resolving the analyzer --all-extras graph in seconds where Poetry's lockless universal resolve hangs), and the subsequent steps execute inside it idiomatically. The --all-extras selection stays in the job matrix, unchanged from main. Wheel-build keeps using the system interpreter, matching the previous Poetry behaviour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: install uv via pip instead of the setup-uv action The astral-sh/setup-uv action is not on the org's allowed-actions list, so adding it made the CI workflow fail at startup (startup_failure) before any job ran. Install a pinned uv from PyPI with the already present setup-python instead; this needs no new third-party action and keeps the resolver benefits that motivated the Poetry -> uv switch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add pip to the uv venv for spaCy model downloads spaCy's `download` command shells out to `python -m pip install`, but uv-created virtualenvs ship without pip, so runtime/first-use model downloads (e.g. the presidio-cli conftest fetching en_core_web_lg) failed with SystemExit: 1. Install pip alongside the test tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: pre-install tomli for the wheel build on Python < 3.11 `python -m build` imports `tomli` on Python < 3.11 to read pyproject.toml, but it is not in the hashed requirements-build.txt (pip-compiled on 3.11+ where tomllib is stdlib). Poetry previously pulled tomli onto the runner as one of its own dependencies, so `pip install --require-hashes` found it already satisfied. uv has no Python dependencies, so the 3.10 wheel-build jobs began failing with "requirements must be pinned with ==". Pre-install a marker-guarded tomli to restore that condition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: build service Docker images with uv to fix the E2E resolve hang The local-build-and-E2E job builds the analyzer/anonymizer/image-redactor images via `docker compose build`. Each Dockerfile ran `poetry install --no-root --only=main -E server`, which triggered the same lockless Poetry backtracking that hangs the test jobs: the analyzer image sat on "Resolving dependencies" for ~40 minutes until the job was cancelled. Switch the dependency install to `uv pip install --system -r pyproject.toml --extra server`, which resolves the same graph in seconds (verified locally: the analyzer install layer completes in ~9s and imports spacy/flask/gunicorn while correctly omitting the project itself, matching --no-root). Drop the now-unused POETRY_VIRTUALENVS_CREATE env and switch `poetry run python install_nlp_models.py` to a direct call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: run gunicorn directly in container entrypoints (drop poetry run) The Docker images no longer install Poetry (deps come from uv), so the `exec poetry run gunicorn ...` entrypoints crashed at startup with "exec: poetry: not found". The analyzer/anonymizer/image-redactor containers never came up, so every E2E test failed with connection refused on ports 5002/5003. uv installs gunicorn system-wide, so invoke it directly. Verified locally: the anonymizer container now starts healthy and /health returns 200. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: keep uv's wheel cache on the /mnt ephemeral disk Restore the intent of the old POETRY_CACHE_DIR=/mnt/poetry_cache setup for uv: point UV_CACHE_DIR at the runner's large ephemeral disk (/mnt, ~65 GB) so the heavy analyzer --all-extras wheel cache does not fill the smaller root volume. /mnt is root-owned, so a prep step creates the directory and hands it to the runner user. UV_LINK_MODE=copy silences the cross-filesystem hardlink warning, since the project venv lives on the root volume under the checkout while the cache is on /mnt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: broaden CHANGELOG/CI comment to cover the Docker image changes Address review feedback that the PR scope grew beyond the test jobs: the CHANGELOG now notes the analyzer/anonymizer/image-redactor image builds and entrypoints also moved from Poetry to uv, and clarifies pyproject.toml metadata is unchanged. Also clarify the ci.yml install-step comment that the wheel-build step deliberately uses the system interpreter (not uv run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: adopt PEP 735 dependency groups and commit uv.lock files Migrate each CI-tested package's development tooling from Poetry's [tool.poetry.group.dev.dependencies] to the standard PEP 735 [dependency-groups] table, and commit a uv.lock per package. This makes pyproject.toml the single source of truth for dev tooling and lets CI and Docker install a pinned, reproducible dependency graph instead of resolving on every run. The [project] metadata and the poetry-core build backend are unchanged, so wheels/sdists are byte-for-byte identical. Covers presidio-analyzer, presidio-anonymizer, presidio-cli, presidio-image-redactor, presidio-structured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: install from the committed uv.lock in CI and Docker builds CI now runs `uv sync --locked --group dev`, which installs exactly the locked graph and fails if pyproject.toml and uv.lock disagree rather than silently resolving new versions. The dev group provides the test tooling (and pip, which spaCy's model download shells out to), so the manual `uv pip install pytest ...` line is gone. The uv cache is persisted across runs via actions/cache keyed on uv.lock and trimmed with `uv cache prune --ci`. The analyzer/anonymizer/image-redactor images consume the same lockfile with `uv sync --locked --no-default-groups --extra server --no-install-project` into UV_PROJECT_ENVIRONMENT=/usr/local, guaranteeing images use the same dependency graph as CI. Splitting the dependency layer from the source COPY keeps it cached until pyproject/uv.lock change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: allow pre-existing transformers advisories in dependency review The newly committed uv.lock files pin exact versions, so dependency-review now sees two transformers advisories that already exist on main (capped at transformers <5 by spacy-huggingface-pipelines and tracked as Dependabot alerts). They are unrelated to the Poetry->uv migration and will be addressed in separate security PRs, so allow-list the two GHSAs here to keep this PR's dependency review unblocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: require uv.lock review and commits alongside pyproject changes Address review feedback from @SharonHart: - CODEOWNERS: require presidio-administrators approval for **/uv.lock, the same as **/pyproject.toml, so lockfile changes get dependency review. - copilot-instructions: switch the local dev commands to uv and add an explicit rule that editing pyproject.toml dependencies requires regenerating and committing that package's uv.lock in the same change, since CI installs with `uv sync --locked` and fails on drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: reference UV_CACHE_DIR in cache path; align dev-docs to uv Address review feedback: - ci.yml: use ${{ env.UV_CACHE_DIR }} for the cache action path instead of hardcoding /mnt/uv-cache, so the cache and uv stay in sync if the dir changes. - copilot-instructions: update the "Technology Stack" bullet to name uv as the dependency/install tool (poetry-core retained only as build backend), matching the uv-based dev commands already documented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * ci: shorten workflow comments; move transformers allow-ghsas out Trim the verbose explanatory comments in the test job to 1-3 lines each, and drop the dependency-review `allow-ghsas` entry for the transformers advisories. That suppression is a security concern rather than part of the Poetry->uv migration, so it moves to the stacked security PR alongside the .trivyignore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: drop stale pip cache; trim CHANGELOG entry Remove `cache: 'pip'` (keyed on pyproject.toml) from the test job's setup-python: it is leftover Poetry-era config. Project dependencies now use the uv cache, and the only remaining pip installs (uv, tomli, build tools) are tiny and unrelated to pyproject.toml. Also condense the unreleased CHANGELOG entry to a single concise line; the implementation detail lives in the PR description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * small commit to trigger the pipeline again. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Ron Shakutai
|
1567e5d892 |
Fix: config path resolution for PyPI installs (langextract recognizer) (#1917)
* init commit * test: add tests for resolve_config_path handling PyPI install * dev mode supposed to be broken * feat: enhance resolve_config_path to support package conf directory |
||
|
Ron ShakutaiandOmri Mendels
|
4ad4f19478 |
Feature - GPU Device Control via Environment Variable (#1844)
* init commit * gpu on windows * delete benchmark script and results file for Presidio Analyzer * ruff fix --------- Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> |
||
|
Ron Shakutai
|
7850688de9 |
Refactor: lazy initialization for device_detector singleton (#1831)
* Refactor: implement lazy initialization for device_detector instance * Fix: improve device detection logic in device_detector and update TransformersRecognizer to use it * doube lock added |
||
|
Ron Shakutai
|
1fd62dc5de |
Feat: add class_name to allow multiple recognizers from same class (#1819)
* fix: Rename method to get_recognizer_class_name for clarity and update usage * fix: Clarify comments regarding excluded recognizer attributes in RecognizerListLoader * feat: Add class_name parameter to BaseRecognizerConfig for improved recognizer identification * fix: Include 'class_name' in custom recognizers exclusion list for improved configuration handling * feat: Enhance Ollama recognizer to support custom instance names and update configuration handling * Enhance recognizers to accept additional keyword arguments - Updated various recognizers across different countries (India, Italy, Korea, Poland, Singapore, Spain, Thailand, UK, US) to accept **kwargs in their constructors. - This change allows for more flexible configuration of recognizers without modifying their signatures. - Adjusted the recognizer loading mechanism to handle the new **kwargs parameter appropriately. * refactor: Simplify Ollama recognizer loading verification and assertions * test: Update Ollama recognizer loading verification to ensure single instance retrieval * feat: Enhance recognizer class name logic in RecognizerListLoader * Refactor recognizers to explicitly handle 'name' parameter in __init__ methods - Updated various recognizers across different countries (Italy, Korea, Poland, Singapore, Spain, Thailand, UK, US) to include an optional 'name' parameter in their constructors. - Adjusted super() calls to pass the 'name' parameter appropriately. - Ensured that the 'Optional' type is imported where necessary. - Added a script to automate the updates for recognizers that were missing the 'name' parameter. * fix: Update Stanza and Transformers recognizers to handle additional kwargs in __init__ methods * fix: Correct the import order for constants in methods.py * refactor: Remove update_recognizers_name.py script as its functionality is no longer needed * check * fix: Remove unnecessary comments and clean up recognizer configuration code * Refactor recognizer constructors to remove unused **kwargs parameter - Updated multiple recognizer classes across various countries (Australia, Finland, India, Italy, Korea, Poland, Singapore, Spain, Thailand, UK, US) to remove the **kwargs parameter from their constructors. - Simplified constructor signatures for better clarity and maintainability. * refactor: Remove unused **kwargs parameter from recognizer initializers * refactor: Remove unused **kwargs parameter from recognizer constructors * fix ci * refactor: format parameters in recognizer constructors for consistency * refactor: format parameters in recognizer constructors for consistency |
||
|
Ron Shakutai
|
8eff3198fe |
GPU optimizations (#1812)
* Gpu first draft * Gliner flow adapted to GPU * deleted testing files * fix: Suppress warnings from spacy_huggingface_pipelines and other libraries * stanza mode added. * ruff issues * adding to to the mock * refactor: remove unused GPU result files and update device detector usage * fix: reorder device_detector import for consistency * docs added around gpu * test: enhance GPU detection tests for DeviceDetector and SpacyNlpEngine * Add comprehensive error path mocking for device_detector and spacy GPU config tests * remove benchmark script. * Enhance device detection to support MPS and update related tests and configurations * Refactor device detection logic to improve clarity and logging for MPS initialization failures * Remove unnecessary comments regarding pipeline configuration in TransformersNlpEngine * Gpu first draft * Gliner flow adapted to GPU * deleted testing files * fix: Suppress warnings from spacy_huggingface_pipelines and other libraries * stanza mode added. * ruff issues * adding to to the mock * refactor: remove unused GPU result files and update device detector usage * fix: reorder device_detector import for consistency * docs added around gpu * test: enhance GPU detection tests for DeviceDetector and SpacyNlpEngine * Add comprehensive error path mocking for device_detector and spacy GPU config tests * remove benchmark script. * Enhance device detection to support MPS and update related tests and configurations * Refactor device detection logic to improve clarity and logging for MPS initialization failures * Remove unnecessary comments regarding pipeline configuration in TransformersNlpEngine * Update GPU acceleration instructions and remove deprecated dependencies * cleaning logs and comments * Enable GPU support for spaCy and transformers in NLP engines * Refactor GPU error logging in SpacyNlpEngine and remove unused import in TransformersNlpEngine * Update documentation and code to reflect MPS (Metal Performance Shaders) support is not available on macOS. Remove references to MPS in device detection and tests. |
||
|
|
2ac7a320ce |
Change parameters in extraction in langextract (#1811)
* Refactor LangExtract recognizers to support customizable extraction parameters and improve configuration clarity for Azure OpenAI and Ollama. * Add tests for parameter configuration and overrides in Ollama and Azure OpenAI recognizers * Refactor Azure OpenAI and Ollama LangExtract recognizers to improve parameter handling and documentation clarity * Update max_char_buffer parameter in OllamaLangExtractRecognizer to improve performance * Update max_char_buffer parameter in Ollama configuration to enhance performance * Update presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/langextract_recognizer.py Co-authored-by: Dor Lugasi-Gal <dorlugasigal@gmail.com> * lx to top level --------- Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: Dor Lugasi-Gal <dorlugasigal@gmail.com> |