mirror of
https://github.com/data-privacy-stack/presidio.git
synced 2026-07-23 11:20:55 -05:00
8b7877ccd0
* ci: use uv for test dependency installation + add timeout safety net The analyzer test job runs `poetry install --all-extras` with no committed lock, so Poetry performs a universal resolve of the full optional-dependencies graph on every run. That resolve backtracks for many minutes — effectively hanging — driven by the langextract extra's deep, loosely-bounded transitive tree. Reproduced locally: Poetry >6 min (never completed); uv ~2s for the same 158-package resolution. Switch the test jobs to uv: - Install uv via astral-sh/setup-uv. - Create a per-component venv with `uv venv --seed` (seed keeps pip for the wheel-build step) and expose it via $GITHUB_PATH / $VIRTUAL_ENV. - `uv pip install -e '.<extras>'` plus the test tooling that previously came from the Poetry dev group (uv does not read that group). - Add a job-level `timeout-minutes: 60` (and 30 on the install step) so a bad resolve fails fast instead of hanging. No package metadata changed — pyproject.toml is untouched; this only swaps the CI install tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: run tests via uv run inside the uv-managed venv Use "uv run --no-sync" for spaCy model downloads, pytest, and diff-cover instead of exposing the venv on $GITHUB_PATH. "uv sync" builds the project's .venv (resolving the analyzer --all-extras graph in seconds where Poetry's lockless universal resolve hangs), and the subsequent steps execute inside it idiomatically. The --all-extras selection stays in the job matrix, unchanged from main. Wheel-build keeps using the system interpreter, matching the previous Poetry behaviour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: install uv via pip instead of the setup-uv action The astral-sh/setup-uv action is not on the org's allowed-actions list, so adding it made the CI workflow fail at startup (startup_failure) before any job ran. Install a pinned uv from PyPI with the already present setup-python instead; this needs no new third-party action and keeps the resolver benefits that motivated the Poetry -> uv switch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add pip to the uv venv for spaCy model downloads spaCy's `download` command shells out to `python -m pip install`, but uv-created virtualenvs ship without pip, so runtime/first-use model downloads (e.g. the presidio-cli conftest fetching en_core_web_lg) failed with SystemExit: 1. Install pip alongside the test tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: pre-install tomli for the wheel build on Python < 3.11 `python -m build` imports `tomli` on Python < 3.11 to read pyproject.toml, but it is not in the hashed requirements-build.txt (pip-compiled on 3.11+ where tomllib is stdlib). Poetry previously pulled tomli onto the runner as one of its own dependencies, so `pip install --require-hashes` found it already satisfied. uv has no Python dependencies, so the 3.10 wheel-build jobs began failing with "requirements must be pinned with ==". Pre-install a marker-guarded tomli to restore that condition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: build service Docker images with uv to fix the E2E resolve hang The local-build-and-E2E job builds the analyzer/anonymizer/image-redactor images via `docker compose build`. Each Dockerfile ran `poetry install --no-root --only=main -E server`, which triggered the same lockless Poetry backtracking that hangs the test jobs: the analyzer image sat on "Resolving dependencies" for ~40 minutes until the job was cancelled. Switch the dependency install to `uv pip install --system -r pyproject.toml --extra server`, which resolves the same graph in seconds (verified locally: the analyzer install layer completes in ~9s and imports spacy/flask/gunicorn while correctly omitting the project itself, matching --no-root). Drop the now-unused POETRY_VIRTUALENVS_CREATE env and switch `poetry run python install_nlp_models.py` to a direct call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: run gunicorn directly in container entrypoints (drop poetry run) The Docker images no longer install Poetry (deps come from uv), so the `exec poetry run gunicorn ...` entrypoints crashed at startup with "exec: poetry: not found". The analyzer/anonymizer/image-redactor containers never came up, so every E2E test failed with connection refused on ports 5002/5003. uv installs gunicorn system-wide, so invoke it directly. Verified locally: the anonymizer container now starts healthy and /health returns 200. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: keep uv's wheel cache on the /mnt ephemeral disk Restore the intent of the old POETRY_CACHE_DIR=/mnt/poetry_cache setup for uv: point UV_CACHE_DIR at the runner's large ephemeral disk (/mnt, ~65 GB) so the heavy analyzer --all-extras wheel cache does not fill the smaller root volume. /mnt is root-owned, so a prep step creates the directory and hands it to the runner user. UV_LINK_MODE=copy silences the cross-filesystem hardlink warning, since the project venv lives on the root volume under the checkout while the cache is on /mnt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: broaden CHANGELOG/CI comment to cover the Docker image changes Address review feedback that the PR scope grew beyond the test jobs: the CHANGELOG now notes the analyzer/anonymizer/image-redactor image builds and entrypoints also moved from Poetry to uv, and clarifies pyproject.toml metadata is unchanged. Also clarify the ci.yml install-step comment that the wheel-build step deliberately uses the system interpreter (not uv run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: adopt PEP 735 dependency groups and commit uv.lock files Migrate each CI-tested package's development tooling from Poetry's [tool.poetry.group.dev.dependencies] to the standard PEP 735 [dependency-groups] table, and commit a uv.lock per package. This makes pyproject.toml the single source of truth for dev tooling and lets CI and Docker install a pinned, reproducible dependency graph instead of resolving on every run. The [project] metadata and the poetry-core build backend are unchanged, so wheels/sdists are byte-for-byte identical. Covers presidio-analyzer, presidio-anonymizer, presidio-cli, presidio-image-redactor, presidio-structured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: install from the committed uv.lock in CI and Docker builds CI now runs `uv sync --locked --group dev`, which installs exactly the locked graph and fails if pyproject.toml and uv.lock disagree rather than silently resolving new versions. The dev group provides the test tooling (and pip, which spaCy's model download shells out to), so the manual `uv pip install pytest ...` line is gone. The uv cache is persisted across runs via actions/cache keyed on uv.lock and trimmed with `uv cache prune --ci`. The analyzer/anonymizer/image-redactor images consume the same lockfile with `uv sync --locked --no-default-groups --extra server --no-install-project` into UV_PROJECT_ENVIRONMENT=/usr/local, guaranteeing images use the same dependency graph as CI. Splitting the dependency layer from the source COPY keeps it cached until pyproject/uv.lock change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: allow pre-existing transformers advisories in dependency review The newly committed uv.lock files pin exact versions, so dependency-review now sees two transformers advisories that already exist on main (capped at transformers <5 by spacy-huggingface-pipelines and tracked as Dependabot alerts). They are unrelated to the Poetry->uv migration and will be addressed in separate security PRs, so allow-list the two GHSAs here to keep this PR's dependency review unblocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: require uv.lock review and commits alongside pyproject changes Address review feedback from @SharonHart: - CODEOWNERS: require presidio-administrators approval for **/uv.lock, the same as **/pyproject.toml, so lockfile changes get dependency review. - copilot-instructions: switch the local dev commands to uv and add an explicit rule that editing pyproject.toml dependencies requires regenerating and committing that package's uv.lock in the same change, since CI installs with `uv sync --locked` and fails on drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: reference UV_CACHE_DIR in cache path; align dev-docs to uv Address review feedback: - ci.yml: use ${{ env.UV_CACHE_DIR }} for the cache action path instead of hardcoding /mnt/uv-cache, so the cache and uv stay in sync if the dir changes. - copilot-instructions: update the "Technology Stack" bullet to name uv as the dependency/install tool (poetry-core retained only as build backend), matching the uv-based dev commands already documented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * ci: shorten workflow comments; move transformers allow-ghsas out Trim the verbose explanatory comments in the test job to 1-3 lines each, and drop the dependency-review `allow-ghsas` entry for the transformers advisories. That suppression is a security concern rather than part of the Poetry->uv migration, so it moves to the stacked security PR alongside the .trivyignore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: drop stale pip cache; trim CHANGELOG entry Remove `cache: 'pip'` (keyed on pyproject.toml) from the test job's setup-python: it is leftover Poetry-era config. Project dependencies now use the uv cache, and the only remaining pip installs (uv, tomli, build tools) are tiny and unrelated to pyproject.toml. Also condense the unreleased CHANGELOG entry to a single concise line; the implementation detail lives in the PR description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * small commit to trigger the pipeline again. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
48 lines
1.6 KiB
Docker
48 lines
1.6 KiB
Docker
FROM python:3.13.13-slim@sha256:2ba73a4dc380f21137fc75296abfa2add90b51fd10b609ce530b40cc097269b1
|
|
|
|
ARG NLP_CONF_FILE
|
|
ARG ANALYZER_CONF_FILE
|
|
ARG RECOGNIZER_REGISTRY_CONF_FILE
|
|
ENV PIP_NO_CACHE_DIR=1
|
|
# Install the locked dependencies into the system environment (no venv).
|
|
ENV UV_PROJECT_ENVIRONMENT=/usr/local
|
|
|
|
ENV ANALYZER_CONF_FILE=${ANALYZER_CONF_FILE}
|
|
ENV RECOGNIZER_REGISTRY_CONF_FILE=${RECOGNIZER_REGISTRY_CONF_FILE}
|
|
ENV NLP_CONF_FILE=${NLP_CONF_FILE}
|
|
|
|
ENV PORT=3000
|
|
ENV WORKERS=1
|
|
|
|
COPY ${ANALYZER_CONF_FILE} /app/${ANALYZER_CONF_FILE}
|
|
COPY ${RECOGNIZER_REGISTRY_CONF_FILE} /app/${RECOGNIZER_REGISTRY_CONF_FILE}
|
|
COPY ${NLP_CONF_FILE} /app/${NLP_CONF_FILE}
|
|
|
|
WORKDIR /app
|
|
|
|
RUN apt-get update \
|
|
&& apt-get install build-essential tesseract-ocr ffmpeg libsm6 libxext6 curl --no-install-recommends -y \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& tesseract -v
|
|
|
|
COPY ./pyproject.toml ./uv.lock /app/
|
|
# Install exactly the locked dependency graph (main + server extra, no dev
|
|
# group); the project itself is run from the copied source, not installed.
|
|
RUN pip install uv==0.11.6 \
|
|
&& uv sync --locked --no-cache --no-default-groups --extra server --no-install-project
|
|
|
|
# Install spaCy model during build (as root) so it's available to non-root user at runtime
|
|
RUN python -m spacy download en_core_web_lg
|
|
|
|
COPY . /app/
|
|
|
|
# Create a non-root user and set ownership
|
|
RUN useradd -m -u 1001 presidio && chown -R presidio:presidio /app
|
|
|
|
USER 1001
|
|
|
|
EXPOSE ${PORT}
|
|
HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 \
|
|
CMD curl -f http://localhost:${PORT}/health || exit 1
|
|
CMD ["./entrypoint.sh"]
|