mirror of
https://github.com/data-privacy-stack/presidio.git
synced 2026-07-23 11:20:55 -05:00
Migrate to ghcr packages (#2103)
* Migrate container publishing to GHCR Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add GHCR transition release metadata Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove release image labels Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert release note automation change Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Inline Compose GHCR defaults Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant Compose env file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use explicit GHCR Compose images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CI image labels Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore local development Compose defaults Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CI Compose override Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unused Compose variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove SBOM and provenance attestations from CI build step * Restore SBOM and provenance attestations in CI build step * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
This commit is contained in:
Sharon Hart
6
.github/copilot-instructions.md
vendored
6
.github/copilot-instructions.md
vendored
@@ -573,7 +573,7 @@ Use atomic grouping: (?>a+)b or possessive quantifier a++b"
|
||||
- **Poetry** - Package manager, not pip
|
||||
- **Ruff** - Linting and formatting (replaces flake8, black, isort)
|
||||
- **spaCy** - Default NLP engine (en_core_web_lg for production), although one can use other NLP engines via provider pattern
|
||||
- **Docker** - Deployment via mcr.microsoft.com registry
|
||||
- **Docker** - Deployment via GitHub Container Registry (`ghcr.io/data-privacy-stack`)
|
||||
|
||||
|
||||
### Critical Files for Cross-Component Changes
|
||||
@@ -604,8 +604,8 @@ ruff format . # Auto-format
|
||||
### Docker Testing
|
||||
```bash
|
||||
# Quick test with pre-built images
|
||||
docker pull mcr.microsoft.com/presidio-analyzer:latest
|
||||
docker run -d -p 5002:3000 --name analyzer mcr.microsoft.com/presidio-analyzer:latest
|
||||
docker pull ghcr.io/data-privacy-stack/presidio-analyzer:latest
|
||||
docker run -d -p 5002:3000 --name analyzer ghcr.io/data-privacy-stack/presidio-analyzer:latest
|
||||
curl http://localhost:5002/health
|
||||
|
||||
# Full build from source (takes 15+ minutes)
|
||||
|
||||
66
.github/workflows/ci.yml
vendored
66
.github/workflows/ci.yml
vendored
@@ -14,7 +14,7 @@ concurrency:
|
||||
permissions: read-all
|
||||
|
||||
env:
|
||||
REGISTRY_NAME: ${{ secrets.ACR_NAME }}.azurecr.io
|
||||
REGISTRY_NAME: ghcr.io/data-privacy-stack
|
||||
TAG: gha${{ github.run_number }}
|
||||
|
||||
jobs:
|
||||
@@ -189,8 +189,8 @@ jobs:
|
||||
needs: [lint]
|
||||
if: github.ref == 'refs/heads/main'
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
packages: write
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
@@ -218,15 +218,11 @@ jobs:
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Azure Login using OIDC
|
||||
uses: azure/login@532459ea530d8321f2fb9bb10d1e0bcf23869a43 # v3.0.0
|
||||
with:
|
||||
client-id: ${{ secrets.AZURE_CLIENT_ID }}
|
||||
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
|
||||
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
|
||||
|
||||
- name: Log in to Azure Container Registry
|
||||
run: az acr login --name ${{ secrets.ACR_NAME }}
|
||||
- name: Log in to GitHub Container Registry
|
||||
env:
|
||||
GHCR_TOKEN: ${{ secrets.GHCR_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GHCR_USERNAME: ${{ vars.GHCR_USERNAME || github.actor }}
|
||||
run: echo "${GHCR_TOKEN}" | docker login ghcr.io -u "${GHCR_USERNAME}" --password-stdin
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
|
||||
@@ -239,7 +235,7 @@ jobs:
|
||||
--platform ${{ matrix.platform }} \
|
||||
--push \
|
||||
--tag ${{ env.REGISTRY_NAME }}/${{ matrix.image }}:${{ env.TAG }}-${PLATFORM_TAG} \
|
||||
--cache-from type=registry,ref=${{ env.REGISTRY_NAME }}/public/${{ matrix.image }}:latest \
|
||||
--cache-from type=registry,ref=${{ env.REGISTRY_NAME }}/${{ matrix.image }}:latest \
|
||||
--cache-to type=inline \
|
||||
--attest type=sbom \
|
||||
--attest type=provenance,mode=max \
|
||||
@@ -254,18 +250,14 @@ jobs:
|
||||
needs: build-platform-images
|
||||
if: github.ref == 'refs/heads/main'
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Azure Login using OIDC
|
||||
uses: azure/login@532459ea530d8321f2fb9bb10d1e0bcf23869a43 # v3.0.0
|
||||
with:
|
||||
client-id: ${{ secrets.AZURE_CLIENT_ID }}
|
||||
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
|
||||
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
|
||||
|
||||
- name: Log in to Azure Container Registry
|
||||
run: az acr login --name ${{ secrets.ACR_NAME }}
|
||||
- name: Log in to GitHub Container Registry
|
||||
env:
|
||||
GHCR_TOKEN: ${{ secrets.GHCR_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GHCR_USERNAME: ${{ vars.GHCR_USERNAME || github.actor }}
|
||||
run: echo "${GHCR_TOKEN}" | docker login ghcr.io -u "${GHCR_USERNAME}" --password-stdin
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
|
||||
@@ -291,8 +283,8 @@ jobs:
|
||||
needs: create-manifests
|
||||
if: github.ref == 'refs/heads/main'
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
packages: read
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
@@ -306,23 +298,19 @@ jobs:
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Azure Login using OIDC
|
||||
uses: azure/login@532459ea530d8321f2fb9bb10d1e0bcf23869a43 # v3.0.0
|
||||
with:
|
||||
client-id: ${{ secrets.AZURE_CLIENT_ID }}
|
||||
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
|
||||
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
|
||||
- name: Log in to GitHub Container Registry
|
||||
env:
|
||||
GHCR_TOKEN: ${{ secrets.GHCR_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GHCR_USERNAME: ${{ vars.GHCR_USERNAME || github.actor }}
|
||||
run: echo "${GHCR_TOKEN}" | docker login ghcr.io -u "${GHCR_USERNAME}" --password-stdin
|
||||
|
||||
- name: Log in to Azure Container Registry
|
||||
run: az acr login --name ${{ secrets.ACR_NAME }}
|
||||
|
||||
- name: Pull Presidio images from ACR
|
||||
- name: Pull Presidio images from GHCR
|
||||
run: |
|
||||
export DOCKER_DEFAULT_PLATFORM=${{ matrix.platform }}
|
||||
docker compose -f docker-compose.yml pull
|
||||
env:
|
||||
REGISTRY_NAME: ${{ env.REGISTRY_NAME }}
|
||||
TAG: :${{ env.TAG }}
|
||||
TAG: ':${{ env.TAG }}'
|
||||
|
||||
- name: Start Presidio cluster
|
||||
run: |
|
||||
@@ -330,7 +318,7 @@ jobs:
|
||||
docker compose -f docker-compose.yml up -d
|
||||
env:
|
||||
REGISTRY_NAME: ${{ env.REGISTRY_NAME }}
|
||||
TAG: :${{ env.TAG }}
|
||||
TAG: ':${{ env.TAG }}'
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
||||
@@ -407,17 +395,17 @@ jobs:
|
||||
# Build all images locally without pushing
|
||||
docker compose -f docker-compose.yml build
|
||||
env:
|
||||
REGISTRY_NAME: 'mcr.microsoft.com'
|
||||
REGISTRY_NAME: 'ghcr.io/data-privacy-stack'
|
||||
IMAGE_PREFIX: ''
|
||||
TAG: gha${{ github.run_number }}
|
||||
TAG: ':gha${{ github.run_number }}'
|
||||
|
||||
- name: Start Presidio cluster locally
|
||||
run: |
|
||||
docker compose -f docker-compose.yml up -d
|
||||
env:
|
||||
REGISTRY_NAME: 'mcr.microsoft.com'
|
||||
REGISTRY_NAME: 'ghcr.io/data-privacy-stack'
|
||||
IMAGE_PREFIX: ''
|
||||
TAG: gha${{ github.run_number }}
|
||||
TAG: ':gha${{ github.run_number }}'
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
|
||||
|
||||
26
.github/workflows/release.yml
vendored
26
.github/workflows/release.yml
vendored
@@ -6,8 +6,8 @@ on:
|
||||
permissions: read-all
|
||||
|
||||
env:
|
||||
REGISTRY_NAME: ${{ secrets.ACR_NAME }}.azurecr.io
|
||||
IMAGE_PREFIX: "public/"
|
||||
IMAGE_REGISTRY: ghcr.io
|
||||
IMAGE_NAMESPACE: data-privacy-stack
|
||||
|
||||
jobs:
|
||||
get-version:
|
||||
@@ -112,8 +112,8 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
needs: get-version
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
packages: write
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -125,22 +125,18 @@ jobs:
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Azure Login using OIDC
|
||||
uses: azure/login@532459ea530d8321f2fb9bb10d1e0bcf23869a43 # v3.0.0
|
||||
with:
|
||||
client-id: ${{ secrets.AZURE_CLIENT_ID }}
|
||||
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
|
||||
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
|
||||
|
||||
- name: Log in to Azure Container Registry
|
||||
run: az acr login --name ${{ secrets.ACR_NAME }}
|
||||
- name: Log in to GitHub Container Registry
|
||||
env:
|
||||
GHCR_TOKEN: ${{ secrets.GHCR_TOKEN || secrets.GITHUB_TOKEN }}
|
||||
GHCR_USERNAME: ${{ vars.GHCR_USERNAME || github.actor }}
|
||||
run: echo "${GHCR_TOKEN}" | docker login ${{ env.IMAGE_REGISTRY }} -u "${GHCR_USERNAME}" --password-stdin
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
|
||||
|
||||
- name: Build and Push ${{ matrix.image }}
|
||||
run: |
|
||||
repo="${{ env.REGISTRY_NAME }}/${{ env.IMAGE_PREFIX }}${{ matrix.image }}"
|
||||
repo="${{ env.IMAGE_REGISTRY }}/${{ env.IMAGE_NAMESPACE }}/${{ matrix.image }}"
|
||||
version_tag="${{ matrix.image == 'presidio-image-redactor' && needs.get-version.outputs.image-version || needs.get-version.outputs.version }}"
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
@@ -153,5 +149,5 @@ jobs:
|
||||
--push \
|
||||
./${{ matrix.image }}
|
||||
env:
|
||||
REGISTRY_NAME: ${{ env.REGISTRY_NAME }}
|
||||
IMAGE_PREFIX: ${{ env.IMAGE_PREFIX }}
|
||||
IMAGE_REGISTRY: ${{ env.IMAGE_REGISTRY }}
|
||||
IMAGE_NAMESPACE: ${{ env.IMAGE_NAMESPACE }}
|
||||
|
||||
@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
|
||||
|
||||
## [unreleased]
|
||||
|
||||
### General
|
||||
#### Changed
|
||||
- Moved container image publishing from Microsoft Container Registry to GitHub Container Registry under `ghcr.io/data-privacy-stack`, and updated Docker usage documentation to explain that `mcr.microsoft.com/presidio-*` images are legacy images that are no longer updated.
|
||||
|
||||
### Anonymizer
|
||||
#### Fixed
|
||||
- Custom operator `validate()` no longer calls the user-supplied lambda with a dummy `"PII"` value. Previously, stateful lambdas (e.g. those accumulating a token-to-original-value map for de-anonymization) would receive a spurious invocation during validation, inserting a junk entry (`{"TOKEN_1": "PII"}`) into the map and skewing all subsequent token counters. The return-type contract is now enforced in `operate()` when the lambda runs on real data. Fixes [#2024](https://github.com/microsoft/presidio/issues/2024).
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
services:
|
||||
presidio-image-redactor:
|
||||
image: ${REGISTRY_NAME}${IMAGE_PREFIX}presidio-image-redactor${TAG}
|
||||
build:
|
||||
context: ./presidio-image-redactor
|
||||
environment:
|
||||
- PORT=5001
|
||||
ports:
|
||||
- "5003:5001"
|
||||
@@ -1,17 +0,0 @@
|
||||
services:
|
||||
presidio-anonymizer:
|
||||
image: ${REGISTRY_NAME}${IMAGE_PREFIX}presidio-anonymizer${TAG}
|
||||
build:
|
||||
context: ./presidio-anonymizer
|
||||
environment:
|
||||
- PORT=5001
|
||||
ports:
|
||||
- "5001:5001"
|
||||
presidio-analyzer:
|
||||
image: ${REGISTRY_NAME}${IMAGE_PREFIX}presidio-analyzer${TAG}
|
||||
build:
|
||||
context: ./presidio-analyzer
|
||||
environment:
|
||||
- PORT=5001
|
||||
ports:
|
||||
- "5002:5001"
|
||||
@@ -1,20 +0,0 @@
|
||||
services:
|
||||
presidio-anonymizer:
|
||||
image: ${REGISTRY_NAME}${IMAGE_PREFIX}presidio-anonymizer${TAG}
|
||||
build:
|
||||
context: ./presidio-anonymizer
|
||||
environment:
|
||||
- PORT=5001
|
||||
ports:
|
||||
- "5001:5001"
|
||||
presidio-analyzer:
|
||||
image: ${REGISTRY_NAME}${IMAGE_PREFIX}presidio-analyzer${TAG}
|
||||
build:
|
||||
context: ./presidio-analyzer
|
||||
args:
|
||||
- NLP_CONF_FILE=presidio_analyzer/conf/transformers.yaml
|
||||
dockerfile: Dockerfile.transformers
|
||||
environment:
|
||||
- PORT=5001
|
||||
ports:
|
||||
- "5002:5001"
|
||||
@@ -1,75 +1,61 @@
|
||||
# Build and release process
|
||||
|
||||
Presidio leverages Azure DevOps YAML pipelines to validate, build, release and deliver presidio.
|
||||
The pipelines make use of [templates](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops)
|
||||
for code reuse using [YAML Schema](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema).
|
||||
Presidio uses GitHub Actions workflows to validate, build, release, and deliver official artifacts.
|
||||
|
||||
## Description
|
||||
## Workflows
|
||||
|
||||
The following pipelines are provided and maintained as part of presidio development process:
|
||||
The following workflows are maintained as part of the Presidio development process:
|
||||
|
||||
- [PR Validation](https://github.com/data-privacy-stack/presidio/blob/main/azure-pipelines.yml) - used to validate pull requests.
|
||||
- Linting
|
||||
- Security and compliance analysis
|
||||
- Unit tests
|
||||
- E2E tests
|
||||
- [CI](https://github.com/data-privacy-stack/presidio/blob/main/azure-pipelines-ci.yml) - triggered on merge to main branch.
|
||||
- Linting
|
||||
- Security and compliance analysis
|
||||
- Unit tests
|
||||
- E2E tests
|
||||
- deploys the artifacts to an internal dev environment.
|
||||
- [Release](https://github.com/data-privacy-stack/presidio/blob/main/azure-pipelines.yml) - manually triggered.
|
||||
- releases presidio official artifacts
|
||||
- pypi
|
||||
- Microsoft container registry (and docker hub)
|
||||
- GitHub
|
||||
- updates the official demo environment.
|
||||
- [CI](https://github.com/data-privacy-stack/presidio/blob/main/.github/workflows/ci.yml) - triggered for pull requests and merges to the main branch.
|
||||
- Linting
|
||||
- Dependency review
|
||||
- Unit tests
|
||||
- Package builds
|
||||
- Container builds and E2E tests
|
||||
- [Release](https://github.com/data-privacy-stack/presidio/blob/main/.github/workflows/release.yml) - manually triggered for official releases.
|
||||
- Publishes Python packages to [PyPI](https://pypi.org/search/?q=presidio)
|
||||
- Publishes container images to [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages)
|
||||
- Creates a draft GitHub release
|
||||
|
||||
### Variables used by the pipelines
|
||||
## Container image publishing
|
||||
|
||||
#### CI Pipeline
|
||||
Official Presidio container images are now published to GitHub Container Registry (GHCR) under the Data Privacy Stack organization:
|
||||
|
||||
- **_ACR_AZURE_SUBSCRIPTION_** - Service connection to Azure subscription where Azure Container Registry is.
|
||||
- **_ACR_REGISTRY_NAME_** - Name of Azure Container Registry.
|
||||
- **_ANALYZER_DEV_APP_NAME_** - Name of existing App Service for Analyzer (development environment).
|
||||
- **_ANONYMIZER_DEV_APP_NAME_** - Name of existing App Service for Anonymizer (development environment).
|
||||
- **_IMAGE_REDACTOR_DEV_APP_NAME_** - Name of existing App Service for Image Redactor (development environment).
|
||||
- **_DEV_AZURE_SUBSCRIPTION_** - Service connection to Azure subscription where App Services are (development environment).
|
||||
- **_DEV_RESOURCE_GROUP_NAME_** - Name of resource group where App Services are (development environment).
|
||||
- [`ghcr.io/data-privacy-stack/presidio-analyzer`](https://github.com/orgs/data-privacy-stack/packages/container/package/presidio-analyzer)
|
||||
- [`ghcr.io/data-privacy-stack/presidio-anonymizer`](https://github.com/orgs/data-privacy-stack/packages/container/package/presidio-anonymizer)
|
||||
- [`ghcr.io/data-privacy-stack/presidio-image-redactor`](https://github.com/orgs/data-privacy-stack/packages/container/package/presidio-image-redactor)
|
||||
|
||||
#### Release Pipeline
|
||||
The legacy Microsoft Container Registry images at `mcr.microsoft.com/presidio-*` are no longer updated. Existing MCR tags can still be pulled for older deployments, but users should update image references to GHCR before upgrading. For example:
|
||||
|
||||
- **_ACR_AZURE_SUBSCRIPTION_** - Service connection to Azure subscription where Azure Container Registry is.
|
||||
- **_ACR_REGISTRY_NAME_** - Name of Azure Container Registry.
|
||||
- **_ANALYZER_PROD_APP_NAME_** - Name of existing App Service for Analyzer (production environment).
|
||||
- **_ANONYMIZER_PROD_APP_NAME_** - Name of existing App Service for Anonymizer (production environment).
|
||||
- **_PROD_AZURE_SUBSCRIPTION_** - Service connection to Azure subscription where App Services are (production environment).
|
||||
- **_PROD_RESOURCE_GROUP_NAME_** - Name of resource group where App Services are (production environment).
|
||||
```sh
|
||||
docker pull ghcr.io/data-privacy-stack/presidio-analyzer:latest
|
||||
```
|
||||
|
||||
### Import a pipeline to Azure Devops
|
||||
For production deployments, avoid relying on the moving `latest` tag. Pin the release tag shown on the relevant [GitHub Packages page](https://github.com/orgs/data-privacy-stack/packages), for example:
|
||||
|
||||
- Sign in to your Azure DevOps organization and navigate to your project.
|
||||
- In your project, navigate to the Pipelines page. Then choose the action to create a new pipeline.
|
||||
- Walk through the steps of the wizard by first selecting 'Use the classic editor, and select GitHub as the location of your source code.
|
||||
- You might be redirected to GitHub to sign in. If so, enter your GitHub credentials.
|
||||
- When the list of repositories appears, select presidio repository.
|
||||
- Point Azure Pipelines to the relevant yaml definition you'd like to import.
|
||||
Set the pipeline's name, the required triggers and variables and Select Save and run.
|
||||
- A new run is started. Wait for the run to finish.
|
||||
```sh
|
||||
docker pull ghcr.io/data-privacy-stack/presidio-analyzer:<release-version>
|
||||
```
|
||||
|
||||
### PyPI Publishing with OIDC
|
||||
If an existing deployment uses `mcr.microsoft.com/presidio-analyzer:latest`, treat it as a legacy reference and replace it with either `ghcr.io/data-privacy-stack/presidio-analyzer:latest` for quick testing or a pinned GHCR release tag for production.
|
||||
|
||||
The GitHub Actions release workflow uses OIDC (OpenID Connect) trusted publishing to PyPI, which provides enhanced security by eliminating the need to manage PyPI API tokens. This requires:
|
||||
Docker registries do not provide a portable way to redirect `docker pull mcr.microsoft.com/presidio-analyzer:latest` to GHCR or show a deprecation warning during `docker pull`. To reduce confusion, each GitHub release includes the GHCR image names, the GHCR images include OCI metadata labels pointing to the source repository, package page, and installation guide, and the MCR tags are documented as legacy references.
|
||||
|
||||
1. **PyPI Configuration**: Each package (presidio_analyzer, presidio_anonymizer, etc.) must be configured on PyPI to trust the GitHub repository and workflow.
|
||||
2. **GitHub Workflow**: The workflow uses `pypa/gh-action-pypi-publish@release/v1` with `id-token: write` permissions.
|
||||
3. **No Secrets Required**: No PyPI API tokens need to be stored as GitHub secrets.
|
||||
## PyPI publishing with OIDC
|
||||
|
||||
The release workflow uses OIDC (OpenID Connect) trusted publishing to PyPI, which eliminates the need to manage PyPI API tokens. This requires:
|
||||
|
||||
1. **PyPI configuration**: Each package (`presidio-analyzer`, `presidio-anonymizer`, and the other Presidio packages) must be configured on PyPI to trust the GitHub repository and workflow.
|
||||
2. **GitHub workflow permissions**: The workflow uses `pypa/gh-action-pypi-publish@release/v1` with `id-token: write`.
|
||||
3. **No PyPI secrets**: No PyPI API tokens need to be stored as GitHub secrets.
|
||||
|
||||
Benefits of OIDC:
|
||||
|
||||
- Enhanced security through short-lived tokens
|
||||
- No manual token management
|
||||
- Automatic token rotation
|
||||
- Audit trail of publishing activities
|
||||
|
||||
Note: The Azure DevOps pipeline continues to use traditional PyPI authentication with service connections.
|
||||
## GHCR publishing permissions
|
||||
|
||||
The container publishing jobs authenticate to GHCR with the workflow `GITHUB_TOKEN` by default and require `packages: write` permissions. If the workflow is run from a repository outside the `data-privacy-stack` organization during the transition, set the `GHCR_TOKEN` secret and optional `GHCR_USERNAME` variable to credentials that can publish packages to the Data Privacy Stack organization, or run the release from the migrated repository.
|
||||
|
||||
@@ -72,7 +72,7 @@ docker pull ghcr.io/data-privacy-stack/presidio-image-redactor
|
||||
2. Run container
|
||||
|
||||
```sh
|
||||
docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor
|
||||
docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest
|
||||
```
|
||||
|
||||
3. Use the API
|
||||
|
||||
@@ -102,7 +102,7 @@ Using Presidio's modules as Python packages to get started:
|
||||
|
||||
## Simple flow - Docker container
|
||||
|
||||
Presidio provides Docker containers that you can use to de-identify text data. Each module, analyzer, and anonymizer, has its own Docker container. The containers are available on Docker Hub.
|
||||
Presidio provides Docker containers that you can use to de-identify text data. Each module, analyzer, and anonymizer, has its own Docker container. New releases are available on [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages); the legacy Microsoft Container Registry images are no longer updated.
|
||||
|
||||
1. Download Docker images
|
||||
|
||||
|
||||
@@ -44,9 +44,9 @@ Pre-requisites:
|
||||
This requires Docker to be installed. [Download Docker](https://docs.docker.com/get-docker/).
|
||||
|
||||
```sh
|
||||
# Download image from Dockerhub
|
||||
# Download image from GitHub Container Registry
|
||||
docker pull ghcr.io/data-privacy-stack/presidio-image-redactor
|
||||
|
||||
|
||||
# Run the container with the default port
|
||||
docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest
|
||||
```
|
||||
@@ -161,19 +161,19 @@ The most basic usage of the engine can be setup like the following in python
|
||||
diOCR = DocumentIntelligenceOCR(endpoint="<your_endpoint>", key="<your_key>")
|
||||
```
|
||||
|
||||
If your environment uses Azure Identity instead of API keys, pass an Azure SDK
|
||||
credential that implements `TokenCredential`, such as `DefaultAzureCredential`.
|
||||
For example, install `azure-identity` and use `DefaultAzureCredential`:
|
||||
|
||||
```
|
||||
from azure.identity import DefaultAzureCredential
|
||||
|
||||
diOCR = DocumentIntelligenceOCR(
|
||||
endpoint="<your_endpoint>",
|
||||
credential=DefaultAzureCredential(),
|
||||
)
|
||||
```
|
||||
|
||||
If your environment uses Azure Identity instead of API keys, pass an Azure SDK
|
||||
credential that implements `TokenCredential`, such as `DefaultAzureCredential`.
|
||||
For example, install `azure-identity` and use `DefaultAzureCredential`:
|
||||
|
||||
```
|
||||
from azure.identity import DefaultAzureCredential
|
||||
|
||||
diOCR = DocumentIntelligenceOCR(
|
||||
endpoint="<your_endpoint>",
|
||||
credential=DefaultAzureCredential(),
|
||||
)
|
||||
```
|
||||
|
||||
The DocumentIntelligenceOCR can also attempt to pull your endpoint and key values from environment variables.
|
||||
|
||||
```
|
||||
|
||||
@@ -96,6 +96,10 @@ To download the Presidio Docker containers, run the following command:
|
||||
|
||||
This requires Docker to be installed. [Download Docker](https://docs.docker.com/get-docker/).
|
||||
|
||||
!!! important "Container registry moved to GitHub Packages"
|
||||
|
||||
New Presidio container releases are published to [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) under the [Data Privacy Stack packages organization](https://github.com/orgs/data-privacy-stack/packages). The legacy Microsoft Container Registry images at `mcr.microsoft.com/presidio-*` are no longer updated. If you previously used `mcr.microsoft.com/presidio-analyzer:latest`, switch to `ghcr.io/data-privacy-stack/presidio-analyzer:latest`; for production, prefer pinning an explicit release tag from the package page.
|
||||
|
||||
### For PII anonymization in text
|
||||
|
||||
For PII detection and anonymization in text, the `presidio-analyzer`
|
||||
|
||||
@@ -32,7 +32,7 @@ During the migration process:
|
||||
|
||||
* Repositories, documentation links, and package references may gradually change.
|
||||
* Some GitHub URLs and documentation pages may redirect temporarily.
|
||||
* New Docker image releases will be published on Docker Hub rather than Microsoft Container Registry (MCR).
|
||||
* New Docker image releases are published to [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages) under the Data Privacy Stack organization. Legacy Microsoft Container Registry (MCR) images remain available for older tags but are no longer updated; update `mcr.microsoft.com/presidio-*` image references to `ghcr.io/data-privacy-stack/presidio-*`.
|
||||
* Contribution and governance processes will evolve and become more community-driven.
|
||||
* The technical steering committee (TSC) will expand to include maintainers and contributors from outside Microsoft.
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ APP_SERVICE_NAME=<name of app service>
|
||||
LOCATION=<location>
|
||||
APP_SERVICE_SKU=<sku>
|
||||
|
||||
IMAGE_NAME=ghcr.io/data-privacy-stack/presidio-analyzer
|
||||
IMAGE_NAME=ghcr.io/data-privacy-stack/presidio-analyzer:latest
|
||||
# the following parameters are only required if you build and deploy your own containers from a private registry
|
||||
ACR_USER_NAME=<user name>
|
||||
ACR_USER_PASSWORD=<password>
|
||||
|
||||
@@ -50,7 +50,7 @@ You can install Presidio locally using [KIND](https://github.com/kubernetes-sigs
|
||||
```
|
||||
|
||||
3. Local [helm](https://helm.sh/) client.
|
||||
4. **Optional** - Container Registry - such as [ACR](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro). Only needed if you are using your own presidio images and not the default ones from the [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages)
|
||||
4. **Optional** - Container Registry - such as [ACR](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro). Only needed if you are using your own Presidio images and not the default ones from [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages).
|
||||
5. Recent presidio repo is cloned on your local machine.
|
||||
|
||||
### Step by step deployment with customizable parameters
|
||||
@@ -69,16 +69,16 @@ You can install Presidio locally using [KIND](https://github.com/kubernetes-sigs
|
||||
# Choose a namespace and ensure it is created
|
||||
NAMESPACE=presidio
|
||||
|
||||
# Choose the tag, from ghcr.io/data-privacy-stack, e.g. `latest`
|
||||
# Choose the tag from ghcr.io/data-privacy-stack, e.g. `latest`
|
||||
TAG=latest
|
||||
|
||||
# Choose a name for the deployment
|
||||
NAME=<name>
|
||||
|
||||
# Use Helm to install all required components
|
||||
helm install $NAME . --set tag=$PRESIDIO_LABEL --namespace $NAMESPACE
|
||||
helm install $NAME . --set tag=$TAG --namespace $NAMESPACE
|
||||
|
||||
# If you have your own images in a separate ACR, run
|
||||
DOCKER_REGISTRY=<your_registry>
|
||||
helm install $NAME . --set registry=$DOCKER_REGISTRY,tag=$PRESIDIO_LABEL . --namespace $NAMESPACE
|
||||
helm install $NAME . --set registry=$DOCKER_REGISTRY,tag=$TAG --namespace $NAMESPACE
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user