upstream-archive/presidio
1
0
Fork 0
mirror of https://github.com/data-privacy-stack/presidio.git synced 2026-07-23 11:20:55 -05:00
Code Issues Packages Projects Releases Wiki Activity

Unified Analyzer Configuration (#1970)

Browse Source 
* Unified Analyzer Configuration

* Update presidio-analyzer/Dockerfile.windows

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/analyzer/customizing_nlp_models.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: remove broken COPY commands for deprecated conf files in all Dockerfiles

Agent-Logs-Url: https://github.com/microsoft/presidio/sessions/88e42ff4-6920-4698-9769-1c702ccf5556

Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com>

* Use unified analyzer config for registry defaults

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Copilot config review comments

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove Docker config changelog entry

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify deprecated analyzer config warning

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove analyzer config warning test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Sharon Hart
2026-05-17 10:02:56 +03:00
committed by GitHub
parent 69cf95deaa
commit beeb881f64
30 changed files with 1179 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/copilot-instructions.md vendored
View File

@@ -236,7 +236,7 @@ Follow best practices in docs/analyzer/developing_recognizers.md:
2. **Make regex patterns specific** to minimize false positives
3. **Document pattern sources** with comments linking to standards/references
4. **Add to configuration** in `conf/default_recognizers.yaml` (set `enabled: false` for country-specific)
4. **Add to configuration** in `conf/analyzer.yaml` (set `enabled: false` for country-specific)
5. **Update imports** in `predefined_recognizers/__init__.py`
6. **Add comprehensive tests** including edge cases
7. **Update supported entities documentation** if adding new entity types

2
CONTRIBUTING.md
View File

@@ -64,7 +64,7 @@ To contribute a new predefined recognizer to Presidio Analyzer:
- Document the source or reference for any new regex logic (e.g., link to a standard, documentation, or example dataset) in the code as a comment.
3. **Add your recognizer to the configuration:**
- Add your recognizer to `presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml`.
- Add your recognizer to `presidio-analyzer/presidio_analyzer/conf/analyzer.yaml`.
- For country-specific recognizers, set `enabled: false` by default in the YAML configuration.
3. **Update imports:** Add your recognizer to `presidio-analyzer/presidio_analyzer/predefined_recognizers/__init__.py` so it is available for import and backward compatibility.

4
docs/analyzer/adding_recognizers.md
View File

@@ -150,7 +150,7 @@ To add a recognizer to the list of pre-defined recognizers:
1. Clone the repo.
2. Create a file containing the new recognizer Python class.
3. Add the recognizer to the `recognizers` in the [`default_recognizers`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) config. Details of recognizer parameters are given [Here](./recognizer_registry_provider.md#the-recognizer-parameters).
3. Add the recognizer to the `recognizers` section in the [`analyzer.yaml`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml) configuration file. Details of recognizer parameters are given [Here](./recognizer_registry_provider.md#the-recognizer-parameters).
4. Optional: Update documentation (e.g., the [supported entities list](../supported_entities.md)).
### Azure AI Language recognizer
@@ -231,7 +231,7 @@ Additional examples can be found in the [OpenAPI spec](../api-docs/api-docs.html
### Reading pattern recognizers from YAML
Recognizers can be loaded from a YAML file, which allows users to add recognition logic without writing code.
An example YAML file can be found [here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml).
An example YAML file can be found [here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml).
Once the YAML file is created, it can be loaded into the `RecognizerRegistry` instance.

76
docs/analyzer/analyzer_engine_provider.md
View File

@@ -1,20 +1,17 @@
# Configuring the Analyzer Engine from file
Presidio uses `AnalyzerEngineProvider` to load `AnalyzerEngine` configuration from file.
Configuration can be loaded in three different ways:
Presidio uses `AnalyzerEngineProvider` to load `AnalyzerEngine` configuration from file.
## Using a single file
## Using the unified configuration file (recommended)
Create an `AnalyzerEngineProvider` using a single configuration file and set its path to `analyzer_engine_conf_file`, then create `AnalyzerEngine` based on it:
The recommended approach is to use a **single configuration file** (`analyzer.yaml`) that contains all settings: supported languages, NLP engine configuration, and recognizer registry.
```python
from presidio_analyzer import AnalyzerEngine, AnalyzerEngineProvider
analyzer_conf_file = "./analyzer/analyzer-config-all.yml"
provider = AnalyzerEngineProvider(
analyzer_engine_conf_file=analyzer_conf_file
)
analyzer_engine_conf_file="./analyzer-config.yaml"
)
analyzer = provider.create_engine()
results = analyzer.analyze(text="My name is Morris", language="en")
@@ -61,7 +58,6 @@ recognizer_registry:
- name: CreditCardRecognizer
supported_languages:
- en
supported_entity: IT_FISCAL_CODE
type: predefined
- name: ItFiscalCodeRecognizer
@@ -79,70 +75,48 @@ The configuration file contains the following parameters:
`supported_languages` must be identical to the same field in recognizer_registry
## Using multiple files
You can also load and validate the configuration using the Pydantic-based `AnalyzerConfiguration` model:
Create an `AnalyzerEngineProvider` using three different configuration files for each of the following components:
```python
from presidio_analyzer.configuration import AnalyzerConfiguration
- Analyzer
- NLP Engine
- Recognizer Registry
config = AnalyzerConfiguration.from_yaml("./analyzer-config.yaml")
```
!!! note "Note"
Each of these parameters is optional and in case it's not set, the default configuration will be used.
## Using the default configuration
Create an `AnalyzerEngineProvider` without any parameters. This will load the default configuration from the built-in [analyzer.yaml](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml):
```python
from presidio_analyzer import AnalyzerEngine, AnalyzerEngineProvider
analyzer_conf_file = "./analyzer/analyzer-config.yml"
nlp_engine_conf_file = "./analyzer/nlp-config.yml"
recognizer_registry_conf_file = "./analyzer/recognizers-config.yml"
provider = AnalyzerEngineProvider(
analyzer_engine_conf_file=analyzer_conf_file,
nlp_engine_conf_file=nlp_engine_conf_file,
recognizer_registry_conf_file=recognizer_registry_conf_file,
)
analyzer = provider.create_engine()
analyzer = AnalyzerEngineProvider().create_engine()
results = analyzer.analyze(text="My name is Morris", language="en")
print(results)
```
The structure of the configuration files is as follows:
## Using multiple files (deprecated)
- Analyzer engine configuration file:
!!! warning "Deprecated"
```yaml
supported_languages:
- en
default_score_threshold: 0
```
Using separate configuration files for the NLP engine and recognizer registry is deprecated.
Use the unified configuration file instead – place the `nlp_configuration` and
`recognizer_registry` sections inside the analyzer configuration file.
- NLP engine configuration file structure is examined thoroughly in the [Customizing the NLP model](customizing_nlp_models.md) section.
- Recognizer registry configuration file structure is examined thoroughly in the [Customizing recognizer registry from file](recognizer_registry_provider.md) section.
## Using the default configuration
Create an `AnalyzerEngineProvider` without any parameters. This will load the default configuration:
The `nlp_engine_conf_file` and `recognizer_registry_conf_file` parameters are still supported for backward compatibility, but will be removed in a future version.
```python
from presidio_analyzer import AnalyzerEngine, AnalyzerEngineProvider
provider = AnalyzerEngineProvider().create_engine()
results = provider.analyze(text="My name is Morris", language="en")
print(results)
provider = AnalyzerEngineProvider(
analyzer_engine_conf_file="./analyzer-config.yml",
nlp_engine_conf_file="./nlp-config.yml", # deprecated
recognizer_registry_conf_file="./recognizers.yml", # deprecated
)
analyzer = provider.create_engine()
```
The default configuration of `AnalyzerEngine` is defined in the following files:
- [Analyzer Engine](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)
- [NLP Engine](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)
- [Recognizer Registry](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)
## Enabling and disabling recognizers
In general, recognizers that are not added to the configuration would not be created, with one exception.

16
docs/analyzer/customizing_nlp_models.md
View File

@@ -50,9 +50,9 @@ Configuration can be done in two ways:
print(results_english)
```
- **Via configuration**: Set up the models which should be used in the [default `conf` file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml).
- **Via configuration**: Set up the models either in a standalone NLP configuration file used by `NlpEngineProvider(conf_file=...)`, or in the `nlp_configuration` section of the [unified analyzer configuration file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml) used by `AnalyzerEngineProvider`.
An example Conf file:
An example standalone NLP configuration file:
```yaml
nlp_engine_name: spacy
@@ -86,7 +86,7 @@ Configuration can be done in two ways:
- `low_confidence_score_multiplier`: A multiplier to apply to the score of entities with low confidence.
- `low_score_entity_names`: A list of entity types to apply the low confidence score multiplier to.
The [default conf file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) is read during the default initialization of the `AnalyzerEngine`. Alternatively, the path to a custom configuration file can be passed to the `NlpEngineProvider`:
To load NLP settings from a standalone NLP configuration file, use `NlpEngineProvider(conf_file=...)`. Alternatively, NLP configuration can be passed directly:
```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry
@@ -112,6 +112,16 @@ Configuration can be done in two ways:
print(results_english)
```
To load NLP settings from the unified analyzer configuration file, use `AnalyzerEngineProvider`:
```python
from presidio_analyzer import AnalyzerEngineProvider
analyzer = AnalyzerEngineProvider(
analyzer_engine_conf_file="presidio_analyzer/conf/analyzer.yaml"
).create_engine()
```
In this examples we:
a. create an `NlpEngine` holding two spaCy models (one in English: `en_core_web_lg` and one in Spanish: `es_core_news_md`).
b. define the `supported_languages` parameter accordingly.

2
docs/analyzer/languages.md
View File

@@ -70,7 +70,7 @@ Link to LANGUAGES_CONFIG_FILE=[languages-config.yml](https://github.com/microsof
When packaging the code into a Docker container, NLP models are automatically installed.
To define which models should be installed,
update the [conf/default.yaml](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) file. This file is read during
update the `nlp_configuration` section in the [analyzer configuration file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml). This file is read during
the `docker build` phase and the models defined in it are installed automatically.
For `transformers` based models, the configuration [can be found here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/transformers.yaml).

18
docs/samples/python/langextract/index.md
View File

@@ -113,7 +113,7 @@ You have two options to set up Ollama:
**Option 1: Enable in configuration file**
Enable the recognizer in [`default_recognizers.yaml`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml):
Enable the recognizer in [`analyzer.yaml`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml):
```yaml
- name: BasicLangExtractRecognizer
enabled: true # Change from false to true
@@ -122,17 +122,13 @@ Enable the recognizer in [`default_recognizers.yaml`](https://github.com/microso
Then load the analyzer using this modified configuration file:
```python
from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.recognizer_registry import RecognizerRegistryProvider
from presidio_analyzer import AnalyzerEngineProvider
# Point to your modified default_recognizers.yaml with Ollama enabled
provider = RecognizerRegistryProvider(
conf_file="/path/to/your/modified/default_recognizers.yaml"
# Point to your modified analyzer.yaml with the recognizer enabled
provider = AnalyzerEngineProvider(
analyzer_engine_conf_file="/path/to/your/modified/analyzer.yaml"
)
registry = provider.create_recognizer_registry()
# Create analyzer with the registry that includes Ollama recognizer
analyzer = AnalyzerEngine(registry=registry, supported_languages=["en"])
analyzer = provider.create_engine()
# Analyze text - Ollama recognizer will participate in detection
results = analyzer.analyze(text="My email is john.doe@example.com", language="en")
@@ -151,7 +147,7 @@ results = analyzer.analyze(text="My email is john.doe@example.com", language="en
```
!!! note "Note"
The recognizer is disabled by default in `default_recognizers.yaml` to avoid requiring Ollama for basic Presidio usage. Enable it when you have Ollama set up and running.
The recognizer is disabled by default in the configuration to avoid requiring Ollama for basic Presidio usage. Enable it when you have Ollama set up and running.
### Custom Configuration

19
docs/samples/python/no_code_config.ipynb
View File

@@ -20,14 +20,13 @@
"3. For team members interested in changing the configuration without writing code.\n",
"\n",
"In this example, we'll show how to create a no-code configuration in Presidio.\n",
"We start by creating YAML configuration files that are based on the default ones. \n",
"Te default configuration files for Presidio can be found here:\n",
"- [Analyzer configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)\n",
"- [Recognizer registry configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)\n",
"- [NLP engine configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)\n",
"We start by creating a YAML configuration file based on the default one.\n",
"The default unified configuration file for Presidio Analyzer can be found here:\n",
"- [Unified analyzer configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml)\n",
"\n",
"Alternatively, one can create one configuration file for all three components.\n",
"In this example, we'll tweak the configuration to reduce the number of predefinedrecognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish).\n"
"> **Note:** The previous approach of using three separate configuration files (`default_analyzer.yaml`, `default_recognizers.yaml`, `default.yaml`) is deprecated. Use the unified `analyzer.yaml` instead.\n",
"\n",
"In this example, we'll tweak the configuration to reduce the number of predefined recognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish)."
]
},
{
@@ -61,7 +60,7 @@
"metadata": {},
"source": [
"### General Analyzer parameters\n",
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml))"
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml))"
]
},
{
@@ -85,7 +84,7 @@
"metadata": {},
"source": [
"### Recognizer Registry parameters\n",
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml))"
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml))"
]
},
{
@@ -173,7 +172,7 @@
"metadata": {},
"source": [
"### NLP Engine parameters\n",
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml))"
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml))"
]
},
{

20
docs/tutorial/08_no_code.md
View File

@@ -7,14 +7,16 @@ No-code configuration can be helpful in three scenarios:
3. For team members interested in changing the configuration without writing code.
In this example, we'll show how to create a no-code configuration in Presidio.
We start by creating YAML configuration files that are based on the default ones.
The default configuration files for Presidio can be found here:
We start by creating a YAML configuration file based on the default one.
The default unified configuration file for Presidio Analyzer can be found here:
- [Analyzer configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)
- [Recognizer registry configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)
- [NLP engine configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)
- [analyzer configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml)
Alternatively, one can create one configuration file for all three components.
!!! warning "Deprecated separate files"
The previous approach of using three separate configuration files
(`default_analyzer.yaml`, `default_recognizers.yaml`, `default.yaml`)
is deprecated. Use the unified `analyzer.yaml` file instead.
In this example, we'll tweak the configuration to reduce the number of predefinedrecognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish).
```python
@@ -31,7 +33,7 @@ In this example we're going to create the yaml as a string for illustration purp
### General Analyzer parameters
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml))
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml))
```python
analyzer_config_yaml = """
@@ -44,7 +46,7 @@ default_score_threshold: 0.4
### Recognizer Registry parameters
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml))
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml))
```python
@@ -121,7 +123,7 @@ recognizer_registry:
### NLP Engine parameters
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml))
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/analyzer.yaml))
```python
nlp_engine_yaml = """

12
presidio-analyzer/Dockerfile
View File

@@ -1,21 +1,16 @@
FROM python:3.12-slim@sha256:804ddf3251a60bbf9c92e73b7566c40428d54d0e79d3428194edf40da6521286
ARG NLP_CONF_FILE=presidio_analyzer/conf/default.yaml
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/default_analyzer.yaml
ARG RECOGNIZER_REGISTRY_CONF_FILE=presidio_analyzer/conf/default_recognizers.yaml
# Analyzer configuration file.
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/analyzer.yaml
ENV PIP_NO_CACHE_DIR=1
ENV POETRY_VIRTUALENVS_CREATE=false
ENV ANALYZER_CONF_FILE=${ANALYZER_CONF_FILE}
ENV RECOGNIZER_REGISTRY_CONF_FILE=${RECOGNIZER_REGISTRY_CONF_FILE}
ENV NLP_CONF_FILE=${NLP_CONF_FILE}
ENV PORT=3000
ENV WORKERS=1
COPY ${ANALYZER_CONF_FILE} /app/${ANALYZER_CONF_FILE}
COPY ${RECOGNIZER_REGISTRY_CONF_FILE} /app/${RECOGNIZER_REGISTRY_CONF_FILE}
COPY ${NLP_CONF_FILE} /app/${NLP_CONF_FILE}
WORKDIR /app
@@ -30,11 +25,10 @@ RUN pip install poetry==2.3.2 \
&& poetry install --no-root --only=main -E server \
&& rm -rf $(poetry config cache-dir)
# install nlp models specified in NLP_CONF_FILE or via nlp_configuration in ANALYZER_CONF_FILE
# install nlp models from the unified analyzer configuration
COPY ./install_nlp_models.py /app/
RUN poetry run python install_nlp_models.py \
--conf_file ${NLP_CONF_FILE} \
--analyzer_conf_file ${ANALYZER_CONF_FILE}
COPY . /app/

7
presidio-analyzer/Dockerfile.dev
View File

@@ -2,15 +2,12 @@ FROM python:3.12-slim@sha256:804ddf3251a60bbf9c92e73b7566c40428d54d0e79d3428194e
ARG DEV_MODE=dev
ARG POETRY_EXTRAS=""
ARG NLP_CONF_FILE=presidio_analyzer/conf/default.yaml
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/default_analyzer.yaml
ARG RECOGNIZER_REGISTRY_CONF_FILE=presidio_analyzer/conf/default_recognizers.yaml
# Unified analyzer configuration file.
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/analyzer.yaml
ENV DEV_MODE=${DEV_MODE}
ENV PIP_NO_CACHE_DIR=1
ENV ANALYZER_CONF_FILE=${ANALYZER_CONF_FILE}
ENV RECOGNIZER_REGISTRY_CONF_FILE=${RECOGNIZER_REGISTRY_CONF_FILE}
ENV NLP_CONF_FILE=${NLP_CONF_FILE}
ENV POETRY_EXTRAS=${POETRY_EXTRAS}
# Install essential build tools

12
presidio-analyzer/Dockerfile.stanza
View File

@@ -1,21 +1,16 @@
FROM python:3.12-slim@sha256:804ddf3251a60bbf9c92e73b7566c40428d54d0e79d3428194edf40da6521286
ARG NLP_CONF_FILE=presidio_analyzer/conf/default.yaml
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/default_analyzer.yaml
ARG RECOGNIZER_REGISTRY_CONF_FILE=presidio_analyzer/conf/default_recognizers.yaml
# Unified analyzer configuration file.
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/analyzer.yaml
ENV PIP_NO_CACHE_DIR=1
ENV POETRY_VIRTUALENVS_CREATE=false
ENV ANALYZER_CONF_FILE=${ANALYZER_CONF_FILE}
ENV RECOGNIZER_REGISTRY_CONF_FILE=${RECOGNIZER_REGISTRY_CONF_FILE}
ENV NLP_CONF_FILE=${NLP_CONF_FILE}
ENV PORT=3000
ENV WORKERS=1
COPY ${ANALYZER_CONF_FILE} /app/${ANALYZER_CONF_FILE}
COPY ${RECOGNIZER_REGISTRY_CONF_FILE} /app/${RECOGNIZER_REGISTRY_CONF_FILE}
COPY ${NLP_CONF_FILE} /app/${NLP_CONF_FILE}
WORKDIR /app
@@ -26,7 +21,7 @@ RUN apt-get update \
COPY ./pyproject.toml /app/
RUN pip install poetry==2.3.2 && poetry install --no-root --only=main -E server -E stanza
# install nlp models specified in NLP_CONF_FILE or via nlp_configuration in ANALYZER_CONF_FILE
# install nlp models from the unified analyzer configuration
COPY ./install_nlp_models.py /app/
# Set Stanza resource directory to a location that will be owned by the presidio user.
@@ -35,7 +30,6 @@ COPY ./install_nlp_models.py /app/
ENV STANZA_RESOURCES_DIR=/app/stanza_resources
RUN poetry run python install_nlp_models.py \
--conf_file ${NLP_CONF_FILE} \
--analyzer_conf_file ${ANALYZER_CONF_FILE}
COPY . /app/

12
presidio-analyzer/Dockerfile.transformers
View File

@@ -1,8 +1,7 @@
FROM python:3.12-slim@sha256:804ddf3251a60bbf9c92e73b7566c40428d54d0e79d3428194edf40da6521286
ARG NLP_CONF_FILE=presidio_analyzer/conf/transformers.yaml
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/default_analyzer.yaml
ARG RECOGNIZER_REGISTRY_CONF_FILE=presidio_analyzer/conf/default_recognizers.yaml
# Unified analyzer configuration file.
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/analyzer.yaml
ENV PIP_NO_CACHE_DIR=1
ENV POETRY_VIRTUALENVS_CREATE=false
ENV PORT=3000
@@ -11,12 +10,8 @@ ENV WORKERS=1
WORKDIR /app
ENV ANALYZER_CONF_FILE=${ANALYZER_CONF_FILE}
ENV RECOGNIZER_REGISTRY_CONF_FILE=${RECOGNIZER_REGISTRY_CONF_FILE}
ENV NLP_CONF_FILE=${NLP_CONF_FILE}
COPY ${ANALYZER_CONF_FILE} /app/${ANALYZER_CONF_FILE}
COPY ${RECOGNIZER_REGISTRY_CONF_FILE} /app/${RECOGNIZER_REGISTRY_CONF_FILE}
COPY ${NLP_CONF_FILE} /app/${NLP_CONF_FILE}
# Install curl for health checks
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
@@ -24,11 +19,10 @@ RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
COPY ./pyproject.toml /app/
RUN pip install poetry==2.3.2 && poetry install --no-root --only=main -E server -E transformers
# install nlp models specified in NLP_CONF_FILE or via nlp_configuration in ANALYZER_CONF_FILE
# install nlp models from the unified analyzer configuration
COPY ./install_nlp_models.py /app/
RUN poetry run python install_nlp_models.py \
--conf_file ${NLP_CONF_FILE} \
--analyzer_conf_file ${ANALYZER_CONF_FILE}
COPY . /app/

14
presidio-analyzer/Dockerfile.windows
View File

@@ -1,16 +1,13 @@
FROM python:3.12-windowsservercore@sha256:f2ab91b547b695c40a69cb1931cf382cb4019f56e1df5a83c86372d0cc8fb5a4
ARG NLP_CONF_FILE=presidio_analyzer/conf/default.yaml
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/default_analyzer.yaml
ARG RECOGNIZER_REGISTRY_CONF_FILE=presidio_analyzer/conf/default_recognizers.yaml
# Unified analyzer configuration file.
ARG ANALYZER_CONF_FILE=presidio_analyzer/conf/analyzer.yaml
ENV PIP_NO_CACHE_DIR=1
ENV POETRY_VIRTUALENVS_CREATE=false
ENV PORT=3000
WORKDIR /app
ENV ANALYZER_CONF_FILE=${ANALYZER_CONF_FILE}
ENV RECOGNIZER_REGISTRY_CONF_FILE=${RECOGNIZER_REGISTRY_CONF_FILE}
ENV NLP_CONF_FILE=${NLP_CONF_FILE}
RUN powershell -Command \
"Write-Host 'Downloading VC++ Redistributable...'; \
@@ -21,16 +18,13 @@ RUN powershell -Command \
Remove-Item 'vc_redist.x64.exe'"
COPY ${ANALYZER_CONF_FILE} /app/${ANALYZER_CONF_FILE}
COPY ${RECOGNIZER_REGISTRY_CONF_FILE} /app/${RECOGNIZER_REGISTRY_CONF_FILE}
COPY ${NLP_CONF_FILE} /app/${NLP_CONF_FILE}
COPY ./pyproject.toml /app/
RUN pip install poetry==2.3.2; poetry install --no-root --only=main -E server -E transformers
# install nlp models specified in NLP_CONF_FILE
# install nlp models specified in the analyzer config
COPY ./install_nlp_models.py .
COPY ${NLP_CONF_FILE} ${NLP_CONF_FILE}
RUN poetry run python install_nlp_models.py --conf_file $Env:NLP_CONF_FILE
RUN poetry run python install_nlp_models.py --analyzer_conf_file $Env:ANALYZER_CONF_FILE
COPY . .

25
presidio-analyzer/app.py
View File

@@ -3,6 +3,7 @@
import json
import logging
import os
import warnings
from logging.config import fileConfig
from pathlib import Path
from typing import Tuple
@@ -49,6 +50,23 @@ class Server:
os.environ.get("RECOGNIZER_REGISTRY_CONF_FILE") or None
)
if nlp_engine_conf_file:
warnings.warn(
"The NLP_CONF_FILE environment variable is deprecated. "
"Use the 'nlp_configuration' section inside the unified "
"ANALYZER_CONF_FILE instead.",
DeprecationWarning,
stacklevel=2,
)
if recognizer_registry_conf_file:
warnings.warn(
"The RECOGNIZER_REGISTRY_CONF_FILE environment variable is "
"deprecated. Use the 'recognizer_registry' section inside "
"the unified ANALYZER_CONF_FILE instead.",
DeprecationWarning,
stacklevel=2,
)
self.logger.info("Starting analyzer engine")
self.engine: AnalyzerEngine = AnalyzerEngineProvider(
analyzer_engine_conf_file=analyzer_conf_file,
@@ -86,7 +104,7 @@ class Server:
texts=batch,
batch_size=min(
len(batch),
int(os.environ.get("BATCH_SIZE", DEFAULT_BATCH_SIZE))
int(os.environ.get("BATCH_SIZE", DEFAULT_BATCH_SIZE)),
),
language=req_data.language,
correlation_id=req_data.correlation_id,
@@ -99,9 +117,8 @@ class Server:
allow_list_match=req_data.allow_list_match,
regex_flags=req_data.regex_flags,
n_process=min(
len(batch),
int(os.environ.get("N_PROCESS", DEFAULT_N_PROCESS))
)
len(batch), int(os.environ.get("N_PROCESS", DEFAULT_N_PROCESS))
),
)
results = []
for recognizer_result_list in iterator:

20
presidio-analyzer/install_nlp_models.py
View File

@@ -25,6 +25,18 @@ logger = logging.getLogger()
logger.setLevel("INFO")
logger.addHandler(logging.StreamHandler())
DEFAULT_ANALYZER_CONF_FILE = "presidio_analyzer/conf/analyzer.yaml"
DEFAULT_NLP_CONF_FILE = "presidio_analyzer/conf/default.yaml"
def _resolve_config_files(
nlp_conf_file: Optional[str], analyzer_conf_file: Optional[str]
) -> tuple[str, Optional[str]]:
"""Resolve CLI defaults while preserving explicit NLP config overrides."""
if analyzer_conf_file or nlp_conf_file:
return nlp_conf_file or DEFAULT_NLP_CONF_FILE, analyzer_conf_file
return DEFAULT_NLP_CONF_FILE, DEFAULT_ANALYZER_CONF_FILE
def install_models(
nlp_conf_file: Optional[str] = None, analyzer_conf_file: Optional[str] = None
@@ -36,6 +48,10 @@ def install_models(
:param analyzer_conf_file: Optional analyzer configuration file
which may include NLP configuration.
"""
nlp_conf_file, analyzer_conf_file = _resolve_config_files(
nlp_conf_file, analyzer_conf_file
)
# Prefer nlp_configuration embedded inside a unified ANALYZER_CONF_FILE.
if analyzer_conf_file:
try:
@@ -153,7 +169,7 @@ if __name__ == "__main__":
default=None,
help=(
"Optional path to an analyzer conf file which may include "
"NLP configuration. When provided, --nlp_conf_file is ignored."
"NLP configuration."
),
)
args = parser.parse_args()
@@ -174,6 +190,6 @@ if __name__ == "__main__":
args.nlp_conf_file = args.conf_file
install_models(
nlp_conf_file=args.nlp_conf_file or "presidio_analyzer/conf/default.yaml",
nlp_conf_file=args.nlp_conf_file,
analyzer_conf_file=args.analyzer_conf_file,
)

62
presidio-analyzer/presidio_analyzer/analyzer_engine_provider.py
View File

@@ -1,4 +1,5 @@
import logging
import warnings
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
@@ -16,12 +17,23 @@ class AnalyzerEngineProvider:
"""
Utility function for loading Presidio Analyzer.
Use this class to load presidio analyzer engine from a yaml file
Use this class to load presidio analyzer engine from a yaml file.
:param analyzer_engine_conf_file: the path to the analyzer configuration file
:param nlp_engine_conf_file: the path to the nlp engine configuration file
:param recognizer_registry_conf_file: the path to the recognizer
registry configuration file
The recommended approach is to use a single unified configuration file
(``analyzer.yaml``) that contains all settings: supported languages,
NLP engine, and recognizer registry.
The separate ``nlp_engine_conf_file`` and ``recognizer_registry_conf_file``
parameters are deprecated and will be removed in a future version.
:param analyzer_engine_conf_file: the path to the analyzer configuration file.
Defaults to the built-in ``conf/analyzer.yaml``.
:param nlp_engine_conf_file: (deprecated) the path to the nlp engine
configuration file. Use the ``nlp_configuration`` section inside the
unified analyzer configuration file instead.
:param recognizer_registry_conf_file: (deprecated) the path to the recognizer
registry configuration file. Use the ``recognizer_registry`` section inside
the unified analyzer configuration file instead.
"""
def __init__(
@@ -34,8 +46,26 @@ class AnalyzerEngineProvider:
ConfigurationValidator.validate_file_path(analyzer_engine_conf_file)
if nlp_engine_conf_file:
ConfigurationValidator.validate_file_path(nlp_engine_conf_file)
warnings.warn(
"The 'nlp_engine_conf_file' parameter is deprecated and will be "
"removed in a future version. Use the 'nlp_configuration' section "
"inside the unified analyzer configuration file instead. "
"See: https://microsoft.github.io/presidio/analyzer/"
"analyzer_engine_provider/",
DeprecationWarning,
stacklevel=2,
)
if recognizer_registry_conf_file:
ConfigurationValidator.validate_file_path(recognizer_registry_conf_file)
warnings.warn(
"The 'recognizer_registry_conf_file' parameter is deprecated and "
"will be removed in a future version. Use the 'recognizer_registry' "
"section inside the unified analyzer configuration file instead. "
"See: https://microsoft.github.io/presidio/analyzer/"
"analyzer_engine_provider/",
DeprecationWarning,
stacklevel=2,
)
self.configuration = self.get_configuration(conf_file=analyzer_engine_conf_file)
self.nlp_engine_conf_file = nlp_engine_conf_file
@@ -76,6 +106,26 @@ class AnalyzerEngineProvider:
ConfigurationValidator.validate_analyzer_configuration(configuration)
logger.debug("Analyzer configuration validation passed")
# Detect deprecated partial configuration format
# (only supported_languages & default_score_threshold, no nlp/registry)
if (
conf_file
and "nlp_configuration" not in configuration
and "recognizer_registry" not in configuration
):
warnings.warn(
f"Configuration file '{conf_file}' uses the deprecated YAML "
f"config format with a separate file for each module "
f"(analyzer, NLP engine, and recognizer registry). Migrate "
f"to the unified analyzer configuration file format that "
f"includes supported_languages, default_score_threshold, "
f"nlp_configuration, and recognizer_registry in one file. "
f"See: https://microsoft.github.io/presidio/analyzer/"
f"analyzer_engine_provider/",
DeprecationWarning,
stacklevel=2,
)
return configuration
def create_engine(self) -> AnalyzerEngine:
@@ -175,7 +225,7 @@ class AnalyzerEngineProvider:
@staticmethod
def _get_full_conf_path(
default_conf_file: Union[Path, str] = "default_analyzer.yaml",
default_conf_file: Union[Path, str] = "analyzer.yaml",
) -> Path:
"""Return a Path to the default conf file."""
return Path(Path(__file__).parent, "conf", default_conf_file)

448
presidio-analyzer/presidio_analyzer/conf/analyzer.yaml Normal file
View File

@@ -0,0 +1,448 @@
# ---------------------------------------------------------
# Presidio Analyzer – Unified Configuration
# ---------------------------------------------------------
#
# This is the single configuration file for Presidio Analyzer.
# It replaces three previously separate configuration files:
#
# default_analyzer.yaml → supported_languages, default_score_threshold
# default.yaml → nlp_configuration
# default_recognizers.yaml → recognizer_registry
#
# Those files are now deprecated; use this file instead.
#
# Load with:
# from presidio_analyzer import AnalyzerEngineProvider
# provider = AnalyzerEngineProvider(analyzer_engine_conf_file="analyzer.yaml")
# engine = provider.create_engine()
#
# Or via Pydantic model:
# from presidio_analyzer.configuration import AnalyzerConfiguration
# config = AnalyzerConfiguration.from_yaml("analyzer.yaml")
# ---------------------------------------------------------
supported_languages:
- en
default_score_threshold: 0.0
# ---------------------------------------------------------
# NLP Engine Configuration
# ---------------------------------------------------------
# Defines the NLP engine used for tokenization, lemmatization,
# and (optionally) named-entity recognition.
# Supported engines: spacy, stanza, transformers.
nlp_configuration:
nlp_engine_name: spacy
models:
- lang_code: en
model_name: en_core_web_lg
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
GPE: LOCATION
LOCATION: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
low_confidence_score_multiplier: 0.4
low_score_entity_names: []
labels_to_ignore:
- ORGANIZATION
- CARDINAL
- EVENT
- LANGUAGE
- LAW
- MONEY
- ORDINAL
- PERCENT
- PRODUCT
- QUANTITY
- WORK_OF_ART
# ---------------------------------------------------------
# Recognizer Registry
# ---------------------------------------------------------
recognizer_registry:
global_regex_flags: 26
recognizers:
# -------------------------------------------------------------------
# Recognizers listed here can be:
# - predefined: loaded from code-defined recognizer classes
# - custom: pattern/deny-list recognizers defined entirely in YAML
#
# For predefined recognizers:
# - Provide the recognizer 'name' matching the class name
# - Use 'class_name' when the display name differs from the class
# - Omitted parameters use the class defaults
# - Set enabled: false to disable a recognizer
#
# For custom recognizers:
# - Provide 'patterns' and/or 'deny_list'
# - Must specify 'supported_entity'
#
# Multi-language recognizers create one instance per language.
# -------------------------------------------------------------------
# ----- Global recognizers (language-independent) -----
- name: CreditCardRecognizer
type: predefined
supported_languages:
- language: en
context: [credit, card, visa, mastercard, cc, amex, discover, jcb, diners, maestro, instapayment]
- language: es
context: [tarjeta, credito, visa, mastercard, cc, amex, discover, jcb, diners, maestro, instapayment]
- language: it
- language: pl
- name: CryptoRecognizer
type: predefined
- name: DateRecognizer
type: predefined
- name: EmailRecognizer
type: predefined
- name: IbanRecognizer
type: predefined
- name: IpRecognizer
type: predefined
- name: MedicalLicenseRecognizer
type: predefined
- name: MacAddressRecognizer
type: predefined
- name: PhoneRecognizer
type: predefined
- name: UrlRecognizer
type: predefined
# ----- US recognizers -----
- name: UsBankRecognizer
type: predefined
supported_languages:
- en
- name: UsLicenseRecognizer
type: predefined
supported_languages:
- en
- name: UsItinRecognizer
type: predefined
supported_languages:
- en
- name: UsPassportRecognizer
type: predefined
supported_languages:
- en
- name: UsSsnRecognizer
type: predefined
supported_languages:
- en
- name: UsMbiRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: UsNpiRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- UK recognizers -----
- name: NhsRecognizer
type: predefined
supported_languages:
- en
- name: UkNinoRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: UkPassportRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: UkPostcodeRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: UkVehicleRegistrationRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- Singapore recognizers -----
- name: SgFinRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- Australia recognizers -----
- name: AuAbnRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: AuAcnRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: AuTfnRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: AuMedicareRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- Nigeria recognizers -----
- name: NgNinRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: NgVehicleRegistrationRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- India recognizers -----
- name: InPanRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: InAadhaarRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: InVehicleRegistrationRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: InPassportRecognizer
type: predefined
supported_languages:
- en
enabled: false
- name: InVoterRecognizer
type: predefined
enabled: false
- name: InGstinRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- Spain recognizers -----
- name: EsNifRecognizer
type: predefined
supported_languages:
- es
- name: EsNieRecognizer
type: predefined
supported_languages:
- es
# ----- Italy recognizers -----
- name: ItDriverLicenseRecognizer
type: predefined
supported_languages:
- it
- name: ItFiscalCodeRecognizer
type: predefined
supported_languages:
- it
- name: ItVatCodeRecognizer
type: predefined
supported_languages:
- it
- name: ItIdentityCardRecognizer
type: predefined
supported_languages:
- it
- name: ItPassportRecognizer
type: predefined
supported_languages:
- it
# ----- Poland recognizers -----
- name: PlPeselRecognizer
type: predefined
supported_languages:
- pl
# ----- South Korea recognizers -----
- name: KrBrnRecognizer
type: predefined
supported_languages:
- ko
- kr
enabled: false
- name: KrRrnRecognizer
type: predefined
supported_languages:
- ko
- kr
enabled: false
- name: KrDriverLicenseRecognizer
type: predefined
supported_languages:
- ko
- kr
enabled: false
- name: KrFrnRecognizer
type: predefined
supported_languages:
- ko
- kr
enabled: false
# ----- Sweden recognizers -----
- name: SeOrganisationsnummerRecognizer
type: predefined
supported_languages:
- sv
enabled: false
- name: SePersonnummerRecognizer
type: predefined
supported_languages:
- sv
enabled: false
# ----- Thailand recognizers -----
- name: ThTninRecognizer
type: predefined
supported_languages:
- th
enabled: false
# ----- Germany recognizers -----
- name: DeTaxIdRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DeTaxNumberRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DePassportRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DeIdCardRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DeSocialSecurityRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DeHealthInsuranceRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DeKfzRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DeHandelsregisterRecognizer
type: predefined
supported_languages:
- de
enabled: false
- name: DePlzRecognizer
type: predefined
supported_languages:
- de
enabled: false
# ----- HuggingFace NER (disabled by default) -----
- name: HuggingFaceNerRecognizer
type: predefined
supported_languages:
- en
enabled: false
# ----- LangExtract (disabled by default) -----
- name: BasicLangExtractRecognizer
type: predefined
supported_languages:
- en
enabled: false
config_path: presidio_analyzer/conf/langextract_config_basic.yaml
# ----- Canada recognizers -----
- name: CaSinRecognizer
type: predefined
supported_languages:
- en
- fr
enabled: false

11
presidio-analyzer/presidio_analyzer/conf/default.yaml
View File

@@ -1,3 +1,14 @@
# ---------------------------------------------------------
# DEPRECATED – Use the nlp_configuration section in
# conf/analyzer.yaml instead.
#
# This standalone NLP configuration file is kept for
# backward compatibility only.
# It will be removed in a future version of Presidio.
#
# See: https://microsoft.github.io/presidio/analyzer/analyzer_engine_provider/
# ---------------------------------------------------------
nlp_engine_name: spacy
models:
-

13
presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml
View File

@@ -1,3 +1,16 @@
# ---------------------------------------------------------
# DEPRECATED – Use conf/analyzer.yaml instead.
#
# This file is kept for backward compatibility only.
# It will be removed in a future version of Presidio.
#
# Migrate to the unified configuration file (analyzer.yaml)
# which contains supported_languages, default_score_threshold,
# nlp_configuration, and recognizer_registry in a single file.
#
# See: https://microsoft.github.io/presidio/analyzer/analyzer_engine_provider/
# ---------------------------------------------------------
supported_languages:
- en
default_score_threshold: 0

11
presidio-analyzer/presidio_analyzer/conf/default_analyzer_full.yaml
View File

@@ -1,3 +1,14 @@
# ---------------------------------------------------------
# DEPRECATED – Use conf/analyzer.yaml instead.
#
# This is an older unified configuration file kept for
# backward compatibility only. The canonical unified
# configuration is now conf/analyzer.yaml, which contains
# the complete and up-to-date set of recognizers.
#
# See: https://microsoft.github.io/presidio/analyzer/analyzer_engine_provider/
# ---------------------------------------------------------
supported_languages:
- en
default_score_threshold: 0

11
presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml
View File

@@ -1,3 +1,14 @@
# ---------------------------------------------------------
# DEPRECATED – Use the recognizer_registry section in
# conf/analyzer.yaml instead.
#
# This standalone recognizer registry configuration file
# is kept for backward compatibility only.
# It will be removed in a future version of Presidio.
#
# See: https://microsoft.github.io/presidio/analyzer/analyzer_engine_provider/
# ---------------------------------------------------------
supported_languages:
- en
global_regex_flags: 26

13
presidio-analyzer/presidio_analyzer/configuration/__init__.py Normal file
View File

@@ -0,0 +1,13 @@
"""Unified configuration module for Presidio Analyzer."""
from presidio_analyzer.configuration.analyzer_configuration import (
AnalyzerConfiguration,
NlpConfiguration,
NlpModelConfig,
)
__all__ = [
"AnalyzerConfiguration",
"NlpConfiguration",
"NlpModelConfig",
]

328
presidio-analyzer/presidio_analyzer/configuration/analyzer_configuration.py Normal file
View File

@@ -0,0 +1,328 @@
"""Unified Pydantic configuration models for Presidio Analyzer.
This module defines the single-file configuration schema for the analyzer.
It consolidates what was previously spread across three separate files:
- ``default_analyzer.yaml`` (engine settings)
- ``default.yaml`` (NLP engine configuration)
- ``default_recognizers.yaml`` (recognizer registry)
Usage::
from presidio_analyzer.configuration import AnalyzerConfiguration
# Load from YAML
config = AnalyzerConfiguration.from_yaml("analyzer.yaml")
# Load with defaults
config = AnalyzerConfiguration()
# Create from dict
config = AnalyzerConfiguration(**my_dict)
"""
import logging
import warnings
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import yaml
from pydantic import (
BaseModel,
ConfigDict,
Field,
ValidationError,
field_validator,
model_validator,
)
from presidio_analyzer.input_validation import (
RecognizerRegistryConfig,
validate_language_codes,
)
from presidio_analyzer.nlp_engine import NerModelConfiguration
logger = logging.getLogger("presidio-analyzer")
class NlpModelConfig(BaseModel):
"""Configuration for a single NLP model.
:param lang_code: Language code (e.g., 'en', 'es').
:param model_name: Model name or dict mapping engine names to model names.
For spaCy/Stanza: a string (e.g., ``en_core_web_lg``).
For Transformers: a dict (e.g.,
``{"spacy": "en_core_web_sm", "transformers": "model_name"}``).
"""
lang_code: str = Field(..., description="Language code (e.g., 'en', 'es')")
model_name: Union[str, Dict[str, str]] = Field(
...,
description="Model name string or dict mapping engine to model name",
)
@field_validator("lang_code")
@classmethod
def validate_lang_code(cls, v: str) -> str:
"""Validate language code format."""
validate_language_codes([v])
return v
class NlpConfiguration(BaseModel):
"""NLP engine configuration.
Defines which NLP engine and models to use for tokenization,
lemmatization, and (optionally) NER.
:param nlp_engine_name: Name of the NLP engine
(``spacy``, ``stanza``, ``transformers``, or ``slim``).
:param models: List of NLP models, one per language.
:param ner_model_configuration: Optional NER model mapping and tuning.
:param generic_tokenizer: Optional generic tokenizer for the slim engine.
"""
nlp_engine_name: str = Field(
default="spacy",
description="NLP engine name: 'spacy', 'stanza', 'transformers', or 'slim'",
)
models: List[NlpModelConfig] = Field(
default_factory=lambda: [
NlpModelConfig(lang_code="en", model_name="en_core_web_lg")
],
description="List of NLP models, one per language",
)
ner_model_configuration: Optional[NerModelConfiguration] = Field(
default=None,
description="NER model configuration for entity mapping and tuning",
)
generic_tokenizer: Optional[str] = Field(
default=None,
description="Generic tokenizer name for the slim NLP engine",
)
model_config = ConfigDict(arbitrary_types_allowed=True)
@field_validator("nlp_engine_name")
@classmethod
def validate_engine_name(cls, v: str) -> str:
"""Validate NLP engine name."""
valid_engines = {"spacy", "stanza", "transformers", "slim"}
if v not in valid_engines:
raise ValueError(
f"Invalid NLP engine name: '{v}'. "
f"Must be one of: {sorted(valid_engines)}"
)
return v
@field_validator("models")
@classmethod
def validate_models_not_empty(cls, v: List[NlpModelConfig]) -> List[NlpModelConfig]:
"""Validate at least one model is provided."""
if not v:
raise ValueError("At least one NLP model must be configured")
return v
@model_validator(mode="after")
def validate_no_duplicate_languages(self) -> "NlpConfiguration":
"""Ensure no duplicate lang_code entries in models."""
lang_codes = [m.lang_code for m in self.models]
duplicates = [lc for lc in lang_codes if lang_codes.count(lc) > 1]
if duplicates:
raise ValueError(
f"Duplicate lang_code entries in NLP models: "
f"{sorted(set(duplicates))}. "
f"Each language should have exactly one model."
)
return self
def to_provider_dict(self) -> Dict[str, Any]:
"""Convert to the dict format expected by NlpEngineProvider.
:return: Dictionary compatible with NlpEngineProvider.
"""
result: Dict[str, Any] = {
"nlp_engine_name": self.nlp_engine_name,
"models": [m.model_dump() for m in self.models],
}
if self.ner_model_configuration:
result["ner_model_configuration"] = self.ner_model_configuration.to_dict()
if self.generic_tokenizer:
result["generic_tokenizer"] = self.generic_tokenizer
return result
class AnalyzerConfiguration(BaseModel):
"""Unified Presidio Analyzer configuration.
Single Pydantic model that consolidates all analyzer configuration:
engine settings, NLP configuration, and recognizer registry.
:param supported_languages: Languages the analyzer supports.
:param default_score_threshold: Minimum confidence score (0.0–1.0)
for returning results.
:param nlp_configuration: NLP engine and model configuration.
:param recognizer_registry: Recognizer registry configuration
(recognizers, regex flags).
Example YAML::
supported_languages:
- en
default_score_threshold: 0.0
nlp_configuration:
nlp_engine_name: spacy
models:
- lang_code: en
model_name: en_core_web_lg
recognizer_registry:
global_regex_flags: 26
recognizers:
- name: CreditCardRecognizer
type: predefined
"""
supported_languages: List[str] = Field(
default=["en"],
description="Languages supported by this analyzer instance",
)
default_score_threshold: float = Field(
default=0.0,
ge=0.0,
le=1.0,
description="Minimum confidence score for entity detection",
)
nlp_configuration: Optional[NlpConfiguration] = Field(
default=None,
description="NLP engine and model configuration",
)
recognizer_registry: Optional[RecognizerRegistryConfig] = Field(
default=None,
description="Recognizer registry configuration",
)
model_config = ConfigDict(
extra="forbid",
arbitrary_types_allowed=True,
)
@model_validator(mode="before")
@classmethod
def propagate_languages_to_registry(cls, data: Any) -> Any:
"""Inject supported_languages into recognizer_registry if missing.
The top-level ``supported_languages`` applies to the entire analyzer,
including the recognizer registry. When the registry section omits
``supported_languages``, copy the top-level value so that recognizer
validation sees the correct language list.
"""
if isinstance(data, dict):
languages = data.get("supported_languages")
registry = data.get("recognizer_registry")
if (
isinstance(registry, dict)
and "supported_languages" not in registry
and languages
):
registry["supported_languages"] = languages
return data
@field_validator("supported_languages")
@classmethod
def validate_languages(cls, v: List[str]) -> List[str]:
"""Validate language code formats."""
if not v:
raise ValueError("At least one supported language must be specified")
validate_language_codes(v)
return v
@model_validator(mode="after")
def validate_nlp_covers_languages(self) -> "AnalyzerConfiguration":
"""Warn if NLP models don't cover all supported languages."""
if self.nlp_configuration and self.nlp_configuration.models:
nlp_langs = {m.lang_code for m in self.nlp_configuration.models}
configured_langs = set(self.supported_languages)
missing = configured_langs - nlp_langs
if missing:
logger.warning(
f"NLP models do not cover all supported languages. "
f"Missing: {sorted(missing)}. "
f"Configured NLP languages: {sorted(nlp_langs)}. "
f"Analysis for these languages will not use NLP features."
)
return self
@classmethod
def from_yaml(cls, file_path: Union[str, Path]) -> "AnalyzerConfiguration":
"""Load and validate configuration from a YAML file.
:param file_path: Path to the YAML configuration file.
:return: Validated AnalyzerConfiguration instance.
:raises FileNotFoundError: If the file does not exist.
:raises ValueError: If the YAML content is invalid.
"""
path = Path(file_path)
if not path.exists():
raise FileNotFoundError(f"Configuration file not found: {path}")
if not path.is_file():
raise ValueError(f"Path is not a file: {path}")
try:
with open(path) as f:
raw = yaml.safe_load(f)
except yaml.YAMLError as e:
raise ValueError(f"Invalid YAML in configuration file {path}: {e}") from e
if raw is None:
raise ValueError(f"Configuration file is empty: {path}")
if not isinstance(raw, dict):
raise ValueError(
f"Configuration file must contain a YAML mapping, "
f"got {type(raw).__name__}: {path}"
)
# Detect and warn about deprecated separate-file format keys
deprecated_top_keys = {"nlp_engine_name", "models"}
found_deprecated = deprecated_top_keys & set(raw.keys())
if found_deprecated:
warnings.warn(
f"Configuration file '{path}' appears to use the deprecated "
f"standalone NLP configuration format "
f"(found top-level keys: {sorted(found_deprecated)}). "
f"Please migrate to the unified analyzer configuration format. "
f"See: https://microsoft.github.io/presidio/analyzer/"
f"analyzer_engine_provider/",
DeprecationWarning,
stacklevel=2,
)
try:
return cls(**raw)
except ValidationError as e:
raise ValueError(f"Invalid analyzer configuration in {path}") from e
@classmethod
def from_dict(cls, config: Dict[str, Any]) -> "AnalyzerConfiguration":
"""Create configuration from a dictionary.
:param config: Configuration dictionary.
:return: Validated AnalyzerConfiguration instance.
"""
return cls(**config)
@staticmethod
def get_default_conf_path() -> Path:
"""Return the path to the default unified configuration file.
:return: Path to ``conf/analyzer.yaml``.
"""
return Path(Path(__file__).parent, "../conf", "analyzer.yaml")
def to_dict(self) -> Dict[str, Any]:
"""Serialize configuration to a dictionary.
:return: Dictionary representation of the configuration.
"""
return self.model_dump(exclude_none=True)

10
presidio-analyzer/presidio_analyzer/nlp_engine/nlp_engine_provider.py
View File

@@ -1,4 +1,5 @@
import logging
import warnings
from pathlib import Path
from typing import Dict, Optional, Tuple, Union
@@ -68,6 +69,15 @@ class NlpEngineProvider:
if conf_file == "":
raise ValueError("conf_file is empty")
ConfigurationValidator.validate_file_path(conf_file)
warnings.warn(
"Loading NLP configuration from a standalone file is deprecated. "
"Use the 'nlp_configuration' section inside the unified analyzer "
"configuration file (analyzer.yaml) instead. "
"See: https://microsoft.github.io/presidio/analyzer/"
"analyzer_engine_provider/",
DeprecationWarning,
stacklevel=2,
)
self.nlp_configuration = self._read_nlp_conf(conf_file)
if conf_file is None and nlp_configuration is None:

52
presidio-analyzer/presidio_analyzer/recognizer_registry/recognizers_loader_utils.py
View File

@@ -2,6 +2,7 @@ from __future__ import annotations
import inspect
import logging
import warnings
from collections.abc import ItemsView
from pathlib import Path
from typing import (
@@ -451,6 +452,33 @@ class RecognizerConfigurationLoader:
# Validation is now handled by Pydantic via ConfigurationValidator
return registry_configuration
@staticmethod
def _extract_recognizer_registry_configuration(configuration: Any) -> Any:
"""Extract recognizer registry settings from unified analyzer config."""
if (
not isinstance(configuration, dict)
or "recognizer_registry" not in configuration
):
return configuration
recognizer_registry = configuration["recognizer_registry"]
if not isinstance(recognizer_registry, dict):
raise TypeError(
"The 'recognizer_registry' section should be a valid mapping, "
f"got {type(recognizer_registry)}"
)
registry_configuration = recognizer_registry.copy()
if (
"supported_languages" not in registry_configuration
and "supported_languages" in configuration
):
registry_configuration["supported_languages"] = configuration[
"supported_languages"
]
return registry_configuration
@staticmethod
def get(
conf_file: Optional[Union[Path, str]] = None,
@@ -484,6 +512,15 @@ class RecognizerConfigurationLoader:
use_defaults = False
if conf_file:
warnings.warn(
"Loading recognizer registry from a standalone file is deprecated. "
"Use the 'recognizer_registry' section inside the unified analyzer "
"configuration file (analyzer.yaml) instead. "
"See: https://microsoft.github.io/presidio/analyzer/"
"analyzer_engine_provider/",
DeprecationWarning,
stacklevel=3,
)
try:
with open(conf_file) as file:
config_from_file = yaml.safe_load(file)
@@ -500,11 +537,22 @@ class RecognizerConfigurationLoader:
except Exception as e:
raise ValueError(f"Failed to parse file {conf_file}. Error: {str(e)}")
config_from_file = (
RecognizerConfigurationLoader._extract_recognizer_registry_configuration(
config_from_file
)
)
# Load defaults if needed (no config provided,
# or registry_configuration is incomplete)
if use_defaults:
with open(RecognizerConfigurationLoader._get_full_conf_path()) as file:
config_from_file = yaml.safe_load(file)
config_from_file = (
RecognizerConfigurationLoader._extract_recognizer_registry_configuration(
config_from_file
)
)
if config_from_file and not isinstance(config_from_file, dict):
raise TypeError(
@@ -543,7 +591,7 @@ class RecognizerConfigurationLoader:
@staticmethod
def _get_full_conf_path(
default_conf_file: Union[Path, str] = "default_recognizers.yaml",
default_conf_file: Union[Path, str] = "analyzer.yaml",
) -> Path:
"""Return a Path to the default conf file."""
"""Return a Path to the built-in configuration file."""
return Path(Path(__file__).parent, "../conf", default_conf_file)

20
presidio-analyzer/tests/test_analyzer_configuration.py Normal file
View File

@@ -0,0 +1,20 @@
import pytest
from presidio_analyzer.configuration import AnalyzerConfiguration
def test_from_yaml_wraps_invalid_yaml_as_value_error(tmp_path):
"""Test invalid YAML parser errors are exposed as ValueError."""
config_file = tmp_path / "invalid.yaml"
config_file.write_text("invalid: [unclosed", encoding="utf-8")
with pytest.raises(ValueError, match="Invalid YAML"):
AnalyzerConfiguration.from_yaml(config_file)
def test_from_yaml_wraps_validation_errors_as_value_error(tmp_path):
"""Test invalid analyzer schemas are exposed as ValueError."""
config_file = tmp_path / "invalid_analyzer.yaml"
config_file.write_text("unknown_key: value\n", encoding="utf-8")
with pytest.raises(ValueError, match="Invalid analyzer configuration"):
AnalyzerConfiguration.from_yaml(config_file)

2
presidio-analyzer/tests/test_analyzer_engine_provider.py
View File

@@ -392,7 +392,7 @@ def test_analyzer_engine_provider_get_full_conf_path():
path = AnalyzerEngineProvider._get_full_conf_path()
assert isinstance(path, Path)
assert path.name == "default_analyzer.yaml"
assert path.name == "analyzer.yaml"
assert path.exists()

35
presidio-analyzer/tests/test_install_nlp_models.py Normal file
View File

@@ -0,0 +1,35 @@
from install_nlp_models import (
DEFAULT_ANALYZER_CONF_FILE,
DEFAULT_NLP_CONF_FILE,
_resolve_config_files,
)
def test_resolve_config_files_defaults_to_unified_analyzer_config():
"""Test no explicit config defaults to unified analyzer config."""
nlp_conf_file, analyzer_conf_file = _resolve_config_files(
nlp_conf_file=None, analyzer_conf_file=None
)
assert nlp_conf_file == DEFAULT_NLP_CONF_FILE
assert analyzer_conf_file == DEFAULT_ANALYZER_CONF_FILE
def test_resolve_config_files_preserves_explicit_nlp_config_without_analyzer_default():
"""Test explicit NLP config is not overridden by default analyzer config."""
nlp_conf_file, analyzer_conf_file = _resolve_config_files(
nlp_conf_file="custom-nlp.yaml", analyzer_conf_file=None
)
assert nlp_conf_file == "custom-nlp.yaml"
assert analyzer_conf_file is None
def test_resolve_config_files_adds_legacy_nlp_fallback_for_analyzer_config():
"""Test analyzer configs keep the legacy NLP fallback config."""
nlp_conf_file, analyzer_conf_file = _resolve_config_files(
nlp_conf_file=None, analyzer_conf_file="custom-analyzer.yaml"
)
assert nlp_conf_file == DEFAULT_NLP_CONF_FILE
assert analyzer_conf_file == "custom-analyzer.yaml"

48
presidio-analyzer/tests/test_recognizers_loader_utils.py
View File

@@ -2,6 +2,7 @@ import functools
import re
import pytest
import yaml
from presidio_analyzer.recognizer_registry.recognizers_loader_utils import (
RecognizerConfigurationLoader,
RecognizerListLoader,
@@ -157,6 +158,53 @@ def test_configuration_loader_bad_yaml_raises_value_error(tmp_path):
RecognizerConfigurationLoader.get(conf_file=str(f))
def test_configuration_loader_defaults_to_unified_analyzer_config():
"""Test that default registry config comes from unified analyzer.yaml."""
configuration = RecognizerConfigurationLoader.get()
analyzer_conf_path = RecognizerConfigurationLoader._get_full_conf_path()
with open(analyzer_conf_path) as file:
analyzer_config = yaml.safe_load(file)
assert analyzer_conf_path.name == "analyzer.yaml"
assert (
configuration["supported_languages"] == analyzer_config["supported_languages"]
)
assert (
configuration["global_regex_flags"]
== analyzer_config["recognizer_registry"]["global_regex_flags"]
)
assert (
configuration["recognizers"]
== analyzer_config["recognizer_registry"]["recognizers"]
)
def test_configuration_loader_extracts_registry_from_unified_config_file(tmp_path):
"""Test that a unified analyzer config file can provide registry config."""
analyzer_yaml = tmp_path / "analyzer.yaml"
analyzer_yaml.write_text(
"""
supported_languages:
- en
recognizer_registry:
global_regex_flags: 26
recognizers:
- CreditCardRecognizer
""",
encoding="utf-8",
)
with pytest.warns(DeprecationWarning):
configuration = RecognizerConfigurationLoader.get(conf_file=analyzer_yaml)
assert configuration == {
"supported_languages": ["en"],
"global_regex_flags": 26,
"recognizers": ["CreditCardRecognizer"],
}
def test_convert_supported_entities_to_entity_uses_first_item():
"""Test that supported_entities list is converted to single supported_entity."""
conf = {"supported_entities": ["ENT1", "ENT2"]}