* Gpu first draft * Gliner flow adapted to GPU * deleted testing files * fix: Suppress warnings from spacy_huggingface_pipelines and other libraries * stanza mode added. * ruff issues * adding to to the mock * refactor: remove unused GPU result files and update device detector usage * fix: reorder device_detector import for consistency * docs added around gpu * test: enhance GPU detection tests for DeviceDetector and SpacyNlpEngine * Add comprehensive error path mocking for device_detector and spacy GPU config tests * remove benchmark script. * Enhance device detection to support MPS and update related tests and configurations * Refactor device detection logic to improve clarity and logging for MPS initialization failures * Remove unnecessary comments regarding pipeline configuration in TransformersNlpEngine * Gpu first draft * Gliner flow adapted to GPU * deleted testing files * fix: Suppress warnings from spacy_huggingface_pipelines and other libraries * stanza mode added. * ruff issues * adding to to the mock * refactor: remove unused GPU result files and update device detector usage * fix: reorder device_detector import for consistency * docs added around gpu * test: enhance GPU detection tests for DeviceDetector and SpacyNlpEngine * Add comprehensive error path mocking for device_detector and spacy GPU config tests * remove benchmark script. * Enhance device detection to support MPS and update related tests and configurations * Refactor device detection logic to improve clarity and logging for MPS initialization failures * Remove unnecessary comments regarding pipeline configuration in TransformersNlpEngine * Update GPU acceleration instructions and remove deprecated dependencies * cleaning logs and comments * Enable GPU support for spaCy and transformers in NLP engines * Refactor GPU error logging in SpacyNlpEngine and remove unused import in TransformersNlpEngine * Update documentation and code to reflect MPS (Metal Performance Shaders) support is not available on macOS. Remove references to MPS in device detection and tests.
5.4 KiB
Getting started with text de-identification with Presidio
Presidio provides a simple way to de-identify text data by detecting and anonymizing personally identifiable information (PII). This guide shows you how to get started with text de-identification using Presidio's Python packages.
Note that Presidio can leverage different NLP packages to analyze text data. The default engine is based on spaCy, but you can also use others. This guide shows two examples: one using spaCy and the other using transformers.
Simple flow - Python package
Using Presidio's modules as Python packages to get started:
===+ "Anonymize PII in text (Default spaCy model)"
1. Install Presidio
```sh
pip install presidio-analyzer
pip install presidio-anonymizer
python -m spacy download en_core_web_lg
```
2. Analyze + Anonymize
```py
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
text="My phone number is 212-555-5555"
# Set up the engine, loads the NLP module (spaCy model by default)
# and other PII recognizers
analyzer = AnalyzerEngine()
# Call analyzer to get results
results = analyzer.analyze(text=text,
entities=["PHONE_NUMBER"],
language='en')
print(results)
# Analyzer results are passed to the AnonymizerEngine for anonymization
anonymizer = AnonymizerEngine()
anonymized_text = anonymizer.anonymize(text=text,analyzer_results=results)
print(anonymized_text)
```
=== "Anonymize PII in text (transformers)"
1. Install Presidio
```sh
pip install "presidio-analyzer[transformers]"
pip install presidio-anonymizer
python -m spacy download en_core_web_sm
```
2. Analyze + Anonymize
```py
from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import TransformersNlpEngine
from presidio_anonymizer import AnonymizerEngine
text = "My name is Don and my phone number is 212-555-5555"
# Define which transformers model to use
model_config = [{"lang_code": "en", "model_name": {
"spacy": "en_core_web_sm", # use a small spaCy model for lemmas, tokens etc.
"transformers": "dslim/bert-base-NER"
}
}]
nlp_engine = TransformersNlpEngine(models=model_config)
# Set up the engine, loads the NLP module (spaCy model by default)
# and other PII recognizers
analyzer = AnalyzerEngine(nlp_engine=nlp_engine)
# Call analyzer to get results
results = analyzer.analyze(text=text, language='en')
print(results)
# Analyzer results are passed to the AnonymizerEngine for anonymization
anonymizer = AnonymizerEngine()
anonymized_text = anonymizer.anonymize(text=text, analyzer_results=results)
print(anonymized_text)
```
!!! tip "Tip: Downloading models"
If not available, the transformers model and the spacy model would be downloaded on the first call to the `AnalyzerEngine`. To pre-download, see [this doc](../analyzer/nlp_engines/transformers.md#downloading-a-pre-trained-model).
=== "GPU Acceleration (Optional)"
For GPU acceleration, install the appropriate dependencies for your hardware:
- **Linux with NVIDIA GPU**: cupy-cuda12x (or the version matching your CUDA installation)
- **macOS with Apple Silicon**: MPS (Metal Performance Shaders) is currently not supported. The analyzer will use CPU for PyTorch operations.
Simple flow - Docker container
Presidio provides Docker containers that you can use to de-identify text data. Each module, analyzer, and anonymizer, has its own Docker container. The containers are available on Docker Hub.
- Download Docker images
docker pull mcr.microsoft.com/presidio-analyzer
docker pull mcr.microsoft.com/presidio-anonymizer
- Run containers
docker run -d -p 5002:3000 mcr.microsoft.com/presidio-analyzer:latest
docker run -d -p 5001:3000 mcr.microsoft.com/presidio-anonymizer:latest
- Use the API
curl -X POST http://localhost:5002/analyze \
-H "Content-Type: application/json" \
-d '{
"text": "My phone number is 555-123-4567.",
"language": "en"
}'
curl -X POST http://localhost:5001/anonymize -H "Content-Type: application/json" -d '
{
"text": "My phone number is 555-123-4567",
"anonymizers": {
"PHONE_NUMBER": {
"type": "replace",
"new_value": "--Redacted phone number--"
}
},
"analyzer_results": [
{
"start": 19,
"end": 31,
"score": 0.95,
"entity_type": "PHONE_NUMBER"
}
]}'