upstream-archive/presidio
1
0
Fork 0
mirror of https://github.com/data-privacy-stack/presidio.git synced 2026-07-26 04:40:54 -05:00
Code Issues Packages Projects Releases Wiki Activity

Two new samples + docs refactoring (#744)

Browse Source
This commit is contained in:
Omri Mendels
2021-07-14 12:17:35 +03:00
committed by GitHub
parent 577cbfe35b
commit 67cb019ff0
13 changed files with 843 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.pipelines/templates/validate-version.yml
View File

@@ -1,7 +1,7 @@
parameters:
- name: UPSTREAM_BRANCH
type: string
default: remotes/origin/master
default: remotes/origin/main
steps:
- task: Bash@3
displayName: 'Verify version change'

16
docs/analyzer/adding_recognizers.md
View File

@@ -1,6 +1,7 @@
# Supporting detection of new types of PII entities
Presidio can be extended to support detection of new types of PII entities, and to support additional languages.
These PII recognizers could be added **via code** or **ad-hoc as part of the request**.
## Introduction to recognizer development
@@ -112,6 +113,8 @@ To create a new recognizer via code:
2. Add it to the recognizer registry using `registry.add_recognizer(my_recognizer)`.
For more examples, see the [Customizing Presidio Analyzer](../samples/python/customizing_presidio_analyzer.ipynb) jupyter notebook.
### Creating a remote recognizer
A remote recognizer is an `EntityRecognizer` object interacting with an external service. The external service could be a 3rd party PII detection service or a custom service deployed in parallel to Presidio.
@@ -133,7 +136,7 @@ To add a recognizer to the list of pre-defined recognizers:
3. Add the recognizer to the `recognizers_map` dict in the `RecognizerRegistry.load_predefined_recognizers` method. In this map, the key is the language the recognizer supports, and the value is the class itself. If your recognizer detects entities in multiple languages, add it to under the "ALL" key.
4. Optional: Update documentation (e.g., the [supported entities list](../supported_entities.md)).
## Azure Text Analytics recognizer
### Azure Text Analytics recognizer
On how to integrate Presidio with Azure Text Analytics,
and a sample for a Text Analytics Remote Recognizer, refer to the
@@ -195,10 +198,9 @@ In both examples, the `/analyze` request is extended with a list of `ad_hoc_reco
Additional examples can be found in the [OpenAPI spec](../api-docs/api-docs.html).
## PII detection in different languages
Further reading:
For recognizers in new languages, refer to the [languages documentation](languages.md).
## Considerations when creating recognizers
[Best practices for developing PII recognizers](developing_recognizers.md).
1. [PII detection in different languages](languages.md).
1. [Customizing the NLP model](customizing_nlp_models.md).
1. [Best practices for developing PII recognizers](developing_recognizers.md).
1. [Code samples for customizing Presidio Analyzer with new recognizers](../samples/python/customizing_presidio_analyzer.ipynb).

145
docs/analyzer/customizing_nlp_models.md Normal file
View File

@@ -0,0 +1,145 @@
# Customizing the NLP models in Presidio Analyzer
Presidio uses NLP engines for two main tasks: NER based PII identification, and feature extraction for custom rule based logic (such as leveraging context words for improved detection).
While Presidio comes with an open-source model (the `en_core_web_lg` model from spaCy), it can be customized by leveraging other NLP models, either public or proprietary.
These models can be trained or downloaded from existing NLP frameworks like [spaCy](https://spacy.io/usage/models) and [Stanza](https://github.com/stanfordnlp/stanza).
In addition, other types of NLP frameworks can be integrated into Presidio.
## Replacing the default NLP model with a different model
### Setting up a new NLP model
As mentioned before, Presidio supports both [spaCy](https://spacy.io/usage/models)
and [Stanza](https://github.com/stanfordnlp/stanza) as its internal NLP engine.
This section describes how new models from either spaCy or Stanza could be obtained,
and how to configure Presidio to use them.
#### Download / create the new model
##### Using a public spaCy/Stanza model
To replace the default model with a different public model, first download the desired spaCy/Stanza NER models.
- To download a new model with spaCy:
```sh
python -m spacy download es_core_news_md
```
In this example we download the medium size model for Spanish.
- To download a new model with Stanza:
<!--pytest-codeblocks:skip-->
```python
import stanza
stanza.download("en") # where en is the language code of the model.
```
For the available models, follow these links: [spaCy](https://spacy.io/usage/models), [stanza](https://stanfordnlp.github.io/stanza/available_models.html#available-ner-models).
!!! tip "Tip"
For Person, Location and Organization detection, it could be useful to try out the `en_core_web_trf` model which uses a more modern deep-learning architecture, but is slower than the default `en_core_web_lg` model.
##### Training your own model
!!! note "Note"
A labeled dataset containing text and labeled PII entities is required for training a new model.
To train your own model, see these links on spaCy and Stanza:
- [Train your own spaCy model](https://spacy.io/usage/training).
- [Train your own Stanza model](https://stanfordnlp.github.io/stanza/training.html).
Once models are trained, they should be installed locally in the same environment as Presidio Analyzer.
#### Configure Presidio to use the new model
Configuration can be done in two ways:
- **Via code**: Create an `NlpEngine` using the `NlpEnginerProvider` class, and pass it to the `AnalyzerEngine` as input:
```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry
from presidio_analyzer.nlp_engine import NlpEngineProvider
# Create configuration containing engine name and models
configuration = {
"nlp_engine_name": "spacy",
"models": [{"lang_code": "es", "model_name": "es_core_news_md"},
{"lang_code": "en", "model_name": "en_core_web_lg"}],
}
# Create NLP engine based on configuration
provider = NlpEngineProvider(nlp_configuration=configuration)
nlp_engine_with_spanish = provider.create_engine()
# Pass the created NLP engine and supported_languages to the AnalyzerEngine
analyzer = AnalyzerEngine(
nlp_engine=nlp_engine_with_spanish,
supported_languages=["en", "es"]
)
# Analyze in different languages
results_spanish = analyzer.analyze(text="Mi nombre es Morris", language="es")
print(results_spanish)
results_english = analyzer.analyze(text="My name is Morris", language="en")
print(results_english)
```
- **Via configuration**: Set up the models which should be used in the [default `conf` file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/conf/default.yaml).
An example Conf file:
```yaml
nlp_engine_name: spacy
models:
-
lang_code: en
model_name: en_core_web_lg
-
lang_code: es
model_name: es_core_news_md
```
The default conf file is read during the default initialization of the `AnalyzerEngine`. Alternatively, the path to a custom configuration file can be passed to the `NlpEngineProvider`:
```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry
from presidio_analyzer.nlp_engine import NlpEngineProvider
LANGUAGES_CONFIG_FILE = "./docs/analyzer/languages-config.yml"
# Create NLP engine based on configuration file
provider = NlpEngineProvider(conf_file=LANGUAGES_CONFIG_FILE)
nlp_engine_with_spanish = provider.create_engine()
# Pass created NLP engine and supported_languages to the AnalyzerEngine
analyzer = AnalyzerEngine(
nlp_engine=nlp_engine_with_spanish,
supported_languages=["en", "es"]
)
# Analyze in different languages
results_spanish = analyzer.analyze(text="Mi nombre es David", language="es")
print(results_spanish)
results_english = analyzer.analyze(text="My name is David", language="en")
print(results_english)
```
In this examples we:
a. create an `NlpEngine` holding two spaCy models (one in English: `en_core_web_lg` and one in Spanish: `es_core_news_md`).
b. define the `supported_languages` parameter accordingly.
c. pass requests in each of these languages.
!!! note "Note"
Presidio can currently use one NLP model per language.
## Leverage frameworks other than spaCy or Stanza for ML based PII detection
In addition to the built-in spaCy/Stanza capabitilies, it is possible to create new recognizers which serve as interfaces to other models.
See the [Remote recognizer documentation](adding_recognizers.md#creating-a-remote-recognizer) and [samples](../samples/python/integrating_with_external_services.ipynb) for more information.
For considerations for creating such recognizers, see the [best practices for adding ML recognizers documentation](developing_recognizers.md#machine-learning--ml--based-or-rule-based).

4
docs/analyzer/index.md
View File

@@ -116,3 +116,7 @@ For a list of the current supported entities:
## API reference
Follow the [API Spec](https://microsoft.github.io/presidio/api-docs/api-docs.html#tag/Analyzer) for the Analyzer REST API reference details and [Analyzer Python API](../api/analyzer_python.md) for Python API reference
## Samples
Samples illustrating the usage of the Presidio Analyzer can be found in the [Python samples](../samples/python/index.md).

101
docs/analyzer/languages.md
View File

@@ -20,103 +20,8 @@ lemmatization, Named Entity Recognition and other NLP tasks.
### Configuring the NLP Engine
As its internal NLP engine, Presidio supports both [spaCy](https://spacy.io/usage/models)
and [Stanza](https://github.com/stanfordnlp/stanza). To set up new models, follow these two steps:
1. Download the spaCy/Stanza NER models for your desired language.
- To download a new model with spaCy:
```sh
python -m spacy download es_core_news_md
```
In this example we download the medium size model for Spanish.
- To download a new model with Stanza:
<!--pytest-codeblocks:skip-->
```python
import stanza
stanza.download("en") # where en is the language code of the model.
```
For the available models, follow these links: [spaCy](https://spacy.io/usage/models), [stanza](https://stanfordnlp.github.io/stanza/available_models.html#available-ner-models).
2. Update the models configuration in one of two ways:
- **Via code**: Create an `NlpEngine` using the `NlpEnginerProvider` class, and pass it to the `AnalyzerEngine` as input:
```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry
from presidio_analyzer.nlp_engine import NlpEngineProvider
# Create configuration containing engine name and models
configuration = {
"nlp_engine_name": "spacy",
"models": [{"lang_code": "es", "model_name": "es_core_news_md"},
{"lang_code": "en", "model_name": "en_core_web_lg"}],
}
# Create NLP engine based on configuration
provider = NlpEngineProvider(nlp_configuration=configuration)
nlp_engine_with_spanish = provider.create_engine()
# Pass the created NLP engine and supported_languages to the AnalyzerEngine
analyzer = AnalyzerEngine(
nlp_engine=nlp_engine_with_spanish,
supported_languages=["en", "es"]
)
# Analyze in different languages
results_spanish = analyzer.analyze(text="Mi nombre es Morris", language="es")
print(results_spanish)
results_english = analyzer.analyze(text="My name is Morris", language="en")
print(results_english)
```
- **Via configuration**: Set up the models which should be used in the [default `conf` file](https://github.com/microsoft/presidio/blob/master/presidio-analyzer/conf/default.yaml).
An example Conf file:
```yaml
nlp_engine_name: spacy
models:
-
lang_code: en
model_name: en_core_web_lg
-
lang_code: es
model_name: es_core_news_md
```
The default conf file is read during the default initialization of the `AnalyzerEngine`. Alternatively, the path to a custom configuration file can be passed to the `NlpEngineProvider`:
```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry
from presidio_analyzer.nlp_engine import NlpEngineProvider
LANGUAGES_CONFIG_FILE = "./docs/analyzer/languages-config.yml"
# Create NLP engine based on configuration file
provider = NlpEngineProvider(conf_file=LANGUAGES_CONFIG_FILE)
nlp_engine_with_spanish = provider.create_engine()
# Pass created NLP engine and supported_languages to the AnalyzerEngine
analyzer = AnalyzerEngine(
nlp_engine=nlp_engine_with_spanish,
supported_languages=["en", "es"]
)
# Analyze in different languages
results_spanish = analyzer.analyze(text="Mi nombre es David", language="es")
print(results_spanish)
results_english = analyzer.analyze(text="My name is David", language="en")
print(results_english)
```
In this examples we create an `NlpEngine` holding two spaCy models (one in English: `en_core_web_lg` and one in Spanish: `es_core_news_md`), define the `supported_languages` parameter accordingly, and can send requests in each of these languages.
Configuring the NLP engine for a new language is done by downloading or using a model trained on a different language.
See the [NLP model customization documentation](customizing_nlp_models.md) for details on how to configure models for new languages.
### Set up language specific recognizers
@@ -162,5 +67,5 @@ analyzer.analyze(text="My name is David", language="en")
When packaging the code into a Docker container, NLP models are automatically installed.
To define which models should be installed,
update the [conf/default.yaml](https://github.com/microsoft/presidio/blob/master/presidio-analyzer/conf/default.yaml) file. This file is read during
update the [conf/default.yaml](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/conf/default.yaml) file. This file is read during
the `docker build` phase and the models defined in it are installed automatically.

2
docs/api-docs/api-docs.yml
View File

@@ -11,7 +11,7 @@ info:
url: "https://upload.wikimedia.org/wikipedia/commons/4/44/Microsoft_logo.svg"
license:
name: MIT
url: 'https://github.com/microsoft/presidio/blob/master/LICENSE'
url: 'https://github.com/microsoft/presidio/blob/main/LICENSE'
externalDocs:
description: Presidio documentation.
url: 'https://microsoft.github.io/presidio/'

14
docs/image-redactor/index.md
View File

@@ -15,11 +15,11 @@ Pre-requisites:
- Install [Tesseract OCR](https://github.com/tesseract-ocr/tesseract#installing-tesseract) by following the
instructions on how to install it for your operating system.
!!! note "Note"
For now, image redactor only supports version 4.0.0
!!! attention "Attention"
For now, image redactor only supports tesseract version 4.0.0
=== "Using pip"
!!! note "Note"
Consider installing the Presidio python packages on a virtual environment like venv or conda.
@@ -32,7 +32,7 @@ Pre-requisites:
```
=== "Using Docker"
!!! note "Note"
This requires Docker to be installed. [Download Docker](https://docs.docker.com/get-docker/).
@@ -45,7 +45,7 @@ Pre-requisites:
```
=== "From source"
First, clone the Presidio repo. [See here for instructions](../installation.md#install-from-source).
Then, build the presidio-image-redactor container:
@@ -58,7 +58,7 @@ Pre-requisites:
## Getting started
=== "Python"
Once the Presidio-image-redactor package is installed, run this simple script:
```python
@@ -82,7 +82,7 @@ Pre-requisites:
```
=== "As an HTTP server"
You can run presidio image redactor as an http server using either python runtime or using a docker container.
#### Using docker container

1
docs/requirements-docs.txt
View File

@@ -7,3 +7,4 @@ mkdocstrings
presidio_analyzer
presidio_anonymizer
presidio_image_redactor
jupyter_contrib_nbextensions

12
docs/samples/index.md
View File

@@ -6,13 +6,15 @@
| Usage | Python Notebook | [Customizing Presidio Analyzer](python/customizing_presidio_analyzer.ipynb) |
| Usage | Python | [Remote Recognizer](python/example_remote_recognizer.py) |
| Usage | Python | [Text Analytics as a Remote Recognizer](python/text_analytics/index.md) |
| Usage | Python | [Integrating with external services](python/integrating_with_external_services.ipynb) |
| Usage | Python | [Integrating with external services](python/integrating_with_external_services.ipynb) |
| Usage | Python | [Passing a lambda as a Presidio anonymizer using Faker](python/example_custom_lambda_anonymizer.py) |
| Usage | Python | [Analyzing structured / semi-structured data in batch](python/batch_processing.ipynb) |
| Usage | Python | [Getting the identified entity value using a custom Operator](python/getting_entity_values.ipynb) |
| Usage | REST API (postman) | [Presidio as a REST endpoint](docker/index.md) |
| Deployment | App Service | [Presidio with App Service](deployments/app-service/index.md) |
| Deployment | Kubernetes | [Presidio with Kubernetes](deployments/k8s/index.md) |
| Deployment | Spark/Azure Databricks | [Presidio with Spark](deployments/spark/index.md) |
| Deployment | Azure Data Factory with App Service | [ETL for small dataset](deployments/data-factory/presidio-data-factory.md#option-1-presidio-as-an-http-rest-endpoint) |
| Deployment | Azure Data Factory with Databricks | [ETL for large datasets](deployments/data-factory/presidio-data-factory.md#option-2-presidio-on-azure-databricks) |
| Usage | Azure Data Factory | [Add Presidio as an HTTP service to your Azure Data Factory](deployments/data-factory/presidio-data-factory-template-gallery-http.md) |
| Usage | Azure Data Factory | [Add Presidio on Databricks to your Azure Data Factory](deployments/data-factory/presidio-data-factory-template-gallery-databricks.md) |
| Deployment | Azure Data Factory with App Service | [ETL for small dataset](deployments/data-factory/presidio-data-factory.md#option-1-presidio-as-an-http-rest-endpoint) |
| Deployment | Azure Data Factory with Databricks | [ETL for large datasets](deployments/data-factory/presidio-data-factory.md#option-2-presidio-on-azure-databricks) |
| Usage | Azure Data Factory | [Add Presidio as an HTTP service to your Azure Data Factory](deployments/data-factory/presidio-data-factory-template-gallery-http.md) |
| Usage | Azure Data Factory | [Add Presidio on Databricks to your Azure Data Factory](deployments/data-factory/presidio-data-factory-template-gallery-databricks.md) |

489
docs/samples/python/batch_processing.ipynb Normal file
View File

@@ -0,0 +1,489 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "gothic-trademark",
"metadata": {},
"source": [
"# Run Presidio on structured / semi-structured data\n",
"\n",
"This sample shows how Presidio could be potentially extended to handle the anonymization of a table or data frame.\n",
"It introduces methods for the analysis and anonymization of both lists and dicts. \n",
"\n",
"In this example we create two classes which implement the base Presidio classes:\n",
"1. `BatchAnalyzerEngine(AnalyzerEngine)`: for the presidio-anlyzer side\n",
"2. `BatchAnonymizerEngine(AnonymizerEngine`): for the presidio-anonymizer side\n",
"\n",
"In addition, we create a `dataclass` (`DictAnalyzerResult`) to serve as the data transfer object between the two.\n",
"\n",
"Note: this sample input here is a Pandas DataFrame, but it can be used in other scenarios such as querying SQL data or using Spark DataFrames."
]
},
{
"cell_type": "markdown",
"id": "roman-allergy",
"metadata": {},
"source": [
"### Set up imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "extensive-greensboro",
"metadata": {},
"outputs": [],
"source": [
"from typing import List, Optional, Dict, Union, Iterator, Iterable\n",
"import collections\n",
"from dataclasses import dataclass\n",
"import pprint\n",
"\n",
"import pandas as pd\n",
"\n",
"from presidio_analyzer import AnalyzerEngine, RecognizerResult\n",
"from presidio_anonymizer import AnonymizerEngine\n",
"from presidio_anonymizer.entities.engine.result import EngineResult\n"
]
},
{
"cell_type": "markdown",
"id": "specialized-durham",
"metadata": {},
"source": [
"### Set up classes for batch processing"
]
},
{
"cell_type": "markdown",
"id": "metropolitan-atlantic",
"metadata": {},
"source": [
"1. Presidio Analyzer: Batch mode"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "medium-ridge",
"metadata": {},
"outputs": [],
"source": [
"@dataclass\n",
"class DictAnalyzerResult:\n",
" \"\"\"Hold the analyzer results per value or list of values.\"\"\"\n",
" key: str\n",
" value: Union[str, List[str]]\n",
" recognizer_results: Union[List[RecognizerResult], List[List[RecognizerResult]]]\n",
"\n",
"\n",
"class BatchAnalyzerEngine(AnalyzerEngine):\n",
" \"\"\"\n",
" Class inheriting from AnalyzerEngine and adds the funtionality to analyze lists or dictionaries.\n",
" \"\"\"\n",
" \n",
" def analyze_list(self, list_of_texts: Iterable[str], **kwargs) -> List[List[RecognizerResult]]:\n",
" \"\"\"\n",
" Analyze an iterable of strings\n",
" \n",
" :param list_of_texts: An iterable containing strings to be analyzed.\n",
" :param kwargs: Additional parameters for the `AnalyzerEngine.analyze` method.\n",
" \"\"\"\n",
" \n",
" list_results = []\n",
" for text in list_of_texts:\n",
" results = self.analyze(text=text, **kwargs) if isinstance(text, str) else []\n",
" list_results.append(results)\n",
" return list_results\n",
"\n",
" def analyze_dict(\n",
" self, input_dict: Dict[str, Union[object, Iterable[object]]], **kwargs) -> Iterator[DictAnalyzerResult]:\n",
" \"\"\"\n",
" Analyze a dictionary of keys (strings) and values (either object or Iterable[object]). \n",
" Non-string values are returned as is.\n",
" \n",
" :param input_dict: The input dictionary for analysis\n",
" :param kwargs: Additional keyword arguments for the `AnalyzerEngine.analyze` method\n",
" \"\"\"\n",
" \n",
" for key, value in input_dict.items():\n",
" if not value:\n",
" results = []\n",
" else:\n",
" if isinstance(value, str):\n",
" results: List[RecognizerResult] = self.analyze(text=value, **kwargs)\n",
" elif isinstance(value, collections.Iterable):\n",
" results: List[List[RecognizerResult]] = self.analyze_list(\n",
" list_of_texts=value, \n",
" **kwargs)\n",
" else:\n",
" results = []\n",
" yield DictAnalyzerResult(key=key, value=value, recognizer_results=results)\n"
]
},
{
"cell_type": "markdown",
"id": "obvious-fifty",
"metadata": {},
"source": [
"2. Presidio Anonymizer"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "outstanding-celebration",
"metadata": {},
"outputs": [],
"source": [
"class BatchAnonymizerEngine(AnonymizerEngine):\n",
" \"\"\"\n",
" Class inheriting from the AnonymizerEngine and adding additional functionality \n",
" for anonymizing lists or dictionaries.\n",
" \"\"\"\n",
" \n",
" def anonymize_list(\n",
" self, \n",
" texts:List[str], \n",
" recognizer_results_list: List[List[RecognizerResult]], \n",
" **kwargs\n",
" ) -> List[EngineResult]:\n",
" \"\"\"\n",
" Anonymize a list of strings.\n",
" \n",
" :param texts: List containing the texts to be anonymized (original texts)\n",
" :param recognizer_results_list: A list of lists of RecognizerResult, \n",
" the output of the AnalyzerEngine on each text in the list.\n",
" :param kwargs: Additional kwargs for the `AnonymizerEngine.anonymize` method\n",
" \"\"\"\n",
" return_list = []\n",
" for text, recognizer_results in zip(texts, recognizer_results_list):\n",
" if isinstance(text,str):\n",
" res = self.anonymize(text=text,analyzer_results=recognizer_results,**kwargs)\n",
" return_list.append(res.text)\n",
" else:\n",
" return_list.append(text)\n",
"\n",
" return return_list\n",
"\n",
"\n",
" def anonymize_dict(self, analyzer_results: Iterator[DictAnalyzerResult],**kwargs) -> Dict[str, str]:\n",
"\n",
" \"\"\"\n",
" Anonymize values in a dictionary.\n",
" \n",
" :param analyzer_results: Iterator of `DictAnalyzerResult` \n",
" containing the output of the AnalyzerEngine.analyze_dict on the input text.\n",
" :param kwargs: Additional kwargs for the `AnonymizerEngine.anonymize` method\n",
" \"\"\"\n",
" \n",
" return_dict = {}\n",
" for result in analyzer_results:\n",
" if isinstance(result.value, str):\n",
" resp = self.anonymize(text=result.value, analyzer_results=result.recognizer_results, **kwargs)\n",
" return_dict[result.key] = resp.text\n",
" elif isinstance(result.value, collections.Iterable):\n",
" anonymize_respones = self.anonymize_list(texts=result.value,\n",
" recognizer_results_list=result.recognizer_results, \n",
" **kwargs)\n",
" return_dict[result.key] = anonymize_respones \n",
" else:\n",
" return_dict[result.key] = result.value\n",
"\n",
" return return_dict"
]
},
{
"cell_type": "markdown",
"id": "fiscal-affair",
"metadata": {},
"source": [
"## Example using sample data"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "bright-maple",
"metadata": {},
"outputs": [],
"source": [
"columns = [\"name phrase\",\"phone number phrase\", \"integer\", \"boolean\" ]\n",
"sample_data = [\n",
" ('Morris likes this','Please call 212-555-1234 after 2pm', 1, True),\n",
" ('You should talk to Mike','his number is 978-428-7111', 2, False),\n",
" ('Mary had a little startup','Phone number: 202-342-1234', 3, False)\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "russian-proceeding",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name phrase</th>\n",
" <th>phone number phrase</th>\n",
" <th>integer</th>\n",
" <th>boolean</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Morris likes this</td>\n",
" <td>Please call 212-555-1234 after 2pm</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>You should talk to Mike</td>\n",
" <td>his number is 978-428-7111</td>\n",
" <td>2</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Mary had a little startup</td>\n",
" <td>Phone number: 202-342-1234</td>\n",
" <td>3</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name phrase phone number phrase integer \\\n",
"0 Morris likes this Please call 212-555-1234 after 2pm 1 \n",
"1 You should talk to Mike his number is 978-428-7111 2 \n",
"2 Mary had a little startup Phone number: 202-342-1234 3 \n",
"\n",
" boolean \n",
"0 True \n",
"1 False \n",
"2 False "
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Create Pandas DataFrame\n",
"df = pd.DataFrame(sample_data,columns=columns)\n",
"\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "brazilian-punch",
"metadata": {},
"outputs": [],
"source": [
"# DataFrame to dict\n",
"df_dict = df.to_dict(orient=\"list\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "fixed-commerce",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'boolean': [True, False, False],\n",
" 'integer': [1, 2, 3],\n",
" 'name phrase': ['Morris likes this',\n",
" 'You should talk to Mike',\n",
" 'Mary had a little startup'],\n",
" 'phone number phrase': ['Please call 212-555-1234 after 2pm',\n",
" 'his number is 978-428-7111',\n",
" 'Phone number: 202-342-1234']}\n"
]
}
],
"source": [
"pprint.pprint(df_dict)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "verified-spirituality",
"metadata": {},
"outputs": [],
"source": [
"batch_analyzer = BatchAnalyzerEngine()\n",
"batch_anonymizer = BatchAnonymizerEngine()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "narrative-freeze",
"metadata": {},
"outputs": [],
"source": [
"analyzer_results = batch_analyzer.analyze_dict(df_dict, language=\"en\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "rural-month",
"metadata": {},
"outputs": [],
"source": [
"anonymizer_results = batch_anonymizer.anonymize_dict(analyzer_results)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "acute-mauritius",
"metadata": {},
"outputs": [],
"source": [
"scrubbed_df = pd.DataFrame(anonymizer_results)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "irish-phoenix",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name phrase</th>\n",
" <th>phone number phrase</th>\n",
" <th>integer</th>\n",
" <th>boolean</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>&lt;PERSON&gt; likes this</td>\n",
" <td>Please call &lt;PHONE_NUMBER&gt; after &lt;DATE_TIME&gt;</td>\n",
" <td>1</td>\n",
" <td>True</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>You should talk to &lt;PERSON&gt;</td>\n",
" <td>his number is &lt;PHONE_NUMBER&gt;</td>\n",
" <td>2</td>\n",
" <td>False</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>&lt;PERSON&gt; had a little startup</td>\n",
" <td>Phone number: &lt;PHONE_NUMBER&gt;</td>\n",
" <td>3</td>\n",
" <td>False</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name phrase \\\n",
"0 <PERSON> likes this \n",
"1 You should talk to <PERSON> \n",
"2 <PERSON> had a little startup \n",
"\n",
" phone number phrase integer boolean \n",
"0 Please call <PHONE_NUMBER> after <DATE_TIME> 1 True \n",
"1 his number is <PHONE_NUMBER> 2 False \n",
"2 Phone number: <PHONE_NUMBER> 3 False "
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"scrubbed_df"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "conceptual-responsibility",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "PyCharm (presidio-research)",
"language": "python",
"name": "pycharm-c8930cf3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

171
docs/samples/python/getting_entity_values.ipynb Normal file
View File

@@ -0,0 +1,171 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "adjusted-jurisdiction",
"metadata": {},
"source": [
"# Getting a list of all identified texts\n",
"\n",
"This sample illustrates how to get a list of all the identified PII entities using Presidio Analyzer for detection and a custom Presidio Anonymizer operator."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "noted-lounge",
"metadata": {},
"outputs": [],
"source": [
"from presidio_analyzer import AnalyzerEngine\n",
"from presidio_anonymizer import AnonymizerEngine\n",
"from presidio_anonymizer.entities.engine.operator_config import OperatorConfig"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "clinical-surface",
"metadata": {},
"outputs": [],
"source": [
"analyzer = AnalyzerEngine()\n",
"anonymizer = AnonymizerEngine()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "referenced-argument",
"metadata": {},
"outputs": [],
"source": [
"text_to_analyze = \"Hi my name is Charles Darwin and my email is cdarwin@hmsbeagle.org\"\n",
"analyzer_results = analyzer.analyze(text_to_analyze, language=\"en\")\n"
]
},
{
"cell_type": "markdown",
"id": "innovative-audio",
"metadata": {},
"source": [
"A naive approach for getting the text values:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "congressional-wiring",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('cdarwin@hmsbeagle.org', 45, 66),\n",
" ('hmsbeagle.org', 53, 66),\n",
" ('Charles Darwin', 14, 28)]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[(text_to_analyze[res.start:res.end], res.start, res.end) for res in analyzer_results]"
]
},
{
"cell_type": "markdown",
"id": "informal-evanescence",
"metadata": {},
"source": [
"Another option is to set up a custom operator* which runs an identity function (`lambda x: x`). This operator doesn't really anonymize, but replaces the identified value with itself. This is useful as the Anonymizer handles the overlaps automatically. \n",
"\n",
"> In this example, the URL (hmsbeagle.org) is contained in the email address, so it's ommitted from the final result.\n",
"\n",
"\\* an `Operator` is usually either an `Anonymizer` or `Deanonymizer` on the presidio-anonymizer library/"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "growing-motivation",
"metadata": {},
"outputs": [],
"source": [
"anonymized_results = anonymizer.anonymize(\n",
" text=text_to_analyze,\n",
" analyzer_results=analyzer_results, \n",
" operators={\"DEFAULT\": OperatorConfig(\"custom\", {\"lambda\": lambda x: x})} \n",
" )"
]
},
{
"cell_type": "markdown",
"id": "hydraulic-association",
"metadata": {},
"source": [
"The operator defined here is `DEFAULT`, meaning it will be used for all entities. The `OperatorConfig` is a custom one and the labmda is the identity function."
]
},
{
"cell_type": "markdown",
"id": "according-rates",
"metadata": {},
"source": [
"Output text, start and end locations for each detected entity"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "serial-arcade",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('cdarwin@hmsbeagle.org', 45, 66), ('Charles Darwin', 14, 28)]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[(item.text, item.start, item.end) for item in anonymized_results.items]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "senior-prayer",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "PyCharm (presidio-research)",
"language": "python",
"name": "pycharm-c8930cf3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

2
docs/samples/python/index.md
View File

@@ -11,3 +11,5 @@ Presidio service can be used as python packages inside python scripts
3. [Remote Recognizer](https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_remote_recognizer.py)
4. [Azure Text Analytics Integration](text_analytics/index.md)
5. [Custom Anonymizer with lambda expression](example_custom_lambda_anonymizer.py)
6. [Running Presidio on structured / semi-structured data in batch](batch_processing.ipynb)
7. [Getting the detected text value using a custom operator](getting_entity_values.ipynb)

4
mkdocs.yml
View File

@@ -18,6 +18,7 @@ nav:
- Tutorial : analyzer/adding_recognizers.md
- Best practices in developing recognizers : analyzer/developing_recognizers.md
- Multi-language support: analyzer/languages.md
- Customizing the NLP model: analyzer/customizing_nlp_models.md
- Tracing the decision process: analyzer/decision_process.md
- Presidio Anonymizer:
- Anonymizer Home: anonymizer/index.md
@@ -62,7 +63,8 @@ theme:
favicon: assets/ms_icon.png
features:
- navigation.instant
- navigation.tabs
# - navigation.tabs
# - navigation.tabs.sticky
plugins:
- search
- mkdocstrings