upstream-archive/presidio
1
0
Fork 0
mirror of https://github.com/data-privacy-stack/presidio.git synced 2026-07-29 06:10:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
ee84d70b0d30666fce5ca66809cfd7f3a83a79bb
presidio/docs/samples/python/example_structured.ipynb
Jakob Serlier 966d17a5a5 Feature/presidio-structured (#1192) 
* presidio-structured

changelog

Static analysis

docstrings, types

preliminary tests engine

static analysis

isort

Minor refactorings

Update README.md

Fix late binding issues and example

removal of old samples

Refactoring, adding example

pre-clean-break-commit

broken commit, fixing TabularConfigBuilder

Rename TabularConfig

pre-breaking replace commit

removal of some old experimental files

rename tabular to structured

restructuring presidio tabular - pre del commit

Add project TODOs

testing dump presidio tabular

* Add unit tests

* rename engine, add buildfile

* Update setup.py

* lint-build-test

* Update lint-build-test.yml

* Add packages to setup.py

* Update presidio-structured to alpha version

* Update Presidio structured README.md

* Add logging configuration to presidio-structured
module

* Refactor AnalysisBuilder constructor to accept an
optional AnalyzerEngine parameter

* Fix entity mapping in JsonAnalysisBuilder

* Drop type in docstring in analysis builder classes

* Refactor TabularAnalysisBuilder to use
BatchAnalyzerEngine for all columns

* Update data_reader.py with type hints for file
paths

* Update data_reader.py to include additional
keyword arguments in read() method

* Update Transformer to Processor term in
StructuredEngine

* Add PandasDataProcessor as default to StructuredEngine
init

* Move structured sample files to the docs

* Add Presidio Structured  Notebook to samples index

* Remove unnecessary imports in structured sample

* Update to processors in structured __init__ files

* Add explanation for structured table sample

* Delete unnecessary __init__s in structured test

* Fix bug in JsonAnalysisBuilder entity mapping

* pr comments, nits, minor tests

* README

* Add TabularAnalysisBuilder

* Some basic logging

* linting

* Fix typo in logger variable name

* Refactor analysis builder to include score
threshold

* Linting, continued

* Update Pipfile

* Refactor JsonAnalysisBuilder to support language
parameter

* Fix not camel case in TabularAnalysisBuilder

* Add score_threshold parameter to AnalysisBuilder

* Refactor JSON analysis builder to gain consistency

* Remove low score results in JsonAnalysisBuilder

* Add tests to json analysis  with score threshold

* Fix bug in JSON analysis to update map with
nested_mappings

* Fix bug in JSON analysis to take only entity types

* Fix typos in test anl json names and assert values

* Update build-structured.yml

* Create __init__.py

* Type hint fix python <3.10, loggger typo

* Update setup.py

* PR comments variety

* further pr comments

* readme, refactor score, refactor tabular analysis

* Update test_analysis_builder.py

* lint

---------

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Co-authored-by: enrique.botia <enrique.botia@netzima.com>
2024-01-14 11:22:44 +02:00

566 lines
16 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from presidio_structured import StructuredEngine, JsonAnalysisBuilder, PandasAnalysisBuilder, StructuredAnalysis, CsvReader, JsonReader, JsonDataProcessor, PandasDataProcessor"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This sample showcases presidio-structured on structured and semi-structured data containing sensitive data like names, emails, and addresses. It differs from the sample for the batch analyzer/anonymizer engines example, which includes narrative phrases that might contain sensitive data. The presence of personal data embedded in these phrases requires to analyze and to anonymize the text inside the cells, which is not the case for our structured sample, where the sensitive data is already separated into columns."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loading in data"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>name</th>\n",
" <th>email</th>\n",
" <th>street</th>\n",
" <th>city</th>\n",
" <th>state</th>\n",
" <th>postal_code</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>John Doe</td>\n",
" <td>john.doe@example.com</td>\n",
" <td>123 Main St</td>\n",
" <td>Anytown</td>\n",
" <td>CA</td>\n",
" <td>12345</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>Jane Smith</td>\n",
" <td>jane.smith@example.com</td>\n",
" <td>456 Elm St</td>\n",
" <td>Somewhere</td>\n",
" <td>TX</td>\n",
" <td>67890</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>Alice Johnson</td>\n",
" <td>alice.johnson@example.com</td>\n",
" <td>789 Pine St</td>\n",
" <td>Elsewhere</td>\n",
" <td>NY</td>\n",
" <td>11223</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id name email street city state \\\n",
"0 1 John Doe john.doe@example.com 123 Main St Anytown CA \n",
"1 2 Jane Smith jane.smith@example.com 456 Elm St Somewhere TX \n",
"2 3 Alice Johnson alice.johnson@example.com 789 Pine St Elsewhere NY \n",
"\n",
" postal_code \n",
"0 12345 \n",
"1 67890 \n",
"2 11223 "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sample_df = CsvReader().read(\"./csv_sample_data/test_structured.csv\")\n",
"sample_df"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 1,\n",
" 'name': 'John Doe',\n",
" 'email': 'john.doe@example.com',\n",
" 'address': {'street': '123 Main St',\n",
" 'city': 'Anytown',\n",
" 'state': 'CA',\n",
" 'postal_code': '12345'}}"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sample_json = JsonReader().read(\"./sample_data/test_structured.json\")\n",
"sample_json"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'users': [{'id': 1,\n",
" 'name': 'John Doe',\n",
" 'email': 'john.doe@example.com',\n",
" 'address': {'street': '123 Main St',\n",
" 'city': 'Anytown',\n",
" 'state': 'CA',\n",
" 'postal_code': '12345'}},\n",
" {'id': 2,\n",
" 'name': 'Jane Smith',\n",
" 'email': 'jane.smith@example.com',\n",
" 'address': {'street': '456 Elm St',\n",
" 'city': 'Somewhere',\n",
" 'state': 'TX',\n",
" 'postal_code': '67890'}},\n",
" {'id': 3,\n",
" 'name': 'Alice Johnson',\n",
" 'email': 'alice.johnson@example.com',\n",
" 'address': {'street': '789 Pine St',\n",
" 'city': 'Elsewhere',\n",
" 'state': 'NY',\n",
" 'postal_code': '11223'}}]}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# contains nested objects in lists\n",
"sample_complex_json = JsonReader().read(\"./sample_data/test_structured_complex.json\")\n",
"sample_complex_json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tabular (csv) data: defining & generating tabular analysis, anonymization."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"StructuredAnalysis(entity_mapping={'name': 'PERSON', 'email': 'EMAIL_ADDRESS', 'city': 'LOCATION', 'state': 'LOCATION'})"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Automatically detect the entity for the columns\n",
"tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df)\n",
"tabular_analysis"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>name</th>\n",
" <th>email</th>\n",
" <th>street</th>\n",
" <th>city</th>\n",
" <th>state</th>\n",
" <th>postal_code</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>123 Main St</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>12345</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>456 Elm St</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>67890</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>789 Pine St</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>11223</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id name email street city state postal_code\n",
"0 1 <None> <None> 123 Main St <None> <None> 12345\n",
"1 2 <None> <None> 456 Elm St <None> <None> 67890\n",
"2 3 <None> <None> 789 Pine St <None> <None> 11223"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# anonymized data defaults to be replaced with None, unless operators is specified\n",
"\n",
"pandas_engine = StructuredEngine(data_processor=PandasDataProcessor())\n",
"df_to_be_anonymized = sample_df.copy() # in-place anonymization\n",
"anonymized_df = pandas_engine.anonymize(df_to_be_anonymized, tabular_analysis, operators=None) # explicit None for clarity\n",
"anonymized_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We can also define operators using OperatorConfig similar as to the AnonymizerEngine:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>id</th>\n",
" <th>name</th>\n",
" <th>email</th>\n",
" <th>street</th>\n",
" <th>city</th>\n",
" <th>state</th>\n",
" <th>postal_code</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>person...</td>\n",
" <td>jamestaylor@example.net</td>\n",
" <td>123 Main St</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>12345</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>person...</td>\n",
" <td>brian49@example.com</td>\n",
" <td>456 Elm St</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>67890</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>person...</td>\n",
" <td>clarkcody@example.org</td>\n",
" <td>789 Pine St</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>&lt;None&gt;</td>\n",
" <td>11223</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" id name email street city state \\\n",
"0 1 person... jamestaylor@example.net 123 Main St <None> <None> \n",
"1 2 person... brian49@example.com 456 Elm St <None> <None> \n",
"2 3 person... clarkcody@example.org 789 Pine St <None> <None> \n",
"\n",
" postal_code \n",
"0 12345 \n",
"1 67890 \n",
"2 11223 "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from presidio_anonymizer.entities.engine import OperatorConfig\n",
"from faker import Faker\n",
"fake = Faker()\n",
"\n",
"operators = {\n",
" \"PERSON\": OperatorConfig(\"replace\", {\"new_value\": \"person...\"}),\n",
" \"EMAIL_ADDRESS\": OperatorConfig(\"custom\", {\"lambda\": lambda x: fake.safe_email()})\n",
" # etc...\n",
" }\n",
"anonymized_df = pandas_engine.anonymize(sample_df, tabular_analysis, operators=operators)\n",
"anonymized_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Semi-structured (JSON) data: simple and complex analysis, anonymization"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"StructuredAnalysis(entity_mapping={'name': 'PERSON', 'email': 'EMAIL_ADDRESS', 'address.city': 'LOCATION', 'address.state': 'LOCATION'})"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"json_analysis = JsonAnalysisBuilder().generate_analysis(sample_json)\n",
"json_analysis"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Analyzer.analyze_iterator only works on primitive types (int, float, bool, str). Lists of objects are not yet supported.\n"
]
}
],
"source": [
"# Currently does not support nested objects in lists\n",
"try:\n",
" json_complex_analysis = JsonAnalysisBuilder().generate_analysis(sample_complex_json)\n",
"except ValueError as e:\n",
" print(e)\n",
"\n",
"# however, we can define it manually:\n",
"json_complex_analysis = StructuredAnalysis(entity_mapping={\n",
" \"users.name\":\"PERSON\",\n",
" \"users.address.street\":\"LOCATION\",\n",
" \"users.address.city\":\"LOCATION\",\n",
" \"users.address.state\":\"LOCATION\",\n",
" \"users.email\": \"EMAIL_ADDRESS\",\n",
"})"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'id': 1,\n",
" 'name': 'person...',\n",
" 'email': 'virginia29@example.org',\n",
" 'address': {'street': '123 Main St',\n",
" 'city': '<None>',\n",
" 'state': '<None>',\n",
" 'postal_code': '12345'}}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# anonymizing simple data\n",
"json_engine = StructuredEngine(data_processor=JsonDataProcessor())\n",
"anonymized_json = json_engine.anonymize(sample_json, json_analysis, operators=operators)\n",
"anonymized_json"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'users': [{'id': 1,\n",
" 'name': 'person...',\n",
" 'email': 'david90@example.org',\n",
" 'address': {'street': '<None>',\n",
" 'city': '<None>',\n",
" 'state': '<None>',\n",
" 'postal_code': '12345'}},\n",
" {'id': 2,\n",
" 'name': 'person...',\n",
" 'email': 'david90@example.org',\n",
" 'address': {'street': '<None>',\n",
" 'city': '<None>',\n",
" 'state': '<None>',\n",
" 'postal_code': '67890'}},\n",
" {'id': 3,\n",
" 'name': 'person...',\n",
" 'email': 'david90@example.org',\n",
" 'address': {'street': '<None>',\n",
" 'city': '<None>',\n",
" 'state': '<None>',\n",
" 'postal_code': '11223'}}]}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"anonymized_complex_json = json_engine.anonymize(sample_complex_json, json_complex_analysis, operators=operators)\n",
"anonymized_complex_json"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink