{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from presidio_structured import StructuredEngine, JsonAnalysisBuilder, PandasAnalysisBuilder, StructuredAnalysis, CsvReader, JsonReader, JsonDataProcessor, PandasDataProcessor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This sample showcases presidio-structured on structured and semi-structured data containing sensitive data like names, emails, and addresses. It differs from the sample for the batch analyzer/anonymizer engines example, which includes narrative phrases that might contain sensitive data. The presence of personal data embedded in these phrases requires to analyze and to anonymize the text inside the cells, which is not the case for our structured sample, where the sensitive data is already separated into columns." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading in data" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idnameemailstreetcitystatepostal_code
01John Doejohn.doe@example.com123 Main StAnytownCA12345
12Jane Smithjane.smith@example.com456 Elm StSomewhereTX67890
23Alice Johnsonalice.johnson@example.com789 Pine StElsewhereNY11223
\n", "
" ], "text/plain": [ " id name email street city state \\\n", "0 1 John Doe john.doe@example.com 123 Main St Anytown CA \n", "1 2 Jane Smith jane.smith@example.com 456 Elm St Somewhere TX \n", "2 3 Alice Johnson alice.johnson@example.com 789 Pine St Elsewhere NY \n", "\n", " postal_code \n", "0 12345 \n", "1 67890 \n", "2 11223 " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_df = CsvReader().read(\"./csv_sample_data/test_structured.csv\")\n", "sample_df" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'id': 1,\n", " 'name': 'John Doe',\n", " 'email': 'john.doe@example.com',\n", " 'address': {'street': '123 Main St',\n", " 'city': 'Anytown',\n", " 'state': 'CA',\n", " 'postal_code': '12345'}}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sample_json = JsonReader().read(\"./sample_data/test_structured.json\")\n", "sample_json" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'users': [{'id': 1,\n", " 'name': 'John Doe',\n", " 'email': 'john.doe@example.com',\n", " 'address': {'street': '123 Main St',\n", " 'city': 'Anytown',\n", " 'state': 'CA',\n", " 'postal_code': '12345'}},\n", " {'id': 2,\n", " 'name': 'Jane Smith',\n", " 'email': 'jane.smith@example.com',\n", " 'address': {'street': '456 Elm St',\n", " 'city': 'Somewhere',\n", " 'state': 'TX',\n", " 'postal_code': '67890'}},\n", " {'id': 3,\n", " 'name': 'Alice Johnson',\n", " 'email': 'alice.johnson@example.com',\n", " 'address': {'street': '789 Pine St',\n", " 'city': 'Elsewhere',\n", " 'state': 'NY',\n", " 'postal_code': '11223'}}]}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# contains nested objects in lists\n", "sample_complex_json = JsonReader().read(\"./sample_data/test_structured_complex.json\")\n", "sample_complex_json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tabular (csv) data: defining & generating tabular analysis, anonymization." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "StructuredAnalysis(entity_mapping={'name': 'PERSON', 'email': 'EMAIL_ADDRESS', 'city': 'LOCATION', 'state': 'LOCATION'})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Automatically detect the entity for the columns\n", "tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df)\n", "tabular_analysis" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idnameemailstreetcitystatepostal_code
01<None><None>123 Main St<None><None>12345
12<None><None>456 Elm St<None><None>67890
23<None><None>789 Pine St<None><None>11223
\n", "
" ], "text/plain": [ " id name email street city state postal_code\n", "0 1 123 Main St 12345\n", "1 2 456 Elm St 67890\n", "2 3 789 Pine St 11223" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# anonymized data defaults to be replaced with None, unless operators is specified\n", "\n", "pandas_engine = StructuredEngine(data_processor=PandasDataProcessor())\n", "df_to_be_anonymized = sample_df.copy() # in-place anonymization\n", "anonymized_df = pandas_engine.anonymize(df_to_be_anonymized, tabular_analysis, operators=None) # explicit None for clarity\n", "anonymized_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### We can also define operators using OperatorConfig similar as to the AnonymizerEngine:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idnameemailstreetcitystatepostal_code
01person...jamestaylor@example.net123 Main St<None><None>12345
12person...brian49@example.com456 Elm St<None><None>67890
23person...clarkcody@example.org789 Pine St<None><None>11223
\n", "
" ], "text/plain": [ " id name email street city state \\\n", "0 1 person... jamestaylor@example.net 123 Main St \n", "1 2 person... brian49@example.com 456 Elm St \n", "2 3 person... clarkcody@example.org 789 Pine St \n", "\n", " postal_code \n", "0 12345 \n", "1 67890 \n", "2 11223 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from presidio_anonymizer.entities.engine import OperatorConfig\n", "from faker import Faker\n", "fake = Faker()\n", "\n", "operators = {\n", " \"PERSON\": OperatorConfig(\"replace\", {\"new_value\": \"person...\"}),\n", " \"EMAIL_ADDRESS\": OperatorConfig(\"custom\", {\"lambda\": lambda x: fake.safe_email()})\n", " # etc...\n", " }\n", "anonymized_df = pandas_engine.anonymize(sample_df, tabular_analysis, operators=operators)\n", "anonymized_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Semi-structured (JSON) data: simple and complex analysis, anonymization" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "StructuredAnalysis(entity_mapping={'name': 'PERSON', 'email': 'EMAIL_ADDRESS', 'address.city': 'LOCATION', 'address.state': 'LOCATION'})" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "json_analysis = JsonAnalysisBuilder().generate_analysis(sample_json)\n", "json_analysis" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Analyzer.analyze_iterator only works on primitive types (int, float, bool, str). Lists of objects are not yet supported.\n" ] } ], "source": [ "# Currently does not support nested objects in lists\n", "try:\n", " json_complex_analysis = JsonAnalysisBuilder().generate_analysis(sample_complex_json)\n", "except ValueError as e:\n", " print(e)\n", "\n", "# however, we can define it manually:\n", "json_complex_analysis = StructuredAnalysis(entity_mapping={\n", " \"users.name\":\"PERSON\",\n", " \"users.address.street\":\"LOCATION\",\n", " \"users.address.city\":\"LOCATION\",\n", " \"users.address.state\":\"LOCATION\",\n", " \"users.email\": \"EMAIL_ADDRESS\",\n", "})" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'id': 1,\n", " 'name': 'person...',\n", " 'email': 'virginia29@example.org',\n", " 'address': {'street': '123 Main St',\n", " 'city': '',\n", " 'state': '',\n", " 'postal_code': '12345'}}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# anonymizing simple data\n", "json_engine = StructuredEngine(data_processor=JsonDataProcessor())\n", "anonymized_json = json_engine.anonymize(sample_json, json_analysis, operators=operators)\n", "anonymized_json" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'users': [{'id': 1,\n", " 'name': 'person...',\n", " 'email': 'david90@example.org',\n", " 'address': {'street': '',\n", " 'city': '',\n", " 'state': '',\n", " 'postal_code': '12345'}},\n", " {'id': 2,\n", " 'name': 'person...',\n", " 'email': 'david90@example.org',\n", " 'address': {'street': '',\n", " 'city': '',\n", " 'state': '',\n", " 'postal_code': '67890'}},\n", " {'id': 3,\n", " 'name': 'person...',\n", " 'email': 'david90@example.org',\n", " 'address': {'street': '',\n", " 'city': '',\n", " 'state': '',\n", " 'postal_code': '11223'}}]}" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "anonymized_complex_json = json_engine.anonymize(sample_complex_json, json_complex_analysis, operators=operators)\n", "anonymized_complex_json" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.7" } }, "nbformat": 4, "nbformat_minor": 2 }