upstream-archive/presidio
1
0
Fork 0
mirror of https://github.com/data-privacy-stack/presidio.git synced 2026-07-23 11:20:55 -05:00
Code Issues Packages Projects Releases Wiki Activity

Docs/dps migration phase1 (#2104)

Browse Source
This commit is contained in:
Omri Mendels
2026-06-28 10:27:33 +03:00
committed by GitHub
parent ab9cca6f77
commit 94f866a96b
88 changed files with 398 additions and 354 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@@ -8,8 +8,9 @@ Fixes #XX
## Checklist ## Checklist
- [ ] I have reviewed the [contribution guidelines](https://github.com/microsoft/presidio/blob/main/CONTRIBUTING.md) - [ ] I have reviewed the [contribution guidelines](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md)
- [ ] I have signed the CLA (if required) - [ ] I agree to follow this project's [Code of Conduct](https://github.com/data-privacy-stack/presidio/blob/main/CODE_OF_CONDUCT.md)
- [ ] I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
- [ ] My code includes unit tests - [ ] My code includes unit tests
- [ ] All unit tests and lint checks pass locally - [ ] All unit tests and lint checks pass locally
- [ ] My PR contains documentation updates / additions if required - [ ] My PR contains documentation updates / additions if required

9
CODE_OF_CONDUCT
View File

@@ -1,9 +0,0 @@
# Microsoft Open Source Code of Conduct
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
Resources:
- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns

83
CODE_OF_CONDUCT.md Normal file
View File

@@ -0,0 +1,83 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at presidio@dataprivacystack.org. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of actions.
**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0, available at [https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations

25
CONTRIBUTING.md
View File

@@ -4,6 +4,10 @@
The following is a set of guidelines for contributing to Presidio. The following is a set of guidelines for contributing to Presidio.
## Code of Conduct
This project has adopted the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to <presidio@dataprivacystack.org>.
## What do I need to know before I get started? ## What do I need to know before I get started?
### Project Presidio ### Project Presidio
@@ -12,6 +16,10 @@ Presidio is a community project aimed at helping everyone handle their private d
Presidio is both a framework and a system. It's a framework in a sense that you could take code parts from it, extend, customize and plug somewhere. It's also a system you could take as a whole or in parts and deploy locally, on-prem on in the cloud. Presidio is both a framework and a system. It's a framework in a sense that you could take code parts from it, extend, customize and plug somewhere. It's also a system you could take as a whole or in parts and deploy locally, on-prem on in the cloud.
When contributing to presidio, it's important to keep this in mind, as some "framework" contributions might not be suitable for a deployment, or vice-versa. When contributing to presidio, it's important to keep this in mind, as some "framework" contributions might not be suitable for a deployment, or vice-versa.
### Rights of contribution
By submitting a contribution, you represent that you have the legal right to contribute the code and that it may be distributed under the project's MIT license. Furthermore, you confirm that you have the right to submit this contribution and that it does not knowingly contain proprietary or confidential information/code.
### PR guidelines ### PR guidelines
Commit message should be clear, explaining the committed changes. Commit message should be clear, explaining the committed changes.
@@ -22,11 +30,6 @@ Under Unreleased section, use the category which is most suitable for your chang
Document the change with simple readable text and push it as part of the commit. Document the change with simple readable text and push it as part of the commit.
Next release, the change will be documented under the new version. Next release, the change will be documented under the new version.
### Build and Release process
The project currently supports [Azure Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/) using YAML pipelines which can be easily imported to any Azure Pipelines instance.
For more details follow the [Build and Release documentation](docs/build_release.md).
## Getting started with the code ## Getting started with the code
To get started, refer to the documentation for [setting up a development environment](docs/development.md). To get started, refer to the documentation for [setting up a development environment](docs/development.md).
@@ -86,7 +89,7 @@ To contribute a new predefined recognizer to Presidio Analyzer:
### Fixing Bugs and improving the code ### Fixing Bugs and improving the code
Please review the open [issues on Github](https://github.com/microsoft/presidio/issues) for known bugs and feature requests. We sometimes add 'good first issue' labels on those we believe are simpler, and 'advanced' labels on those which require more work or multiple changes across the solution. Please review the open [issues on Github](https://github.com/data-privacy-stack/presidio/issues) for known bugs and feature requests. We sometimes add 'good first issue' labels on those we believe are simpler, and 'advanced' labels on those which require more work or multiple changes across the solution.
### Adding samples ### Adding samples
@@ -96,12 +99,4 @@ When contributing a sample, make sure it is self contained (e.g. external depend
## Contacting Us ## Contacting Us
For any questions, please email <presidio@microsoft.com>. For any questions, please email <presidio@dataprivacystack.org>.
## Contribution guidelines
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit <https://cla.microsoft.com>.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact <opencode@microsoft.com> with any additional questions or comments.

59
README.MD
View File

@@ -4,18 +4,19 @@
--- ---
[![Build Status](https://github.com/microsoft/presidio/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/microsoft/presidio/actions/workflows/ci.yml) [![Build Status](https://github.com/data-privacy-stack/presidio/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/data-privacy-stack/presidio/actions/workflows/ci.yml)
[![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT) [![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT)
![Release](https://img.shields.io/github/release/Microsoft/presidio.svg) ![Release](https://img.shields.io/github/release/data-privacy-stack/presidio.svg)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/6076/badge)](https://www.bestpractices.dev/projects/6076) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/6076/badge)](https://www.bestpractices.dev/projects/6076)
[![PyPI pyversions](https://img.shields.io/pypi/pyversions/presidio-analyzer.svg)](https://pypi.python.org/pypi/presidio-analyzer/) [![PyPI pyversions](https://img.shields.io/pypi/pyversions/presidio-analyzer.svg)](https://pypi.python.org/pypi/presidio-analyzer/)
[![Discord](https://img.shields.io/badge/Discord-Join%20the%20community-5865F2?logo=discord&logoColor=white)](https://discord.gg/MewtBGFquB)
| Component | Downloads | Coverage | | Component | Downloads | Coverage |
|-----------|-----------|----------| |-----------|-----------|----------|
| Presidio Analyzer | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-analyzer.svg)](https://pypi.python.org/pypi/presidio-analyzer/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/microsoft/presidio/coverage-data-presidio-analyzer/endpoint.json)](https://github.com/microsoft/presidio/tree/coverage-data-presidio-analyzer) | | Presidio Analyzer | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-analyzer.svg)](https://pypi.python.org/pypi/presidio-analyzer/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/data-privacy-stack/presidio/coverage-data-presidio-analyzer/endpoint.json)](https://github.com/data-privacy-stack/presidio/tree/coverage-data-presidio-analyzer) |
| Presidio Anonymizer | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-anonymizer.svg)](https://pypi.python.org/pypi/presidio-anonymizer/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/microsoft/presidio/coverage-data-presidio-anonymizer/endpoint.json)](https://github.com/microsoft/presidio/tree/coverage-data-presidio-anonymizer) | | Presidio Anonymizer | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-anonymizer.svg)](https://pypi.python.org/pypi/presidio-anonymizer/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/data-privacy-stack/presidio/coverage-data-presidio-anonymizer/endpoint.json)](https://github.com/data-privacy-stack/presidio/tree/coverage-data-presidio-anonymizer) |
| Presidio Image-Redactor | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-image-redactor.svg)](https://pypi.python.org/pypi/presidio-image-redactor/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/microsoft/presidio/coverage-data-presidio-image-redactor/endpoint.json)](https://github.com/microsoft/presidio/tree/coverage-data-presidio-image-redactor) | | Presidio Image-Redactor | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-image-redactor.svg)](https://pypi.python.org/pypi/presidio-image-redactor/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/data-privacy-stack/presidio/coverage-data-presidio-image-redactor/endpoint.json)](https://github.com/data-privacy-stack/presidio/tree/coverage-data-presidio-image-redactor) |
| Presidio Structured | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-structured.svg)](https://pypi.python.org/pypi/presidio-structured/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/microsoft/presidio/coverage-data-presidio-structured/endpoint.json)](https://github.com/microsoft/presidio/tree/coverage-data-presidio-structured) | | Presidio Structured | [![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-structured.svg)](https://pypi.python.org/pypi/presidio-structured/) | [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/data-privacy-stack/presidio/coverage-data-presidio-structured/endpoint.json)](https://github.com/data-privacy-stack/presidio/tree/coverage-data-presidio-structured) |
## What is Presidio ## What is Presidio
Presidio _(Origin from Latin praesidium ‘protection, garrison’)_ helps to ensure sensitive data is properly managed and governed. It provides fast **_identification_** and **_anonymization_** modules for private entities in text such as credit card numbers, names, locations, social security numbers, bitcoin wallets, US phone numbers, financial data and more. Presidio _(Origin from Latin praesidium ‘protection, garrison’)_ helps to ensure sensitive data is properly managed and governed. It provides fast **_identification_** and **_anonymization_** modules for private entities in text such as credit card numbers, names, locations, social security numbers, bitcoin wallets, US phone numbers, financial data and more.
@@ -24,15 +25,15 @@ Presidio _(Origin from Latin praesidium ‘protection, garrison’)_ helps to en
--- ---
### :blue_book: [Full documentation](https://microsoft.github.io/presidio) ### :blue_book: [Full documentation](https://data-privacy-stack.github.io/presidio)
### :mega: [Project transition update](docs/project_transition.md) ### :mega: [Project transition update](docs/project_transition.md)
### :question: [Frequently Asked Questions](docs/faq.md) ### :question: [Frequently Asked Questions](docs/faq.md)
### :thought_balloon: [Demo](https://aka.ms/presidio-demo) ### :thought_balloon: [Demo](https://huggingface.co/spaces/presidio/presidio_demo)
### :flight_departure: [Examples](https://microsoft.github.io/presidio/samples/) ### :flight_departure: [Examples](https://data-privacy-stack.github.io/presidio/samples/)
--- ---
@@ -55,43 +56,33 @@ Presidio _(Origin from Latin praesidium ‘protection, garrison’)_ helps to en
## Installing Presidio ## Installing Presidio
1. [Using pip](https://microsoft.github.io/presidio/installation/#using-pip) 1. [Using pip](https://data-privacy-stack.github.io/presidio/installation/#using-pip)
2. [Using Docker](https://microsoft.github.io/presidio/installation/#using-docker) 2. [Using Docker](https://data-privacy-stack.github.io/presidio/installation/#using-docker)
3. [From source](https://microsoft.github.io/presidio/installation/#install-from-source) 3. [From source](https://data-privacy-stack.github.io/presidio/installation/#install-from-source)
4. [Migrating from V1 to V2](./docs/presidio_V2.md) 4. [Migrating from V1 to V2](./docs/presidio_V2.md)
## Running Presidio ## Running Presidio
1. [Getting started](https://microsoft.github.io/presidio/getting_started) 1. [Getting started](https://data-privacy-stack.github.io/presidio/getting_started)
2. [Setting up a development environment](https://microsoft.github.io/presidio/development) 2. [Setting up a development environment](https://data-privacy-stack.github.io/presidio/development)
3. [PII de-identification in text](https://microsoft.github.io/presidio/text_anonymization) 3. [PII de-identification in text](https://data-privacy-stack.github.io/presidio/text_anonymization)
4. [PII de-identification in images](https://microsoft.github.io/presidio/image-redactor) 4. [PII de-identification in images](https://data-privacy-stack.github.io/presidio/image-redactor)
5. [Usage samples and example deployments](https://microsoft.github.io/presidio/samples) 5. [Usage samples and example deployments](https://data-privacy-stack.github.io/presidio/samples)
--- ---
## Support ## Support
- Before you submit an issue, please go over the [documentation](https://microsoft.github.io/presidio/). - Before you submit an issue, please go over the [documentation](https://data-privacy-stack.github.io/presidio/).
- For general discussions, please use the [GitHub repo's discussion board](https://github.com/microsoft/presidio/discussions). - For general discussions, please use the [GitHub repo's discussion board](https://github.com/data-privacy-stack/presidio/discussions).
- If you have a usage question, found a bug or have a suggestion for improvement, please file a [GitHub issue](https://github.com/microsoft/presidio/issues). - If you have a usage question, found a bug or have a suggestion for improvement, please file a [GitHub issue](https://github.com/data-privacy-stack/presidio/issues).
- For other matters, please email [presidio@microsoft.com](mailto:presidio@microsoft.com). - For other matters, please email [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org).
## Contributing ## Contributing
For details on contributing to this repository, see the [contributing guide](CONTRIBUTING.md). For details on contributing to this repository, see the [contributing guide](CONTRIBUTING.md).
This project welcomes contributions and suggestions. Most contributions require you to agree to a This project has adopted the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md).
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit [https://cla.microsoft.com](https://cla.microsoft.com).
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
## Contributors ## Contributors
@@ -101,6 +92,6 @@ contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additio
<!-- markdownlint-restore --> <!-- markdownlint-restore -->
<!-- prettier-ignore-end --> <!-- prettier-ignore-end -->
<!-- ALL-CONTRIBUTORS-LIST:END --> <!-- ALL-CONTRIBUTORS-LIST:END -->
<a href="https://github.com/microsoft/presidio/graphs/contributors"> <a href="https://github.com/data-privacy-stack/presidio/graphs/contributors">
<img src="https://contrib.rocks/image?repo=microsoft/presidio" /> <img src="https://contrib.rocks/image?repo=data-privacy-stack/presidio" />
</a> </a>

47
SECURITY.md
View File

@@ -1,41 +1,22 @@
<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK --> # Security Policy
## Security We take the security of this project seriously. If you believe you have found a security vulnerability, please report it to us responsibly using the instructions below.
Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). ## Reporting a Vulnerability
If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. **Please do not open public GitHub issues for security vulnerabilities.**
## Reporting Security Issues Instead, please use GitHub's **Private Vulnerability Reporting** feature:
**Please do not report security vulnerabilities through public GitHub issues.** 1. Navigate to the main page of the repository.
2. Under the repository name, click **Security**.
3. In the left sidebar, click **Reporting**.
4. Click **Report a vulnerability** to open the advisory form.
Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). Alternatively, you can use this direct link: [Report a Security Vulnerability](https://github.com/data-privacy-stack/presidio/security/advisories/new)
If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). ## Our Process
You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). * **Acknowledgment:** We will acknowledge receipt of your report within 48 hours.
* **Evaluation:** We will investigate the issue and keep you updated via the private GitHub Advisory thread.
Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: * **Disclosure:** Once a fix is ready, we will coordinate the release and publish a Security Advisory to credit your work (if desired).
- Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
- Full paths of source file(s) related to the manifestation of the issue
- The location of the affected source code (tag/branch/commit or direct URL)
- Any special configuration required to reproduce the issue
- Step-by-step instructions to reproduce the issue
- Proof-of-concept or exploit code (if possible)
- Impact of the issue, including how an attacker might exploit the issue
This information will help us triage your report more quickly.
If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.
## Preferred Languages
We prefer all communications to be in English.
## Policy
Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).
<!-- END MICROSOFT SECURITY.MD BLOCK -->

2
SUPPORT.md
View File

@@ -2,4 +2,4 @@
This open source project does not have any SLA or official support. This open source project does not have any SLA or official support.
The best way to get support is to open an issue or start a discussion. The best way to get support is to open an issue or start a discussion.
Before doing so, consider searching on the [Github repo](https://github.com/search?q=repo%3Amicrosoft%2Fpresidio%20&type=code) or the [docs website](https://microsoft.github.io/presidio/). Before doing so, consider searching on the [Github repo](https://github.com/search?q=repo%3Adata-privacy-stack%2Fpresidio%20&type=code) or the [docs website](https://data-privacy-stack.github.io/presidio/).

12
docs/analyzer/adding_recognizers.md
View File

@@ -102,8 +102,8 @@ engine.analyze(...)
To create a new recognizer via code: To create a new recognizer via code:
1. Create a new Python class which implements [LocalRecognizer](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/local_recognizer.py). 1. Create a new Python class which implements [LocalRecognizer](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/local_recognizer.py).
(`LocalRecognizer` implements the base [EntityRecognizer](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) class) (`LocalRecognizer` implements the base [EntityRecognizer](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) class)
This class has the following functions: This class has the following functions:
@@ -124,9 +124,9 @@ To create a new recognizer via code:
Notes: Notes:
1. Each recognizer has access to different NLP assets such as tokens, [lemmas](https://en.wikipedia.org/wiki/Lemma_(morphology)), and more. 1. Each recognizer has access to different NLP assets such as tokens, [lemmas](https://en.wikipedia.org/wiki/Lemma_(morphology)), and more.
These are given through the `nlp_artifacts` parameter. These are given through the `nlp_artifacts` parameter.
Refer to the [source code](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) for more information. Refer to the [source code](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) for more information.
2. The `analyze` method should return a list of [RecognizerResult](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/recognizer_result.py). 2. The `analyze` method should return a list of [RecognizerResult](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/recognizer_result.py).
2. Add it to the recognizer registry using `registry.add_recognizer(my_recognizer)`. 2. Add it to the recognizer registry using `registry.add_recognizer(my_recognizer)`.
@@ -150,7 +150,7 @@ To add a recognizer to the list of pre-defined recognizers:
1. Clone the repo. 1. Clone the repo.
2. Create a file containing the new recognizer Python class. 2. Create a file containing the new recognizer Python class.
3. Add the recognizer to the `recognizers` in the [`default_recognizers`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) config. Details of recognizer parameters are given [Here](./recognizer_registry_provider.md#the-recognizer-parameters). 3. Add the recognizer to the `recognizers` in the [`default_recognizers`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) config. Details of recognizer parameters are given [Here](./recognizer_registry_provider.md#the-recognizer-parameters).
4. If the recognizer is country-specific (file lives under `predefined_recognizers/country_specific/<country>/`), declare its country by setting the class-level `COUNTRY_CODE = "<iso-3166-1-alpha-2>"` attribute. This makes the recognizer participate in the country filter — see [Filtering recognizers by country](./filtering_by_country.md). Generic / locale-agnostic recognizers leave `COUNTRY_CODE` unset (defaults to `None`). 4. If the recognizer is country-specific (file lives under `predefined_recognizers/country_specific/<country>/`), declare its country by setting the class-level `COUNTRY_CODE = "<iso-3166-1-alpha-2>"` attribute. This makes the recognizer participate in the country filter — see [Filtering recognizers by country](./filtering_by_country.md). Generic / locale-agnostic recognizers leave `COUNTRY_CODE` unset (defaults to `None`).
5. Optional: Update documentation (e.g., the [supported entities list](../supported_entities.md)). 5. Optional: Update documentation (e.g., the [supported entities list](../supported_entities.md)).
@@ -232,7 +232,7 @@ Additional examples can be found in the [OpenAPI spec](../api-docs/api-docs.html
### Reading pattern recognizers from YAML ### Reading pattern recognizers from YAML
Recognizers can be loaded from a YAML file, which allows users to add recognition logic without writing code. Recognizers can be loaded from a YAML file, which allows users to add recognition logic without writing code.
An example YAML file can be found [here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml). An example YAML file can be found [here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml).
Once the YAML file is created, it can be loaded into the `RecognizerRegistry` instance. Once the YAML file is created, it can be loaded into the `RecognizerRegistry` instance.

6
docs/analyzer/analyzer_engine_provider.md
View File

@@ -139,9 +139,9 @@ print(results)
The default configuration of `AnalyzerEngine` is defined in the following files: The default configuration of `AnalyzerEngine` is defined in the following files:
- [Analyzer Engine](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml) - [Analyzer Engine](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)
- [NLP Engine](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) - [NLP Engine](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)
- [Recognizer Registry](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) - [Recognizer Registry](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)
## Enabling and disabling recognizers ## Enabling and disabling recognizers
In general, recognizers that are not added to the configuration would not be created, with one exception. In general, recognizers that are not added to the configuration would not be created, with one exception.

6
docs/analyzer/customizing_nlp_models.md
View File

@@ -50,7 +50,7 @@ Configuration can be done in two ways:
print(results_english) print(results_english)
``` ```
- **Via configuration**: Set up the models which should be used in the [default `conf` file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml). - **Via configuration**: Set up the models which should be used in the [default `conf` file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml).
An example Conf file: An example Conf file:
@@ -86,7 +86,7 @@ Configuration can be done in two ways:
- `low_confidence_score_multiplier`: A multiplier to apply to the score of entities with low confidence. - `low_confidence_score_multiplier`: A multiplier to apply to the score of entities with low confidence.
- `low_score_entity_names`: A list of entity types to apply the low confidence score multiplier to. - `low_score_entity_names`: A list of entity types to apply the low confidence score multiplier to.
The [default conf file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) is read during the default initialization of the `AnalyzerEngine`. Alternatively, the path to a custom configuration file can be passed to the `NlpEngineProvider`: The [default conf file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) is read during the default initialization of the `AnalyzerEngine`. Alternatively, the path to a custom configuration file can be passed to the `NlpEngineProvider`:
```python ```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer import AnalyzerEngine, RecognizerRegistry
@@ -119,7 +119,7 @@ Configuration can be done in two ways:
!!! note "Note" !!! note "Note"
Presidio can currently use one NER model per language via the `NlpEngine`. If multiple are required, Presidio can currently use one NER model per language via the `NlpEngine`. If multiple are required,
consider wrapping NER models as additional recognizers ([see sample here](https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_remote_recognizer.py)). consider wrapping NER models as additional recognizers ([see sample here](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py)).
## Leverage frameworks other than spaCy, Stanza and transformers for ML based PII detection ## Leverage frameworks other than spaCy, Stanza and transformers for ML based PII detection

2
docs/analyzer/decision_process.md
View File

@@ -102,7 +102,7 @@ The following traces will be written to log, with this format:
When creating new PII recognizers, it is possible to add information about the recognizer's decision process. This information will be traced or returned to the user, depending on the configuration. When creating new PII recognizers, it is possible to add information about the recognizer's decision process. This information will be traced or returned to the user, depending on the configuration.
For example, the [spacy_recognizer.py](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/nlp_engine_recognizers/spacy_recognizer.py) implements a custom trace as follows: For example, the [spacy_recognizer.py](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/nlp_engine_recognizers/spacy_recognizer.py) implements a custom trace as follows:
```python ```python
SPACY_DEFAULT_EXPLANATION = "Identified as {} by Spacy's Named Entity Recognition" SPACY_DEFAULT_EXPLANATION = "Identified as {} by Spacy's Named Entity Recognition"

16
docs/analyzer/developing_recognizers.md
View File

@@ -9,13 +9,13 @@ Recognizers define the logic for detection, as well as the confidence a predicti
Each recognizer, regardless of its complexity, could have false positives and false negatives. When adding new recognizers, we try to balance the effect of each recognizer on the entire system. Each recognizer, regardless of its complexity, could have false positives and false negatives. When adding new recognizers, we try to balance the effect of each recognizer on the entire system.
A recognizer with many false positives would affect the system's usability, while a recognizer with many false negatives might require more work before it can be integrated. For reproducibility purposes, it is be best to note how the recognizer's accuracy was tested, and on which datasets. A recognizer with many false positives would affect the system's usability, while a recognizer with many false negatives might require more work before it can be integrated. For reproducibility purposes, it is be best to note how the recognizer's accuracy was tested, and on which datasets.
For tools and documentation on evaluating and analyzing recognizers, refer to the [presidio-research Github repository](https://github.com/microsoft/presidio-research). For tools and documentation on evaluating and analyzing recognizers, refer to the [presidio-research GitHub repository](https://github.com/data-privacy-stack/presidio-research).
!!! note "Note" !!! note "Note"
When contributing recognizers to the Presidio OSS, When contributing recognizers to the Presidio OSS,
new predefined recognizers should be added to the new predefined recognizers should be added to the
[supported entities list](../supported_entities.md), [supported entities list](../supported_entities.md),
and follow the [contribution guidelines](https://github.com/microsoft/presidio/blob/main/CONTRIBUTING.md). and follow the [contribution guidelines](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md).
### Performance ### Performance
@@ -24,7 +24,7 @@ Make sure your recognizer doesn't take too long to process text. Anything above
### Environment ### Environment
When adding new recognizers that have 3rd party dependencies, make sure that the new dependencies don't interfere with Presidio's dependencies. When adding new recognizers that have 3rd party dependencies, make sure that the new dependencies don't interfere with Presidio's dependencies.
In the case of a conflict, one can create an isolated model environment (outside the main presidio-analyzer process) and implement a [`RemoteRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/remote_recognizer.py) on the presidio-analyzer side to interact with the model's endpoint. In the case of a conflict, one can create an isolated model environment (outside the main presidio-analyzer process) and implement a [`RemoteRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/remote_recognizer.py) on the presidio-analyzer side to interact with the model's endpoint.
## Recognizer Types ## Recognizer Types
@@ -34,17 +34,17 @@ Generally speaking, there are three types of recognizers:
A deny list is a list of words that should be removed during text analysis. For example, it can include a list of titles (`["Mr.", "Mrs.", "Ms.", "Dr."]` to detect a "Title" entity.) A deny list is a list of words that should be removed during text analysis. For example, it can include a list of titles (`["Mr.", "Mrs.", "Ms.", "Dr."]` to detect a "Title" entity.)
See [this documentation](adding_recognizers.md) on adding a new recognizer. The [`PatternRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/pattern_recognizer.py) class has built-in support for a deny-list input. See [this documentation](adding_recognizers.md) on adding a new recognizer. The [`PatternRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/pattern_recognizer.py) class has built-in support for a deny-list input.
### Pattern Based ### Pattern Based
Pattern based recognizers use regular expressions to identify entities in text. Pattern based recognizers use regular expressions to identify entities in text.
See [this documentation](adding_recognizers.md) on adding a new recognizer via code. See [this documentation](adding_recognizers.md) on adding a new recognizer via code.
The [`PatternRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/pattern_recognizer.py) class should be extended. The [`PatternRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/pattern_recognizer.py) class should be extended.
See some examples here: See some examples here:
!!! example "Examples" !!! example "Examples"
Examples of pattern based recognizers are the [`CreditCardRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py) and [`EmailRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/email_recognizer.py). Examples of pattern based recognizers are the [`CreditCardRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py) and [`EmailRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/email_recognizer.py).
### Machine Learning (ML) Based or Rule-Based ### Machine Learning (ML) Based or Rule-Based
@@ -58,13 +58,13 @@ Presidio currently uses [spaCy](https://spacy.io/) as a framework for text analy
`spaCy`, `stanza` and `transformers` models could be trained from scratch, used in combination with pre-trained embeddings, or be fine-tuned. `spaCy`, `stanza` and `transformers` models could be trained from scratch, used in combination with pre-trained embeddings, or be fine-tuned.
In addition to those, it is also possible to use other ML models. In that case, a new `EntityRecognizer` should be created. In addition to those, it is also possible to use other ML models. In that case, a new `EntityRecognizer` should be created.
See an example using [Flair here](https://github.com/microsoft/presidio/blob/main/docs/samples/python/flair_recognizer.py). See an example using [Flair here](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py).
#### Apply Custom Logic #### Apply Custom Logic
In some cases, rule-based logic provides reasonable ways for detecting entities. In some cases, rule-based logic provides reasonable ways for detecting entities.
The Presidio `EntityRecognizer` API allows you to use `spaCy` extracted features like lemmas, part of speech, dependencies and more to create your logic. The Presidio `EntityRecognizer` API allows you to use `spaCy` extracted features like lemmas, part of speech, dependencies and more to create your logic.
When integrating such logic into Presidio, a class inheriting from the [`EntityRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) should be created. When integrating such logic into Presidio, a class inheriting from the [`EntityRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py) should be created.
!!! attention "Considerations for selecting one option over another" !!! attention "Considerations for selecting one option over another"

2
docs/analyzer/index.md
View File

@@ -195,7 +195,7 @@ For a list of the current supported entities:
## API reference ## API reference
Follow the [API Spec](https://microsoft.github.io/presidio/api-docs/api-docs.html#tag/Analyzer) for the Analyzer REST API reference details and [Analyzer Python API](../api/analyzer_python.md) for Python API reference Follow the [API Spec](https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html#tag/Analyzer) for the Analyzer REST API reference details and [Analyzer Python API](../api/analyzer_python.md) for Python API reference
## Samples ## Samples

8
docs/analyzer/languages.md
View File

@@ -64,14 +64,14 @@ analyzer = AnalyzerEngine(
analyzer.analyze(text="My name is David", language="en") analyzer.analyze(text="My name is David", language="en")
``` ```
Link to LANGUAGES_CONFIG_FILE=[languages-config.yml](https://github.com/microsoft/presidio/blob/main/docs/analyzer/languages-config.yml) Link to LANGUAGES_CONFIG_FILE=[languages-config.yml](https://github.com/data-privacy-stack/presidio/blob/main/docs/analyzer/languages-config.yml)
### Automatically install NLP models into the Docker container ### Automatically install NLP models into the Docker container
When packaging the code into a Docker container, NLP models are automatically installed. When packaging the code into a Docker container, NLP models are automatically installed.
To define which models should be installed, To define which models should be installed,
update the [conf/default.yaml](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) file. This file is read during update the [conf/default.yaml](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) file. This file is read during
the `docker build` phase and the models defined in it are installed automatically. the `docker build` phase and the models defined in it are installed automatically.
For `transformers` based models, the configuration [can be found here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/transformers.yaml). For `transformers` based models, the configuration [can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/transformers.yaml).
A docker file supporting transformers models [can be found here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/Dockerfile.transformers). A docker file supporting transformers models [can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/Dockerfile.transformers).

2
docs/analyzer/nlp_engines/gpu_usage.md
View File

@@ -18,7 +18,7 @@ export PRESIDIO_DEVICE=cuda:0
export PRESIDIO_DEVICE=cuda:1 export PRESIDIO_DEVICE=cuda:1
# Or when running Docker containers # Or when running Docker containers
docker run -e PRESIDIO_DEVICE=cuda mcr.microsoft.com/presidio-analyzer:latest docker run -e PRESIDIO_DEVICE=cuda ghcr.io/data-privacy-stack/presidio-analyzer:latest
``` ```
!!! note "Priority" !!! note "Priority"

2
docs/analyzer/nlp_engines/spacy_stanza.md
View File

@@ -55,7 +55,7 @@ sequenceDiagram
!!! note "Note" !!! note "Note"
A labeled dataset containing text and labeled PII entities is required for training a new model. A labeled dataset containing text and labeled PII entities is required for training a new model.
For more information on model training and evaluation for Presidio, see the [Presidio-Research Github repository](https://github.com/microsoft/presidio-research). For more information on model training and evaluation for Presidio, see the [Presidio-Research Github repository](https://github.com/data-privacy-stack/presidio-research).
To train your own model, see these links on spaCy and Stanza: To train your own model, see these links on spaCy and Stanza:

2
docs/analyzer/nlp_engines/transformers.md
View File

@@ -224,7 +224,7 @@ Once created, see [the NLP configuration documentation](../customizing_nlp_model
!!! note "Note" !!! note "Note"
A labeled dataset containing text and labeled PII entities is required for training a new model. A labeled dataset containing text and labeled PII entities is required for training a new model.
For more information on model training and evaluation for Presidio, see the [Presidio-Research Github repository](https://github.com/microsoft/presidio-research). For more information on model training and evaluation for Presidio, see the [Presidio-Research Github repository](https://github.com/data-privacy-stack/presidio-research).
To train your own model, see this tutorial: [Train your own transformers model](https://huggingface.co/docs/transformers/training). To train your own model, see this tutorial: [Train your own transformers model](https://huggingface.co/docs/transformers/training).

2
docs/anonymizer/adding_operators.md
View File

@@ -11,7 +11,7 @@ Presidio anonymizer can be easily extended to support additional anonymization a
## Extending presidio-anonymizer for additional PII operators ## Extending presidio-anonymizer for additional PII operators
1. Create new python class implementing the abstract [Operator](https://github.com/microsoft/presidio/blob/main/presidio-anonymizer/presidio_anonymizer/operators/operator.py) class. 1. Create new python class implementing the abstract [Operator](https://github.com/data-privacy-stack/presidio/blob/main/presidio-anonymizer/presidio_anonymizer/operators/operator.py) class.
2. Implement the methods: 2. Implement the methods:
- `operate` - gets the data and returns a new text expected to replace the old one. - `operate` - gets the data and returns a new text expected to replace the old one.
- `validate` - validate the parameters entered for the anonymizer exists and valid. - `validate` - validate the parameters entered for the anonymizer exists and valid.

2
docs/anonymizer/index.md
View File

@@ -384,6 +384,6 @@ for more information.
## API reference ## API reference
Follow Follow
the [API Spec](https://microsoft.github.io/presidio/api-docs/api-docs.html#tag/Anonymizer) the [API Spec](https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html#tag/Anonymizer)
for the Anonymizer REST API reference details for the Anonymizer REST API reference details
and [Anonymizer Python API](../api/anonymizer_python.md) for Python API reference. and [Anonymizer Python API](../api/anonymizer_python.md) for Python API reference.

8
docs/api-docs/api-docs.yml
View File

@@ -5,18 +5,18 @@ info:
title: Presidio title: Presidio
contact: contact:
name: Presidio support name: Presidio support
email: presidio@microsoft.com email: presidio@dataprivacystack.org
url: https://github.com/microsoft/presidio url: https://github.com/data-privacy-stack/presidio
x-logo: x-logo:
url: "logo.svg" url: "logo.svg"
altText: "Presidio" altText: "Presidio"
backgroundColor: "#ffffff" backgroundColor: "#ffffff"
license: license:
name: MIT name: MIT
url: 'https://github.com/microsoft/presidio/blob/main/LICENSE' url: 'https://github.com/data-privacy-stack/presidio/blob/main/LICENSE'
externalDocs: externalDocs:
description: Presidio documentation. description: Presidio documentation.
url: 'https://microsoft.github.io/presidio/' url: 'https://data-privacy-stack.github.io/presidio/'
tags: tags:
- name: Analyzer - name: Analyzer
description: Detecting PII entities in text description: Detecting PII entities in text

6
docs/build_release.md
View File

@@ -8,18 +8,18 @@ for code reuse using [YAML Schema](https://docs.microsoft.com/en-us/azure/devops
The following pipelines are provided and maintained as part of presidio development process: The following pipelines are provided and maintained as part of presidio development process:
- [PR Validation](https://github.com/microsoft/presidio/blob/main/azure-pipelines.yml) - used to validate pull requests. - [PR Validation](https://github.com/data-privacy-stack/presidio/blob/main/azure-pipelines.yml) - used to validate pull requests.
- Linting - Linting
- Security and compliance analysis - Security and compliance analysis
- Unit tests - Unit tests
- E2E tests - E2E tests
- [CI](https://github.com/microsoft/presidio/blob/main/azure-pipelines-ci.yml) - triggered on merge to main branch. - [CI](https://github.com/data-privacy-stack/presidio/blob/main/azure-pipelines-ci.yml) - triggered on merge to main branch.
- Linting - Linting
- Security and compliance analysis - Security and compliance analysis
- Unit tests - Unit tests
- E2E tests - E2E tests
- deploys the artifacts to an internal dev environment. - deploys the artifacts to an internal dev environment.
- [Release](https://github.com/microsoft/presidio/blob/main/azure-pipelines.yml) - manually triggered. - [Release](https://github.com/data-privacy-stack/presidio/blob/main/azure-pipelines.yml) - manually triggered.
- releases presidio official artifacts - releases presidio official artifacts
- pypi - pypi
- Microsoft container registry (and docker hub) - Microsoft container registry (and docker hub)

2
docs/community.md
View File

@@ -24,6 +24,6 @@ This section collects different resources developed with Presidio.
| [DataOps for the Modern Data Warehouse](https://github.com/Azure-Samples/modern-data-warehouse-dataops) | Contains numerous code samples and artifacts on how to apply DevOps principles to data pipelines built according to the [Modern Data Warehouse (MDW)](https://azure.microsoft.com/en-au/solutions/architecture/modern-data-warehouse/) architectural pattern on [Microsoft Azure](https://azure.microsoft.com/en-au/). | | [DataOps for the Modern Data Warehouse](https://github.com/Azure-Samples/modern-data-warehouse-dataops) | Contains numerous code samples and artifacts on how to apply DevOps principles to data pipelines built according to the [Modern Data Warehouse (MDW)](https://azure.microsoft.com/en-au/solutions/architecture/modern-data-warehouse/) architectural pattern on [Microsoft Azure](https://azure.microsoft.com/en-au/). |
| [Extending Power BI with Python and R](https://github.com/PacktPublishing/Extending-Power-BI-with-Python-and-R) | Code repository for [Extending Power BI with Python and R](https://www.packtpub.com/product/extending-power-bi-with-python-and-r/9781801078207), published by Packt. | | [Extending Power BI with Python and R](https://github.com/PacktPublishing/Extending-Power-BI-with-Python-and-R) | Code repository for [Extending Power BI with Python and R](https://www.packtpub.com/product/extending-power-bi-with-python-and-r/9781801078207), published by Packt. |
| [HebSafeHarbor](https://github.com/8400TheHealthNetwork/HebSafeHarbor) | Clinical notes anonymization in Hebrew. | | [HebSafeHarbor](https://github.com/8400TheHealthNetwork/HebSafeHarbor) | Clinical notes anonymization in Hebrew. |
| [Presidio Github Action](https://github.com/marketplace/actions/presidio-action) | Github Action that analyzes text for PII entities with Microsoft's Presidio framework. | | [Presidio GitHub Action](https://github.com/marketplace/actions/presidio-action) | GitHub Action that analyzes text for PII entities with the Presidio framework. |
* Please create a PR if you're interested in adding your tool to this list. * Please create a PR if you're interested in adding your tool to this list.

14
docs/evaluation/index.md
View File

@@ -31,17 +31,17 @@ In addition, it provides simple data generation tools to help you create a datas
### Evaluating the Presidio Analyzer using Presidio-Research ### Evaluating the Presidio Analyzer using Presidio-Research
Presidio-Research is a python package with a set of tools that help you evaluate the performance of the Presidio Analyzer. Presidio-Research is a python package with a set of tools that help you evaluate the performance of the Presidio Analyzer.
To get started, follow the instructions in the [Presidio-Research repository](https://github.com/microsoft/presidio-research). To get started, follow the instructions in the [Presidio-Research repository](https://github.com/data-privacy-stack/presidio-research).
The easiest way to get started is by reviewing the notebooks: The easiest way to get started is by reviewing the notebooks:
- [Notebook 1](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb): Shows how to use the PII data generator. - [Notebook 1](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb): Shows how to use the PII data generator.
- [Notebook 2](https://github.com/microsoft/presidio-research/blob/master/notebooks/2_PII_EDA.ipynb): Shows a simple analysis of the PII dataset. - [Notebook 2](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/2_PII_EDA.ipynb): Shows a simple analysis of the PII dataset.
- [Notebook 3](https://github.com/microsoft/presidio-research/blob/master/notebooks/3_Split_by_pattern_number.ipynb): Provides tools to split the dataset into train/test/validation sets while avoiding leakage due to the same pattern appearing in multiple folds (only applicable for synthetically generated data). - [Notebook 3](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/3_Split_by_pattern_number.ipynb): Provides tools to split the dataset into train/test/validation sets while avoiding leakage due to the same pattern appearing in multiple folds (only applicable for synthetically generated data).
- [Notebook 4](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb): Shows how to use the evaluation tools to evaluate how well Presidio detects PII. Note that this is using the vanilla Presidio, and the results aren't very accurate. - [Notebook 4](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb): Shows how to use the evaluation tools to evaluate how well Presidio detects PII. Note that this is using the vanilla Presidio, and the results aren't very accurate.
- [Notebook 5](https://github.com/microsoft/presidio-research/blob/master/notebooks/5_Evaluate_Custom_Presidio_Analyzer.ipynb): Shows how one can configure Presidio to detect PII much more accurately, and boost the f score in ~30%. - [Notebook 5](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/5_Evaluate_Custom_Presidio_Analyzer.ipynb): Shows how one can configure Presidio to detect PII much more accurately, and boost the f score in ~30%.
For more information and advanced usage, refer to the [Presidio-Research repository](https://github.com/microsoft/presidio-research). For more information and advanced usage, refer to the [Presidio-Research repository](https://github.com/data-privacy-stack/presidio-research).
### Evaluating DICOM redaction with Presidio Image Redactor ### Evaluating DICOM redaction with Presidio Image Redactor

54
docs/faq.md
View File

@@ -2,8 +2,8 @@
- [General](#general) - [General](#general)
- [What is Presidio?](#what-is-presidio) - [What is Presidio?](#what-is-presidio)
- [Why did Microsoft create Presidio?](#why-did-microsoft-create-presidio) - [Why was Presidio created?](#why-was-presidio-created)
- [Is Microsoft Presidio an official Microsoft product?](#is-microsoft-presidio-an-official-microsoft-product) - [Who owns and maintains Presidio?](#who-owns-and-maintains-presidio)
- [What is the difference between Presidio and different PII detection services like Azure AI Language, Azure Health Data Services, and Amazon Comprehend?](#what-is-the-difference-between-presidio-and-different-pii-detection-services-like-azure-ai-language-and-amazon-comprehend) - [What is the difference between Presidio and different PII detection services like Azure AI Language, Azure Health Data Services, and Amazon Comprehend?](#what-is-the-difference-between-presidio-and-different-pii-detection-services-like-azure-ai-language-and-amazon-comprehend)
- [Using Presidio](#using-presidio) - [Using Presidio](#using-presidio)
- [How can I start using Presidio?](#how-can-i-start-using-presidio) - [How can I start using Presidio?](#how-can-i-start-using-presidio)
@@ -35,7 +35,7 @@ Presidio (Origin from Latin praesidium ‘protection, garrison’) helps to ensu
!!! warning "Warning" !!! warning "Warning"
Presidio can help identify sensitive/PII data in un/structured text. However, because it is using automated detection mechanisms, there is no guarantee that Presidio will find all sensitive information. Consequently, additional systems and protections should be employed. Presidio can help identify sensitive/PII data in un/structured text. However, because it is using automated detection mechanisms, there is no guarantee that Presidio will find all sensitive information. Consequently, additional systems and protections should be employed.
### Why did Microsoft create Presidio? ### Why was Presidio created?
By developing Presidio, our goals are: By developing Presidio, our goals are:
@@ -43,35 +43,35 @@ By developing Presidio, our goals are:
2. Embrace extensibility and customizability to a specific business need. 2. Embrace extensibility and customizability to a specific business need.
3. Facilitate both fully automated and semi-automated PII de-identification flows on multiple platforms. 3. Facilitate both fully automated and semi-automated PII de-identification flows on multiple platforms.
### Is Microsoft Presidio an official Microsoft product? ### Who owns and maintains Presidio?
The authors and maintainers of Presidio come from the [Industry Solutions Engineering](https://microsoft.github.io/code-with-engineering-playbook) team. We work with customers on various engineering problems, and have found the proper handling of private and sensitive data a recurring challenge across many customers and industries. Presidio is a community-governed open source project under the [Data Privacy Stack](https://github.com/data-privacy-stack) organization. It was originally created at Microsoft and has since transitioned to an independent, vendor-neutral project maintained by contributors and volunteers from across the community. See the [project transition update](project_transition.md) for more details.
!!! note "Note" !!! note "Note"
Microsoft Presidio is not an official Microsoft product. Usage terms are defined in the [repository's license](https://github.com/microsoft/presidio/blob/main/LICENSE). Presidio is not an official product of any company and comes with no warranty or SLA. Usage terms are defined in the [repository's license](https://github.com/data-privacy-stack/presidio/blob/main/LICENSE).
### What is the difference between Presidio and different PII detection services like Azure AI Language, Azure Health Data Services, and Amazon Comprehend? ### What is the difference between Presidio and different PII detection services like Azure AI Language, Azure Health Data Services, and Amazon Comprehend?
In a nutshell, Presidio is a library which is meant to be customized, whereas different SaaS tools for PII detection have less customization capabilities. Most of these SaaS offerings use dedicated ML models and other logic for PII detection and often have better entity coverage or accuracy than Presidio. In a nutshell, Presidio is a library which is meant to be customized, whereas different SaaS tools for PII detection have less customization capabilities. Most of these SaaS offerings use dedicated ML models and other logic for PII detection and often have better entity coverage or accuracy than Presidio.
Based on our internal research, leveraging Presidio in parallel to 3rd party PII detection services like Azure AI Language can bring optimal results mainly when the data in hand has entity types or values not supported by the 3rd party service. ([see example here](https://microsoft.github.io/presidio/samples/python/text_analytics/)). Based on our internal research, leveraging Presidio in parallel to 3rd party PII detection services like Azure AI Language can bring optimal results mainly when the data in hand has entity types or values not supported by the 3rd party service. ([see example here](https://data-privacy-stack.github.io/presidio/samples/python/text_analytics/)).
## Using Presidio ## Using Presidio
### How can I start using Presidio? ### How can I start using Presidio?
1. Check out the [installation docs](https://microsoft.github.io/presidio/installation/). 1. Check out the [installation docs](https://data-privacy-stack.github.io/presidio/installation/).
2. Take a look at the [different samples](https://microsoft.github.io/presidio/samples/). 2. Take a look at the [different samples](https://data-privacy-stack.github.io/presidio/samples/).
3. Try the [demo website](https://aka.ms/presidio-demo). 3. Try the [demo website](https://huggingface.co/spaces/presidio/presidio_demo).
### What are the main building blocks in Presidio? ### What are the main building blocks in Presidio?
Presidio is a suite built of several packages and building blocks: Presidio is a suite built of several packages and building blocks:
1. [Presidio Analyzer](https://microsoft.github.io/presidio/analyzer/): a package for detecting PII entities in natural language. 1. [Presidio Analyzer](https://data-privacy-stack.github.io/presidio/analyzer/): a package for detecting PII entities in natural language.
2. [Presidio Anonymizer](https://microsoft.github.io/presidio/anonymizer/): a package for manipulating PII entities in text (e.g. remove, redact, hash, encrypt). 2. [Presidio Anonymizer](https://data-privacy-stack.github.io/presidio/anonymizer/): a package for manipulating PII entities in text (e.g. remove, redact, hash, encrypt).
3. [Presidio Image Redactor](https://microsoft.github.io/presidio/image-redactor/): A package for detecting PII entities in image using OCR. 3. [Presidio Image Redactor](https://data-privacy-stack.github.io/presidio/image-redactor/): A package for detecting PII entities in image using OCR.
4. [Presidio Structured](https://microsoft.github.io/presidio/structured/): A package for detecting PII entities in structured/semi-structured data. 4. [Presidio Structured](https://data-privacy-stack.github.io/presidio/structured/): A package for detecting PII entities in structured/semi-structured data.
5. A set of sample deployments as Python packages or Docker containers for Kubernetes, Azure Data Factory, Spark and more. 5. A set of sample deployments as Python packages or Docker containers for Kubernetes, Azure Data Factory, Spark and more.
## Customizing Presidio ## Customizing Presidio
@@ -80,11 +80,11 @@ Presidio is a suite built of several packages and building blocks:
Users can customize Presidio in multiple ways: Users can customize Presidio in multiple ways:
1. Create new or updated PII recognizers ([docs](https://microsoft.github.io/presidio/analyzer/adding_recognizers/)). 1. Create new or updated PII recognizers ([docs](https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/)).
2. Adapt Presidio to new languages ([docs](https://microsoft.github.io/presidio/analyzer/languages/)). 2. Adapt Presidio to new languages ([docs](https://data-privacy-stack.github.io/presidio/analyzer/languages/)).
3. Leverage state of the art Named Entity Recognition models ([docs](https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/)). 3. Leverage state of the art Named Entity Recognition models ([docs](https://data-privacy-stack.github.io/presidio/analyzer/customizing_nlp_models/)).
4. Add new types of anonymizers ([docs](https://microsoft.github.io/presidio/anonymizer/adding_operators/)). 4. Add new types of anonymizers ([docs](https://data-privacy-stack.github.io/presidio/anonymizer/adding_operators/)).
5. Create PII analysis and anonymization pipelines on different environments using Docker or Python ([samples](https://microsoft.github.io/presidio/samples/)). 5. Create PII analysis and anonymization pipelines on different environments using Docker or Python ([samples](https://data-privacy-stack.github.io/presidio/samples/)).
And more. And more.
@@ -92,21 +92,21 @@ And more.
Presidio supports spaCy version 3+ for Named Entity Recognition, tokenization, lemmatization and more. We also support [Stanza](https://stanfordnlp.github.io/stanza/) using the [spacy-stanza](https://spacy.io/universe/project/spacy-stanza) package, and it is further possible to create PII recognizers leveraging other frameworks like [transformers](https://huggingface.co/transformers/usage.html#named-entity-recognition) or [Flair](https://github.com/flairNLP/flair). Presidio supports spaCy version 3+ for Named Entity Recognition, tokenization, lemmatization and more. We also support [Stanza](https://stanfordnlp.github.io/stanza/) using the [spacy-stanza](https://spacy.io/universe/project/spacy-stanza) package, and it is further possible to create PII recognizers leveraging other frameworks like [transformers](https://huggingface.co/transformers/usage.html#named-entity-recognition) or [Flair](https://github.com/flairNLP/flair).
For more information, see the [docs](https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/). For more information, see the [docs](https://data-privacy-stack.github.io/presidio/analyzer/customizing_nlp_models/).
### Can Presidio be used for Pseudonymization? ### Can Presidio be used for Pseudonymization?
Pseudonymization is a de-identification technique in which the real data is replaced with fake data in a reversible way. Since there are various ways and approaches for this, we provide a simple [sample](https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_custom_lambda_anonymizer.py) which can be extended for more sophisticated usage. If you have a question or a request on this topic, please open an issue on the repo. Pseudonymization is a de-identification technique in which the real data is replaced with fake data in a reversible way. Since there are various ways and approaches for this, we provide a simple [sample](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_custom_lambda_anonymizer.py) which can be extended for more sophisticated usage. If you have a question or a request on this topic, please open an issue on the repo.
### Does Presidio work on structured/tabular data? ### Does Presidio work on structured/tabular data?
[Presidio-structured](https://microsoft.github.io/presidio/structured/) is a capability in Presidio for detecting PII entities in structured/semi-structured data. It scans datasets for PII using Presidio Analyzer, and supports the redaction of text, cells, or columns in a tabular dataset. [Presidio-structured](https://data-privacy-stack.github.io/presidio/structured/) is a capability in Presidio for detecting PII entities in structured/semi-structured data. It scans datasets for PII using Presidio Analyzer, and supports the redaction of text, cells, or columns in a tabular dataset.
## Improving detection accuracy ## Improving detection accuracy
### What can I do if Presidio does not detect some of the PII entities in my data (False Negatives)? ### What can I do if Presidio does not detect some of the PII entities in my data (False Negatives)?
Presidio comes loaded with several PII recognizers (see [list here](https://microsoft.github.io/presidio/supported_entities/)), Presidio comes loaded with several PII recognizers (see [list here](https://data-privacy-stack.github.io/presidio/supported_entities/)),
however its main strength lies in its customization capabilities to new entities, specific datasets, file types, languages or use cases. however its main strength lies in its customization capabilities to new entities, specific datasets, file types, languages or use cases.
### What can I do if Presidio falsely detects text as PII entities (False Positives)? ### What can I do if Presidio falsely detects text as PII entities (False Positives)?
@@ -126,13 +126,13 @@ Every PII identification logic would have its errors, and there is a trade-off b
### How can I evaluate the performance of my Presidio instance? ### How can I evaluate the performance of my Presidio instance?
In addition to Presidio, we maintain a repo focused on evaluation of models and PII recognizers [here](https://github.com/microsoft/presidio-research). It also features a simple PII data generator. In addition to Presidio, we maintain a repo focused on evaluation of models and PII recognizers [here](https://github.com/data-privacy-stack/presidio-research). It also features a simple PII data generator.
## Deployment ## Deployment
### How can I deploy Presidio into my environment? ### How can I deploy Presidio into my environment?
The main Presidio modules (analyzer, anonymizer, image-redactor) can be used both as a Python package and as a dockerized REST API. See the [different deployment samples](https://microsoft.github.io/presidio/samples/) for example deployments. The main Presidio modules (analyzer, anonymizer, image-redactor) can be used both as a Python package and as a dockerized REST API. See the [different deployment samples](https://data-privacy-stack.github.io/presidio/samples/) for example deployments.
!!! warning "Authentication and Authorization" !!! warning "Authentication and Authorization"
Presidio API endpoints do not include built-in authentication by design. The containers are intentionally kept lean to allow flexibility for different deployment scenarios. Authentication and authorization should be implemented at a separate infrastructure layer (e.g., an API gateway, reverse proxy, or service mesh) according to your specific use case and security requirements. It is strongly recommended not to expose Presidio services directly to untrusted networks without an authentication layer in front of them. Presidio API endpoints do not include built-in authentication by design. The containers are intentionally kept lean to allow flexibility for different deployment scenarios. Authentication and authorization should be implemented at a separate infrastructure layer (e.g., an API gateway, reverse proxy, or service mesh) according to your specific use case and security requirements. It is strongly recommended not to expose Presidio services directly to untrusted networks without an authentication layer in front of them.
@@ -141,8 +141,8 @@ The main Presidio modules (analyzer, anonymizer, image-redactor) can be used bot
### How can I contribute to Presidio? ### How can I contribute to Presidio?
First, review the [contribution guidelines](https://github.com/microsoft/presidio/blob/main/CONTRIBUTING.md), and feel free to reach out by opening an issue, posting a discussion or emailing us at <presidio@microsoft.com> First, review the [contribution guidelines](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md), and feel free to reach out by opening an issue, posting a discussion or emailing us at <presidio@dataprivacystack.org>
### How can I report security vulnerabilities? ### How can I report security vulnerabilities?
Please see the [security information](https://github.com/microsoft/presidio/blob/main/SECURITY.md). Please see the [security information](https://github.com/data-privacy-stack/presidio/blob/main/SECURITY.md).

2
docs/getting_started.md
View File

@@ -1,4 +1,4 @@
# Getting started with Microsoft Presidio # Getting started with Presidio
The core functionality in Presidio is to detect PII in text. Presidio further contains a set of tools that build on top of text PII detection, for example in images, structured data, JSON and more. The core functionality in Presidio is to detect PII in text. Presidio further contains a set of tools that build on top of text PII detection, for example in images, structured data, JSON and more.

4
docs/getting_started/getting_started_images.md
View File

@@ -66,13 +66,13 @@ Presidio provides a Docker containers that you can use to de-identify image data
1. Download Docker image 1. Download Docker image
```sh ```sh
docker pull mcr.microsoft.com/presidio-image-redactor docker pull ghcr.io/data-privacy-stack/presidio-image-redactor
``` ```
2. Run container 2. Run container
```sh ```sh
docker run -d -p 5003:3000 mcr.microsoft.com/presidio-image-redactor docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor
``` ```
3. Use the API 3. Use the API

8
docs/getting_started/getting_started_text.md
View File

@@ -107,16 +107,16 @@ Presidio provides Docker containers that you can use to de-identify text data. E
1. Download Docker images 1. Download Docker images
```sh ```sh
docker pull mcr.microsoft.com/presidio-analyzer docker pull ghcr.io/data-privacy-stack/presidio-analyzer
docker pull mcr.microsoft.com/presidio-anonymizer docker pull ghcr.io/data-privacy-stack/presidio-anonymizer
``` ```
2. Run containers 2. Run containers
```sh ```sh
docker run -d -p 5002:3000 mcr.microsoft.com/presidio-analyzer:latest docker run -d -p 5002:3000 ghcr.io/data-privacy-stack/presidio-analyzer:latest
docker run -d -p 5001:3000 mcr.microsoft.com/presidio-anonymizer:latest docker run -d -p 5001:3000 ghcr.io/data-privacy-stack/presidio-anonymizer:latest
``` ```
3. Use the API 3. Use the API

6
docs/image-redactor/index.md
View File

@@ -45,10 +45,10 @@ Pre-requisites:
```sh ```sh
# Download image from Dockerhub # Download image from Dockerhub
docker pull mcr.microsoft.com/presidio-image-redactor docker pull ghcr.io/data-privacy-stack/presidio-image-redactor
# Run the container with the default port # Run the container with the default port
docker run -d -p 5003:3000 mcr.microsoft.com/presidio-image-redactor:latest docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest
``` ```
=== "From source" === "From source"
@@ -225,6 +225,6 @@ The DICOM data used for unit and integration testing for `DicomImageRedactorEngi
## API reference ## API reference
the [API Spec](https://microsoft.github.io/presidio/api-docs/api-docs.html#tag/Image-redactor) the [API Spec](https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html#tag/Image-redactor)
for the Image Redactor REST API reference details for the Image Redactor REST API reference details
and [Image Redactor Python API](../api/image_redactor_python.md) for Python API reference and [Image Redactor Python API](../api/image_redactor_python.md) for Python API reference

8
docs/index.md
View File

@@ -44,7 +44,7 @@ bitcoin wallets, US phone numbers, financial data and more.
## Demo ## Demo
Link to demo: <https://aka.ms/presidio-demo> Link to demo: <https://huggingface.co/spaces/presidio/presidio_demo>
<iframe width="560" height="315" src="https://www.youtube.com/embed/RPJ3-kEUybU?si=9HolTiQRZmxV7BqV" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> <iframe width="560" height="315" src="https://www.youtube.com/embed/RPJ3-kEUybU?si=9HolTiQRZmxV7BqV" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
@@ -77,6 +77,6 @@ Link to demo: <https://aka.ms/presidio-demo>
## Support ## Support
- Before you submit an issue, please go over the documentation. For general discussions, please use the [Github repo's discussion board](https://github.com/microsoft/presidio/discussions). - Before you submit an issue, please go over the documentation. For general discussions, please use the [GitHub repo's discussion board](https://github.com/data-privacy-stack/presidio/discussions).
- If you have a usage question, found a bug or have a suggestion for improvement, please file a [Github issue](https://github.com/microsoft/presidio/issues). - If you have a usage question, found a bug or have a suggestion for improvement, please file a [GitHub issue](https://github.com/data-privacy-stack/presidio/issues).
- For other matters, please email [presidio@microsoft.com](mailto:presidio@microsoft.com). - For other matters, please email [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org).

16
docs/installation.md
View File

@@ -103,13 +103,13 @@ and `presidio-anonymizer` modules are required.
```sh ```sh
# Download Docker images # Download Docker images
docker pull mcr.microsoft.com/presidio-analyzer docker pull ghcr.io/data-privacy-stack/presidio-analyzer
docker pull mcr.microsoft.com/presidio-anonymizer docker pull ghcr.io/data-privacy-stack/presidio-anonymizer
# Run containers with default ports # Run containers with default ports
docker run -d -p 5002:3000 mcr.microsoft.com/presidio-analyzer:latest docker run -d -p 5002:3000 ghcr.io/data-privacy-stack/presidio-analyzer:latest
docker run -d -p 5001:3000 mcr.microsoft.com/presidio-anonymizer:latest docker run -d -p 5001:3000 ghcr.io/data-privacy-stack/presidio-anonymizer:latest
``` ```
### For PII redaction in images ### For PII redaction in images
@@ -118,10 +118,10 @@ For PII detection in images, the `presidio-image-redactor` is required.
```sh ```sh
# Download Docker image # Download Docker image
docker pull mcr.microsoft.com/presidio-image-redactor docker pull ghcr.io/data-privacy-stack/presidio-image-redactor
# Run container with the default port # Run container with the default port
docker run -d -p 5003:3000 mcr.microsoft.com/presidio-image-redactor:latest docker run -d -p 5003:3000 ghcr.io/data-privacy-stack/presidio-image-redactor:latest
``` ```
Once the services are running, their APIs are available. Once the services are running, their APIs are available.
@@ -134,13 +134,13 @@ To install Presidio from source, first clone the repo:
* using HTTPS * using HTTPS
```sh ```sh
git clone https://github.com/microsoft/presidio.git git clone https://github.com/data-privacy-stack/presidio.git
``` ```
* Using SSH * Using SSH
```sh ```sh
git clone git@github.com:microsoft/presidio.git git clone git@github.com:data-privacy-stack/presidio.git
``` ```
Then, build the containers locally. Then, build the containers locally.

2
docs/learn_presidio/concepts.md
View File

@@ -1,4 +1,4 @@
# Concepts in Microsoft Presidio # Concepts in Presidio
## High-level concepts ## High-level concepts

4
docs/presidio_V2.md
View File

@@ -23,7 +23,7 @@ Version V1 (legacy) is still available for download. To continue using the previ
- For python packages, download version < 2 (e.g. pip install presidio-analyzer==0.95) - For python packages, download version < 2 (e.g. pip install presidio-analyzer==0.95)
!!! note "Note" !!! note "Note"
The legacy V1 code base will continue to be available under branch [V1](https://github.com/microsoft/presidio/tree/V1) but will no longer be officially supported. The legacy V1 code base will continue to be available under branch [V1](https://github.com/data-privacy-stack/presidio/tree/V1) but will no longer be officially supported.
## API Changes ## API Changes
@@ -188,4 +188,4 @@ Below is a detailed outline of all changes made to the Analyzer and Anonymizer.
| FPE (now Encrypt) | <pre>string key = 3t6w9z$C&F)J@NcR;<br>int32 tweak = D8E7920AFA330A73</pre> | <pre>{"key": "3t6w9z$C&F)J@NcR"}</pre> | | FPE (now Encrypt) | <pre>string key = 3t6w9z$C&F)J@NcR;<br>int32 tweak = D8E7920AFA330A73</pre> | <pre>{"key": "3t6w9z$C&F)J@NcR"}</pre> |
!!! note "Note" !!! note "Note"
The V2 API is continuously evolving. please [follow the change log](https://github.com/microsoft/presidio/blob/main/CHANGELOG.md) for updates. The V2 API is continuously evolving. please [follow the change log](https://github.com/data-privacy-stack/presidio/blob/main/CHANGELOG.md) for updates.

18
docs/recipes/CONTRIBUTING.md
View File

@@ -10,7 +10,7 @@ A good recipe should:
2. **Be Reproducible**: Include working code that others can run 2. **Be Reproducible**: Include working code that others can run
3. **Follow the Research Pattern**: Build upon the end-to-end approach from [presidio-research](https://github.com/microsoft/presidio-research), showing: 3. **Follow the Research Pattern**: Build upon the end-to-end approach from [presidio-research](https://github.com/data-privacy-stack/presidio-research), showing:
- **Data Synthesis**: How to generate or obtain test data - **Data Synthesis**: How to generate or obtain test data
- **Configuration**: Your Presidio setup with any custom recognizers - **Configuration**: Your Presidio setup with any custom recognizers
- **Evaluation**: Metrics (precision, recall, F₂) showing performance - **Evaluation**: Metrics (precision, recall, F₂) showing performance
@@ -21,11 +21,11 @@ A good recipe should:
### Follow Presidio Research Examples ### Follow Presidio Research Examples
Your recipe should follow the end-to-end evaluation approach from [presidio-research](https://github.com/microsoft/presidio-research). Your recipe should follow the end-to-end evaluation approach from [presidio-research](https://github.com/data-privacy-stack/presidio-research).
**Key reference notebooks:** **Key reference notebooks:**
- [Evaluate Presidio Analyzer](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete evaluation workflow example - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete evaluation workflow example
- [Generate Synthetic Data](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Using Presidio Evaluator data generator - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Using Presidio Evaluator data generator
### Option 1: Single Jupyter Notebook (Recommended for Simple Cases) ### Option 1: Single Jupyter Notebook (Recommended for Simple Cases)
@@ -62,7 +62,7 @@ your-recipe-name/
### Required Components ### Required Components
Your recipe should include: Your recipe should include:
- **Data Synthesis**: Generate synthetic data using [Presidio Evaluator](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or your own method - **Data Synthesis**: Generate synthetic data using [Presidio Evaluator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or your own method
- **Presidio Configuration**: Show your setup (default, custom recognizers, or custom models) - **Presidio Configuration**: Show your setup (default, custom recognizers, or custom models)
- **Evaluation**: Measure and report precision, recall, F₂ score, latency - **Evaluation**: Measure and report precision, recall, F₂ score, latency
- **Key Findings**: Brief summary of results and when to use this approach - **Key Findings**: Brief summary of results and when to use this approach
@@ -90,9 +90,9 @@ Include at minimum:
## Examples to Learn From ## Examples to Learn From
**Presidio Research Notebooks** (recommended starting point): **Presidio Research Notebooks** (recommended starting point):
- [Evaluate Presidio Analyzer](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete end-to-end evaluation example - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete end-to-end evaluation example
- [Generate Synthetic Data](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Presidio Evaluator data generator - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Presidio Evaluator data generator
- [Other presidio-research notebooks](https://github.com/microsoft/presidio-research/tree/master/notebooks) - Additional examples and tools - [Other presidio-research notebooks](https://github.com/data-privacy-stack/presidio-research/tree/master/notebooks) - Additional examples and tools
**Additional Resources:** **Additional Resources:**
- [Presidio Samples](../samples/index.md): Integration patterns and usage examples - [Presidio Samples](../samples/index.md): Integration patterns and usage examples
@@ -100,7 +100,7 @@ Include at minimum:
## Questions? ## Questions?
- Open an issue with the `recipe` label - Open an issue with the `recipe` label
- Email [presidio@microsoft.com](mailto:presidio@microsoft.com) - Email [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org)
- Tag @omri374 in your PR for guidance - Tag @omri374 in your PR for guidance
## License ## License

4
docs/recipes/german-language-support/README.md
View File

@@ -99,9 +99,9 @@ LOCATION, and ORGANIZATION on top of the pattern recognizers.
## Results ## Results
Formal evaluation against a labelled German dataset has not yet been performed. Formal evaluation against a labelled German dataset has not yet been performed.
To benchmark this recipe follow the [Presidio Research evaluation workflow](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb): To benchmark this recipe follow the [Presidio Research evaluation workflow](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb):
1. Generate synthetic German text with the [data generator](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) 1. Generate synthetic German text with the [data generator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb)
2. Configure the analyzer with `spacy_en_de.yaml` 2. Configure the analyzer with `spacy_en_de.yaml`
3. Run the evaluator and report precision / recall / F₂ / latency 3. Run the evaluator and report precision / recall / F₂ / latency

16
docs/recipes/index.md
View File

@@ -1,6 +1,6 @@
# Recipes Gallery # Recipes Gallery
Welcome to the Presidio Recipes Gallery! This section provides curated, end-to-end examples demonstrating how to customize Microsoft Presidio for specific data privacy and de-identification scenarios. Welcome to the Presidio Recipes Gallery! This section provides curated, end-to-end examples demonstrating how to customize Presidio for specific data privacy and de-identification scenarios.
## What are Recipes? ## What are Recipes?
@@ -25,10 +25,10 @@ While Presidio's documentation covers the fundamentals, recipes bridge the gap b
Each recipe typically includes: Each recipe typically includes:
- **Scenario Description**: The domain and data type - **Scenario Description**: The domain and data type
- **Data Synthesis**: Methods for generating test data using [Presidio Evaluator](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or custom methods - **Data Synthesis**: Methods for generating test data using [Presidio Evaluator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or custom methods
- **Configuration**: Presidio setup with any custom recognizers or models - **Configuration**: Presidio setup with any custom recognizers or models
- **Evaluation**: Performance metrics (precision, recall, F₂ score, latency) - **Evaluation**: Performance metrics (precision, recall, F₂ score, latency)
- **Implementation**: Jupyter notebook or Python scripts showing the end-to-end flow (see [example](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb)) - **Implementation**: Jupyter notebook or Python scripts showing the end-to-end flow (see [example](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb))
- **Key Findings**: When to use this approach and trade-offs to consider - **Key Findings**: When to use this approach and trade-offs to consider
For complex flows, consider breaking into multiple notebooks or scripts for better organization. For complex flows, consider breaking into multiple notebooks or scripts for better organization.
@@ -75,8 +75,8 @@ Each cell will contain:
We welcome community contributions! See our [contribution guidelines](CONTRIBUTING.md) for details. We welcome community contributions! See our [contribution guidelines](CONTRIBUTING.md) for details.
**Reference Examples:** **Reference Examples:**
- [Evaluate Presidio Analyzer](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete end-to-end evaluation workflow - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete end-to-end evaluation workflow
- [Generate Synthetic Data](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Presidio Evaluator data generator - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Presidio Evaluator data generator
Follow the pattern: **Data Synthesis** → **Configuration** → **Evaluation** Follow the pattern: **Data Synthesis** → **Configuration** → **Evaluation**
@@ -87,15 +87,15 @@ For complex flows, break into multiple notebooks or scripts. Focus on getting wo
- [Presidio Samples](../samples/index.md): Additional usage examples and integration patterns - [Presidio Samples](../samples/index.md): Additional usage examples and integration patterns
- [Tutorial Series](../tutorial/index.md): Step-by-step guide to Presidio features - [Tutorial Series](../tutorial/index.md): Step-by-step guide to Presidio features
- [Best Practices for Developing Recognizers](../analyzer/developing_recognizers.md): Deep dive into creating custom PII recognizers - [Best Practices for Developing Recognizers](../analyzer/developing_recognizers.md): Deep dive into creating custom PII recognizers
- [Presidio Research Repository](https://github.com/microsoft/presidio-research): Evaluation tools and research datasets - [Presidio Research Repository](https://github.com/data-privacy-stack/presidio-research): Evaluation tools and research datasets
- [FAQ](../faq.md): Common questions about improving detection accuracy - [FAQ](../faq.md): Common questions about improving detection accuracy
## Questions or Feedback? ## Questions or Feedback?
If you have questions about recipes or suggestions for new scenarios to cover, please: If you have questions about recipes or suggestions for new scenarios to cover, please:
- Open an issue on [GitHub](https://github.com/microsoft/presidio/issues) - Open an issue on [GitHub](https://github.com/data-privacy-stack/presidio/issues)
- Email us at [presidio@microsoft.com](mailto:presidio@microsoft.com) - Email us at [presidio@dataprivacystack.org](mailto:presidio@dataprivacystack.org)
- Join the discussion in our [community channels](../community.md) - Join the discussion in our [community channels](../community.md)
--- ---

8
docs/recipes/template.md
View File

@@ -57,15 +57,15 @@ Describe your approach in a few sentences. What recognizers, models, or techniqu
### Recommended Approach: Follow Presidio Research Patterns ### Recommended Approach: Follow Presidio Research Patterns
Provide an end-to-end example following the patterns from [presidio-research](https://github.com/microsoft/presidio-research): Provide an end-to-end example following the patterns from [presidio-research](https://github.com/data-privacy-stack/presidio-research):
**Reference notebooks:** **Reference notebooks:**
- [Evaluate Presidio Analyzer](https://github.com/microsoft/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete evaluation workflow - [Evaluate Presidio Analyzer](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/4_Evaluate_Presidio_Analyzer.ipynb) - Complete evaluation workflow
- [Generate Synthetic Data](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Using Presidio Evaluator data generator - [Generate Synthetic Data](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) - Using Presidio Evaluator data generator
**Structure your example:** **Structure your example:**
1. **Data Synthesis**: Generate synthetic data using [Presidio Evaluator](https://github.com/microsoft/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or your own method 1. **Data Synthesis**: Generate synthetic data using [Presidio Evaluator](https://github.com/data-privacy-stack/presidio-research/blob/master/notebooks/1_Generate_data.ipynb) or your own method
2. **Configuration**: Set up Presidio with your custom recognizers or models 2. **Configuration**: Set up Presidio with your custom recognizers or models
3. **Evaluation**: Measure precision, recall, and F₂ score 3. **Evaluation**: Measure precision, recall, and F₂ score
4. **Analysis**: Discuss results and trade-offs 4. **Analysis**: Discuss results and trade-offs

4
docs/samples/deployments/app-service/index.md
View File

@@ -7,7 +7,7 @@ Azure App Service provides a managed production environment, which supports dock
Use the following button to deploy presidio services to your Azure subscription. Use the following button to deploy presidio services to your Azure subscription.
[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fapp-service%2Fpresidio-services.json) [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdata-privacy-stack%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fapp-service%2Fpresidio-services.json)
## Deploy using command-line script ## Deploy using command-line script
@@ -21,7 +21,7 @@ APP_SERVICE_NAME=<name of app service>
LOCATION=<location> LOCATION=<location>
APP_SERVICE_SKU=<sku> APP_SERVICE_SKU=<sku>
IMAGE_NAME=mcr.microsoft.com/presidio-analyzer IMAGE_NAME=ghcr.io/data-privacy-stack/presidio-analyzer
# the following parameters are only required if you build and deploy your own containers from a private registry # the following parameters are only required if you build and deploy your own containers from a private registry
ACR_USER_NAME=<user name> ACR_USER_NAME=<user name>
ACR_USER_PASSWORD=<password> ACR_USER_PASSWORD=<password>

12
docs/samples/deployments/app-service/presidio-services.json
View File

@@ -102,21 +102,21 @@
}, },
"analyzerImage": { "analyzerImage": {
"type": "string", "type": "string",
"defaultValue": "mcr.microsoft.com/presidio-analyzer:latest", "defaultValue": "ghcr.io/data-privacy-stack/presidio-analyzer:latest",
"metadata": { "metadata": {
"description": "The full analyzer image name. Only required if image is not hosted on a public registry (imageIsPublic is false)" "description": "The full analyzer image name. Only required if image is not hosted on a public registry (imageIsPublic is false)"
} }
}, },
"anonymizerImage": { "anonymizerImage": {
"type": "string", "type": "string",
"defaultValue": "mcr.microsoft.com/presidio-anonymizer:latest", "defaultValue": "ghcr.io/data-privacy-stack/presidio-anonymizer:latest",
"metadata": { "metadata": {
"description": "The full anonymizer image name. Only required if image is not hosted on a public registry (imageIsPublic is false)" "description": "The full anonymizer image name. Only required if image is not hosted on a public registry (imageIsPublic is false)"
} }
}, },
"imageRedactorImage": { "imageRedactorImage": {
"type": "string", "type": "string",
"defaultValue": "mcr.microsoft.com/presidio-image-redactor:latest", "defaultValue": "ghcr.io/data-privacy-stack/presidio-image-redactor:latest",
"metadata": { "metadata": {
"description": "The full image redactor image name. Only required if image is not hosted on a public registry (imageIsPublic is false)" "description": "The full image redactor image name. Only required if image is not hosted on a public registry (imageIsPublic is false)"
} }
@@ -168,7 +168,7 @@
"properties": { "properties": {
"mode": "Incremental", "mode": "Incremental",
"templateLink": { "templateLink": {
"uri": "https://raw.githubusercontent.com/microsoft/presidio/main/docs/samples/deployments/app-service/presidio-app-service.json", "uri": "https://raw.githubusercontent.com/data-privacy-stack/presidio/main/docs/samples/deployments/app-service/presidio-app-service.json",
"contentVersion": "1.0.0.0" "contentVersion": "1.0.0.0"
}, },
"parameters": { "parameters": {
@@ -213,7 +213,7 @@
"properties": { "properties": {
"mode": "Incremental", "mode": "Incremental",
"templateLink": { "templateLink": {
"uri": "https://raw.githubusercontent.com/microsoft/presidio/main/docs/samples/deployments/app-service/presidio-app-service.json", "uri": "https://raw.githubusercontent.com/data-privacy-stack/presidio/main/docs/samples/deployments/app-service/presidio-app-service.json",
"contentVersion": "1.0.0.0" "contentVersion": "1.0.0.0"
}, },
"parameters": { "parameters": {
@@ -258,7 +258,7 @@
"properties": { "properties": {
"mode": "Incremental", "mode": "Incremental",
"templateLink": { "templateLink": {
"uri": "https://raw.githubusercontent.com/microsoft/presidio/main/docs/samples/deployments/app-service/presidio-app-service.json", "uri": "https://raw.githubusercontent.com/data-privacy-stack/presidio/main/docs/samples/deployments/app-service/presidio-app-service.json",
"contentVersion": "1.0.0.0" "contentVersion": "1.0.0.0"
}, },
"parameters": { "parameters": {

4
docs/samples/deployments/data-factory/arm-templates/azure-deploy-adf-app-service.json
View File

@@ -71,7 +71,7 @@
"variables": { "variables": {
"AzureBlobStorage_connectionString": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('AzureBlobStorage_accountName'), ';EndpointSuffix=core.windows.net')]", "AzureBlobStorage_connectionString": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('AzureBlobStorage_accountName'), ';EndpointSuffix=core.windows.net')]",
"factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('dataFactoryName'))]", "factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('dataFactoryName'))]",
"Dataset_url": "https://raw.githubusercontent.com/microsoft/presidio-research/master/tests/data/generated_large.txt", "Dataset_url": "https://raw.githubusercontent.com/data-privacy-stack/presidio-research/master/tests/data/generated_large.txt",
"Analyzer_url": "[concat('https://', parameters('Analyzer_appName'), '.azurewebsites.net/analyze')]", "Analyzer_url": "[concat('https://', parameters('Analyzer_appName'), '.azurewebsites.net/analyze')]",
"Anonymizer_url": "[concat('https://', parameters('Anonymizer_appName'), '.azurewebsites.net/anonymize')]", "Anonymizer_url": "[concat('https://', parameters('Anonymizer_appName'), '.azurewebsites.net/anonymize')]",
"AzureKeyVault_baseUrl": "[concat('https://', parameters('AzureKeyVault_name'), '.vault.azure.net/')]" "AzureKeyVault_baseUrl": "[concat('https://', parameters('AzureKeyVault_name'), '.vault.azure.net/')]"
@@ -84,7 +84,7 @@
"properties": { "properties": {
"mode": "Incremental", "mode": "Incremental",
"templateLink": { "templateLink": {
"uri": "https://raw.githubusercontent.com/microsoft/presidio/main/docs/samples/deployments/app-service/presidio-services.json", "uri": "https://raw.githubusercontent.com/data-privacy-stack/presidio/main/docs/samples/deployments/app-service/presidio-services.json",
"contentVersion": "1.0.0.0" "contentVersion": "1.0.0.0"
}, },
"parameters": { "parameters": {

2
docs/samples/deployments/data-factory/arm-templates/azure-deploy-adf-databricks.json
View File

@@ -81,7 +81,7 @@
"variables": { "variables": {
"AzureBlobStorage_connectionString": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('AzureBlobStorage_accountName'), ';EndpointSuffix=core.windows.net')]", "AzureBlobStorage_connectionString": "[concat('DefaultEndpointsProtocol=https;AccountName=', parameters('AzureBlobStorage_accountName'), ';EndpointSuffix=core.windows.net')]",
"factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('dataFactoryName'))]", "factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('dataFactoryName'))]",
"Dataset_url": "https://raw.githubusercontent.com/microsoft/presidio-research/master/tests/data/generated_large.txt", "Dataset_url": "https://raw.githubusercontent.com/data-privacy-stack/presidio-research/master/tests/data/generated_large.txt",
"AzureKeyVault_baseUrl": "[concat('https://', parameters('AzureKeyVault_name'), '.vault.azure.net/')]", "AzureKeyVault_baseUrl": "[concat('https://', parameters('AzureKeyVault_name'), '.vault.azure.net/')]",
"storageAccountId": "[concat(resourceGroup().id, '/providers/Microsoft.Storage/storageAccounts/', parameters('AzureBlobStorage_accountName'))]" "storageAccountId": "[concat(resourceGroup().id, '/providers/Microsoft.Storage/storageAccounts/', parameters('AzureBlobStorage_accountName'))]"

2
docs/samples/deployments/data-factory/arm-templates/azure-deploy-adf-template-gallery-http.json
View File

@@ -68,7 +68,7 @@
"properties": { "properties": {
"mode": "Incremental", "mode": "Incremental",
"templateLink": { "templateLink": {
"uri": "https://raw.githubusercontent.com/microsoft/presidio/main/docs/samples/deployments/app-service/presidio-services.json", "uri": "https://raw.githubusercontent.com/data-privacy-stack/presidio/main/docs/samples/deployments/app-service/presidio-services.json",
"contentVersion": "1.0.0.0" "contentVersion": "1.0.0.0"
}, },
"parameters": { "parameters": {

2
docs/samples/deployments/data-factory/presidio-data-factory-template-gallery-http.md
View File

@@ -46,7 +46,7 @@ To use this template you should first setup the required infrastructure for the
Create the Azure App Service, the storage accounts and an Azure Key Vault by clicking the Deploy-to-Azure button, or by running the following script to provision the [provided ARM template](./arm-templates/azure-deploy-adf-template-gallery-http.json). Create the Azure App Service, the storage accounts and an Azure Key Vault by clicking the Deploy-to-Azure button, or by running the following script to provision the [provided ARM template](./arm-templates/azure-deploy-adf-template-gallery-http.json).
[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fdata-factory%2Farm-templates%2Fazure-deploy-adf-template-gallery-http.json) [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdata-privacy-stack%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fdata-factory%2Farm-templates%2Fazure-deploy-adf-template-gallery-http.json)
```bash ```bash
RESOURCE_GROUP=[Name of resource group] RESOURCE_GROUP=[Name of resource group]

4
docs/samples/deployments/data-factory/presidio-data-factory.md
View File

@@ -11,7 +11,7 @@ The samples deploy and use the following Azure Services:
* Azure Storage - Persistence layer of this sample. * Azure Storage - Persistence layer of this sample.
* Azure Databricks/ Azure App Service - Host presidio to anonymize the data. * Azure Databricks/ Azure App Service - Host presidio to anonymize the data.
The input file used by the samples is hosted on [presidio-research](https://github.com/microsoft/presidio-research/) repository. It is setup as a variable on the provided ARM template and used by Azure Data Factory as the input source. The input file used by the samples is hosted on [presidio-research](https://github.com/data-privacy-stack/presidio-research/) repository. It is setup as a variable on the provided ARM template and used by Azure Data Factory as the input source.
## Option 1: Presidio as an HTTP REST endpoint ## Option 1: Presidio as an HTTP REST endpoint
@@ -23,7 +23,7 @@ By using Presidio as an HTTP endpoint, the user can select which infrastructure
Create the Azure App Service and the ADF pipeline by clicking the Deploy-to-Azure button, or by running the following script to provision the [provided ARM template](./arm-templates/azure-deploy-adf-app-service.json). Create the Azure App Service and the ADF pipeline by clicking the Deploy-to-Azure button, or by running the following script to provision the [provided ARM template](./arm-templates/azure-deploy-adf-app-service.json).
[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fdata-factory%2Farm-templates%2Fazure-deploy-adf-app-service.json) [![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fdata-privacy-stack%2Fpresidio%2Fmain%2Fdocs%2Fsamples%2Fdeployments%2Fdata-factory%2Farm-templates%2Fazure-deploy-adf-app-service.json)
```bash ```bash
RESOURCE_GROUP=[Name of resource group] RESOURCE_GROUP=[Name of resource group]

4
docs/samples/deployments/k8s/charts/presidio/values.yaml
View File

@@ -1,4 +1,4 @@
registry: mcr.microsoft.com registry: ghcr.io/data-privacy-stack
# Image pull secret # Image pull secret
# privateRegistry: acr-auth # privateRegistry: acr-auth
@@ -11,7 +11,7 @@ ingress:
# For any of the services below, a service mesh like Linkerd or Istio # For any of the services below, a service mesh like Linkerd or Istio
# is needed to properly balance traffic to more than 1 replica. # is needed to properly balance traffic to more than 1 replica.
# https://github.com/microsoft/presidio/issues/304 # https://github.com/data-privacy-stack/presidio/issues/304
analyzer: analyzer:
name: presidio-analyzer name: presidio-analyzer

2
docs/samples/deployments/k8s/deployment/deploy-presidio.sh
View File

@@ -1,5 +1,5 @@
#!/bin/bash #!/bin/bash
REGISTRY=${1:-mcr.microsoft.com} REGISTRY=${1:-ghcr.io/data-privacy-stack}
TAG=${2:-latest} TAG=${2:-latest}
RELEASE=${3:-demo} RELEASE=${3:-demo}
helm install $RELEASE --set registry=$REGISTRY,tag=$TAG ../charts/presidio --namespace presidio helm install $RELEASE --set registry=$REGISTRY,tag=$TAG ../charts/presidio --namespace presidio

4
docs/samples/deployments/k8s/index.md
View File

@@ -50,7 +50,7 @@ You can install Presidio locally using [KIND](https://github.com/kubernetes-sigs
``` ```
3. Local [helm](https://helm.sh/) client. 3. Local [helm](https://helm.sh/) client.
4. **Optional** - Container Registry - such as [ACR](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro). Only needed if you are using your own presidio images and not the default ones from from [Microsoft syndicates container catalog](https://azure.microsoft.com/en-in/blog/microsoft-syndicates-container-catalog/) 4. **Optional** - Container Registry - such as [ACR](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-intro). Only needed if you are using your own presidio images and not the default ones from the [GitHub Container Registry](https://github.com/orgs/data-privacy-stack/packages)
5. Recent presidio repo is cloned on your local machine. 5. Recent presidio repo is cloned on your local machine.
### Step by step deployment with customizable parameters ### Step by step deployment with customizable parameters
@@ -69,7 +69,7 @@ You can install Presidio locally using [KIND](https://github.com/kubernetes-sigs
# Choose a namespace and ensure it is created # Choose a namespace and ensure it is created
NAMESPACE=presidio NAMESPACE=presidio
# Choose the tag, from mcr.microsoft.com, e.g. `latest` # Choose the tag, from ghcr.io/data-privacy-stack, e.g. `latest`
TAG=latest TAG=latest
# Choose a name for the deployment # Choose a name for the deployment

2
docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/docs/sample_for_presidio_pr/anonymization_toolkit_sample.md
View File

@@ -61,7 +61,7 @@ The following code can exist as part of a service available to the appropriate a
self.deanonymizer.add_deanonymizer(InstanceCounterDeanonymizer) self.deanonymizer.add_deanonymizer(InstanceCounterDeanonymizer)
``` ```
`InstanceCounterAnonymizer` and `InstanceCounterDeanonymizer` taken from [these examples](https://microsoft.github.io/presidio/samples/python/pseudonomyzation/). `InstanceCounterAnonymizer` and `InstanceCounterDeanonymizer` taken from [these examples](https://data-privacy-stack.github.io/presidio/samples/python/pseudonymization/).
We anonymize data with the following: We anonymize data with the following:

2
docs/samples/deployments/openai-anonymaztion-and-deanonymaztion-best-practices/index.md
View File

@@ -81,7 +81,7 @@ The following code can exist as part of a service available to the appropriate a
self.deanonymizer.add_deanonymizer(InstanceCounterDeanonymizer) self.deanonymizer.add_deanonymizer(InstanceCounterDeanonymizer)
``` ```
`InstanceCounterAnonymizer` and `InstanceCounterDeanonymizer` taken from [these examples](https://microsoft.github.io/presidio/samples/python/pseudonomyzation/). `InstanceCounterAnonymizer` and `InstanceCounterDeanonymizer` taken from [these examples](https://data-privacy-stack.github.io/presidio/samples/python/pseudonymization/).
We anonymize data with the following: We anonymize data with the following:

4
docs/samples/deployments/redacting-telemetry/docker-compose.yml
View File

@@ -1,7 +1,7 @@
services: services:
# Presidio Analyzer - detects PII in text # Presidio Analyzer - detects PII in text
presidio-analyzer: presidio-analyzer:
image: mcr.microsoft.com/presidio-analyzer:latest image: ghcr.io/data-privacy-stack/presidio-analyzer:latest
ports: ports:
- "5002:3000" - "5002:3000"
networks: networks:
@@ -9,7 +9,7 @@ services:
# Presidio Anonymizer - masks/redacts PII # Presidio Anonymizer - masks/redacts PII
presidio-anonymizer: presidio-anonymizer:
image: mcr.microsoft.com/presidio-anonymizer:latest image: ghcr.io/data-privacy-stack/presidio-anonymizer:latest
ports: ports:
- "5001:3000" - "5001:3000"
networks: networks:

2
docs/samples/deployments/redacting-telemetry/index.md
View File

@@ -1,6 +1,6 @@
# Redacting Telemetry with Presidio # Redacting Telemetry with Presidio
**A practical example of redacting PII in logs using Microsoft Presidio.** **A practical example of redacting PII in logs using Presidio.**
> ⚠️ This is a proof-of-concept demonstration, not a production-ready solution. > ⚠️ This is a proof-of-concept demonstration, not a production-ready solution.

2
docs/samples/docker/index.md
View File

@@ -2,7 +2,7 @@
## Description ## Description
Presidio can expose REST endpoints for each service using Flask and Docker. Follow the [installation guide](https://github.com/microsoft/presidio/blob/V2/docs/installation.md#using-docker) to learn how to install and run presidio-analyzer and presidio-anonymizer using docker. Presidio can expose REST endpoints for each service using Flask and Docker. Follow the [installation guide](https://github.com/data-privacy-stack/presidio/blob/V2/docs/installation.md#using-docker) to learn how to install and run presidio-analyzer and presidio-anonymizer using docker.
## Postman collection ## Postman collection

4
docs/samples/docker/litellm.md
View File

@@ -6,9 +6,9 @@ Run Presidio PII Masking across Anthropic/Gemini/Bedrock/etc. calls with [LiteLL
**Flow:** App <-> `LiteLLM Proxy + Presidio PII Masking` <-> LLM Provider **Flow:** App <-> `LiteLLM Proxy + Presidio PII Masking` <-> LLM Provider
## Pre-Requiesites ## Prerequisites
- Run `pip install 'litellm[proxy]'` [Docs](https://docs.litellm.ai/docs/proxy/quick_start) - Run `pip install 'litellm[proxy]'` [Docs](https://docs.litellm.ai/docs/proxy/quick_start)
- Setup [Presidio Docker](https://microsoft.github.io/presidio/installation/#using-docker) - Setup [Presidio Docker](https://data-privacy-stack.github.io/presidio/installation/#using-docker)
## Quick Start ## Quick Start

6
docs/samples/index.md
View File

@@ -20,9 +20,9 @@
| Usage | Text | Python file | [Azure Health Data Services de-identification Service as a Remote Recognizer](python/ahds/index.md) | | Usage | Text | Python file | [Azure Health Data Services de-identification Service as a Remote Recognizer](python/ahds/index.md) |
| Usage | Text | Python file | [AHDS Surrogate Example](python/ahds/example_ahds_surrogate.py) | | Usage | Text | Python file | [AHDS Surrogate Example](python/ahds/example_ahds_surrogate.py) |
| Usage | Text | Python file | [Language Model-based PII/PHI Detection using LangExtract](python/langextract/index.md) | | Usage | Text | Python file | [Language Model-based PII/PHI Detection using LangExtract](python/langextract/index.md) |
| Usage | CSV | Python file | [Analyze and Anonymize CSV file](https://github.com/microsoft/presidio/blob/main/docs/samples/python/process_csv_file.py) | | Usage | CSV | Python file | [Analyze and Anonymize CSV file](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/process_csv_file.py) |
| Usage | Text | Python | [Using Flair as an external PII model](https://github.com/microsoft/presidio/blob/main/docs/samples/python/flair_recognizer.py)| | Usage | Text | Python | [Using Flair as an external PII model](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py)|
| Usage | Text | Python file | [Using Span Marker as an external PII model](https://github.com/microsoft/presidio/blob/main/docs/samples/python/span_marker_recognizer.py)| | Usage | Text | Python file | [Using Span Marker as an external PII model](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/span_marker_recognizer.py)|
| Usage | Text | Python file | [Using Transformers as an external PII model](python/transformers_recognizer/index.md)| | Usage | Text | Python file | [Using Transformers as an external PII model](python/transformers_recognizer/index.md)|
| Usage | Text | Python file | [Pseudonymization (replace PII values using mappings)](python/pseudonymization.ipynb)| | Usage | Text | Python file | [Pseudonymization (replace PII values using mappings)](python/pseudonymization.ipynb)|
| Usage | Text | Python file | [Passing a lambda as a Presidio anonymizer using Faker](python/example_custom_lambda_anonymizer.py)| | Usage | Text | Python file | [Passing a lambda as a Presidio anonymizer using Faker](python/example_custom_lambda_anonymizer.py)|

8
docs/samples/python/Anonymizing known values.ipynb
View File

@@ -20,7 +20,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/Anonymizing%20known%20values.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/Anonymizing%20known%20values.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/Anonymizing%20known%20values.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/Anonymizing%20known%20values.ipynb)"
] ]
}, },
{ {
@@ -330,9 +330,9 @@
"source": [ "source": [
"Read more:\n", "Read more:\n",
"\n", "\n",
"- For more info on Presidio Analyzer, see [this documentation](https://microsoft.github.io/presidio/analyzer/)\n", "- For more info on Presidio Analyzer, see [this documentation](https://data-privacy-stack.github.io/presidio/analyzer/)\n",
"- For more info on Presidio Anonymize, see [this documentation](https://microsoft.github.io/presidio/anonymizer/)\n", "- For more info on Presidio Anonymize, see [this documentation](https://data-privacy-stack.github.io/presidio/anonymizer/)\n",
"- To further customize the anonymization type, see [this tutorial](https://microsoft.github.io/presidio/tutorial/11_custom_anonymization/)\n" "- To further customize the anonymization type, see [this tutorial](https://data-privacy-stack.github.io/presidio/tutorial/11_custom_anonymization/)\n"
] ]
}, },
{ {

2
docs/samples/python/ahds/index.md
View File

@@ -39,7 +39,7 @@ az login
For more details, see the [AHDS Integration Authentication documentation](../../../ahds_integration.md#authentication). For more details, see the [AHDS Integration Authentication documentation](../../../ahds_integration.md#authentication).
## Azure Health Data Services de-identification Recognizer ## Azure Health Data Services de-identification Recognizer
[The implementation of a `AzureHealthDeid` recognizer can be found here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/ahds_recognizer.py). [The implementation of a `AzureHealthDeid` recognizer can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/ahds_recognizer.py).
## How to integrate Azure Health Data Services de-identification into Presidio ## How to integrate Azure Health Data Services de-identification into Presidio

2
docs/samples/python/batch_processing.ipynb
View File

@@ -20,7 +20,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/batch_processing.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/batch_processing.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/batch_processing.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/batch_processing.ipynb)"
] ]
}, },
{ {

24
docs/samples/python/customizing_presidio_analyzer.ipynb
View File

@@ -4,14 +4,14 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/customizing_presidio_analyzer.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/customizing_presidio_analyzer.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/customizing_presidio_analyzer.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/customizing_presidio_analyzer.ipynb)"
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# Customizing the PII analysis process in Microsoft Presidio\n", "# Customizing the PII analysis process in Presidio\n",
"\n", "\n",
"This notebooks covers different customization use cases to:\n", "This notebooks covers different customization use cases to:\n",
"\n", "\n",
@@ -25,7 +25,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Installation\n", "## Installation\n",
"First, let's install presidio using `pip`. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation).\n", "First, let's install presidio using `pip`. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation).\n",
"\n", "\n",
"Install from PyPI:" "Install from PyPI:"
] ]
@@ -50,9 +50,9 @@
"## Getting started\n", "## Getting started\n",
"\n", "\n",
"The high level process in Presidio-Analyzer is the following:\n", "The high level process in Presidio-Analyzer is the following:\n",
"![image.png](https://github.com/microsoft/presidio/raw/main/docs/assets/analyzer-design.png)\n", "![image.png](https://github.com/data-privacy-stack/presidio/raw/main/docs/assets/analyzer-design.png)\n",
"\n", "\n",
"Load the `presidio-analyzer` modules. For more information, see the [analyzer docs](https://microsoft.github.io/presidio/analyzer/).\n" "Load the `presidio-analyzer` modules. For more information, see the [analyzer docs](https://data-privacy-stack.github.io/presidio/analyzer/).\n"
] ]
}, },
{ {
@@ -312,7 +312,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"It's important to mention that recognizers is likely to have errors, both false-positive and false-negative, which would impact the entire performance of Presidio. Consider testing each recognizer on a representative dataset prior to integrating it into Presidio. For more info, see the [best practices for developing recognizers documentation](https://microsoft.github.io/presidio/analyzer/developing_recognizers/)." "It's important to mention that recognizers is likely to have errors, both false-positive and false-negative, which would impact the entire performance of Presidio. Consider testing each recognizer on a representative dataset prior to integrating it into Presidio. For more info, see the [best practices for developing recognizers documentation](https://data-privacy-stack.github.io/presidio/analyzer/developing_recognizers/)."
] ]
}, },
{ {
@@ -326,7 +326,7 @@
"\n", "\n",
"Notes:\n", "Notes:\n",
"\n", "\n",
"- In this example we would create a new class, which implements [`EntityRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py), the basic recognizer in Presidio. This abstract class requires us to implement the `load` method and `analyze` method. \n", "- In this example we would create a new class, which implements [`EntityRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py), the basic recognizer in Presidio. This abstract class requires us to implement the `load` method and `analyze` method. \n",
"\n", "\n",
"- Each recognizer accepts an object of type `NlpArtifacts`, which holds pre-computed attributes on the input text.\n", "- Each recognizer accepts an object of type `NlpArtifacts`, which holds pre-computed attributes on the input text.\n",
"\n", "\n",
@@ -456,9 +456,9 @@
"### Example 4: Calling an external service for PII detection\n", "### Example 4: Calling an external service for PII detection\n",
"\n", "\n",
"In a similar way to example 3, we can write logic to call external services for PII detection. \n", "In a similar way to example 3, we can write logic to call external services for PII detection. \n",
"For a detailed example, see [this part of the documentation](https://microsoft.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer).\n", "For a detailed example, see [this part of the documentation](https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer).\n",
"\n", "\n",
"[This is a sample implementation of such remote recognizer](https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_remote_recognizer.py).\n", "[This is a sample implementation of such remote recognizer](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py).\n",
"\n" "\n"
] ]
}, },
@@ -481,7 +481,7 @@
"\n", "\n",
"#### Adapting the NLP engine\n", "#### Adapting the NLP engine\n",
"\n", "\n",
"As its internal NLP engine, Presidio supports both spaCy and Stanza. Make sure you download the required models from spacy/stanza prior to using them. More details [here](https://microsoft.github.io/presidio/analyzer/languages/#configuring-the-nlp-engine). For example, to download the Spanish medium spaCy model: `python -m spacy download es_core_news_md`\n", "As its internal NLP engine, Presidio supports both spaCy and Stanza. Make sure you download the required models from spacy/stanza prior to using them. More details [here](https://data-privacy-stack.github.io/presidio/analyzer/languages/#configuring-the-nlp-engine). For example, to download the Spanish medium spaCy model: `python -m spacy download es_core_news_md`\n",
"\n", "\n",
"In this example we will configure Presidio to use spaCy as its underlying NLP framework, with NLP models in English and Spanish:" "In this example we will configure Presidio to use spaCy as its underlying NLP framework, with NLP models in English and Spanish:"
] ]
@@ -540,7 +540,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"- [See this documentation](https://microsoft.github.io/presidio/analyzer/languages/) for more details on how to configure Presidio support additional NLP models and languages.\n", "- [See this documentation](https://data-privacy-stack.github.io/presidio/analyzer/languages/) for more details on how to configure Presidio support additional NLP models and languages.\n",
"- [See this sample](ner_model_configuration.ipynb) for more implemention examples of various NLP engines and NER models." "- [See this sample](ner_model_configuration.ipynb) for more implemention examples of various NLP engines and NER models."
] ]
}, },
@@ -777,7 +777,7 @@
"- Confidence scores before and after each step\n", "- Confidence scores before and after each step\n",
"And more.\n", "And more.\n",
"\n", "\n",
"For more information, refer to the [decision process documentation](https://microsoft.github.io/presidio/analyzer/decision_process/).\n", "For more information, refer to the [decision process documentation](https://data-privacy-stack.github.io/presidio/analyzer/decision_process/).\n",
"\n", "\n",
"Let's use the decision process output to understand how the zip code value was detected:\n" "Let's use the decision process output to understand how the zip code value was detected:\n"
] ]

2
docs/samples/python/encrypt_decrypt.ipynb
View File

@@ -19,7 +19,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/encrypt_decrypt.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/encrypt_decrypt.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/encrypt_decrypt.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/encrypt_decrypt.ipynb)"
] ]
}, },
{ {

4
docs/samples/python/example_dicom_image_redactor.ipynb
View File

@@ -5,7 +5,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_dicom_image_redactor.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_dicom_image_redactor.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_dicom_image_redactor.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_dicom_image_redactor.ipynb)"
] ]
}, },
{ {
@@ -27,7 +27,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Prerequisites\n", "## Prerequisites\n",
"Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation)." "Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation)."
] ]
}, },
{ {

4
docs/samples/python/example_dicom_redactor_evaluation.ipynb
View File

@@ -6,7 +6,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_dicom_redactor_evaluation.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_dicom_redactor_evaluation.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_dicom_redactor_evaluation.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_dicom_redactor_evaluation.ipynb)"
] ]
}, },
{ {
@@ -26,7 +26,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Prerequisites\n", "## Prerequisites\n",
"Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation)." "Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation)."
] ]
}, },
{ {

4
docs/samples/python/example_pdf_annotation.ipynb
View File

@@ -4,7 +4,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_pdf_annotation.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_pdf_annotation.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_pdf_annotation.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_pdf_annotation.ipynb)"
] ]
}, },
{ {
@@ -16,7 +16,7 @@
"This sample takes a PDF as an input, extracts the text, identifies PII using Presidio and annotates the PII using highlight annotations.\n", "This sample takes a PDF as an input, extracts the text, identifies PII using Presidio and annotates the PII using highlight annotations.\n",
"\n", "\n",
"## Prerequisites\n", "## Prerequisites\n",
"Before getting started, make sure the following packages are installed. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation).\n", "Before getting started, make sure the following packages are installed. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation).\n",
"\n", "\n",
"Install from PyPI:" "Install from PyPI:"
] ]

2
docs/samples/python/example_structured.ipynb
View File

@@ -4,7 +4,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_structured.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/example_structured.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_structured.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_structured.ipynb)"
] ]
}, },
{ {

2
docs/samples/python/getting_entity_values.ipynb
View File

@@ -19,7 +19,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/getting_entity_values.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/getting_entity_values.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/getting_entity_values.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/getting_entity_values.ipynb)"
] ]
}, },
{ {

4
docs/samples/python/image_redaction_allow_list_approach.ipynb
View File

@@ -5,7 +5,7 @@
"id": "0fa6b428", "id": "0fa6b428",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/image_redaction_allow_list_approach.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/image_redaction_allow_list_approach.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/image_redaction_allow_list_approach.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/image_redaction_allow_list_approach.ipynb)"
] ]
}, },
{ {
@@ -18,7 +18,7 @@
"> **Note**: Always place the `allow_list` argument last in your redact call as this is considered a text analyzer kwarg.\n", "> **Note**: Always place the `allow_list` argument last in your redact call as this is considered a text analyzer kwarg.\n",
"\n", "\n",
"## Prerequisites\n", "## Prerequisites\n",
"Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation)." "Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation)."
] ]
}, },
{ {

4
docs/samples/python/integrating_with_external_services.ipynb
View File

@@ -19,7 +19,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"####### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/integrating_with_external_services.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/integrating_with_external_services.ipynb)" "####### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/integrating_with_external_services.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/integrating_with_external_services.ipynb)"
] ]
}, },
{ {
@@ -51,7 +51,7 @@
"first be created under an Azure subscription. Follow the [official documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/how-tos/text-analytics-how-to-call-api?tabs=synchronous#create-a-text-analytics-resource)\n", "first be created under an Azure subscription. Follow the [official documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/text-analytics/how-tos/text-analytics-how-to-call-api?tabs=synchronous#create-a-text-analytics-resource)\n",
"for instructions. The key and endpoint, generated once the resource is created, should replace the placeholders `<YOUR_TEXT_ANALYTICS_KEY>` and `<YOUR_TEXT_ANALYTICS_ENDPOINT>` in this notebook, respectively. \n", "for instructions. The key and endpoint, generated once the resource is created, should replace the placeholders `<YOUR_TEXT_ANALYTICS_KEY>` and `<YOUR_TEXT_ANALYTICS_ENDPOINT>` in this notebook, respectively. \n",
"## Text Analytics Recognizer\n", "## Text Analytics Recognizer\n",
"In this example we will use the [`TextAnalyticsRecognizer`](https://github.com/microsoft/presidio/blob/main/docs/samples/python/text_analytics/example_text_analytics_recognizer.py) sample implementation. This class extends Presidio's [Remote Recognizer](https://microsoft.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer) for calling the Text Analytics service REST API. For additional information of a remote recognizer, see the [ExampleRemoteRecognizer](https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_remote_recognizer.py) sample." "In this example we will use the [`TextAnalyticsRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/text_analytics/example_text_analytics_recognizer.py) sample implementation. This class extends Presidio's [Remote Recognizer](https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer) for calling the Text Analytics service REST API. For additional information of a remote recognizer, see the [ExampleRemoteRecognizer](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py) sample."
] ]
}, },
{ {

2
docs/samples/python/keep_entities.ipynb
View File

@@ -19,7 +19,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/keep_entities.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/keep_entities.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/keep_entities.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/keep_entities.ipynb)"
] ]
}, },
{ {

20
docs/samples/python/langextract/index.md
View File

@@ -19,7 +19,7 @@ Unlike pattern-based recognizers, language model-based detection is flexible and
The default configuration includes examples for common PII/PHI entities such as PERSON, EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, MEDICAL_LICENSE, and more. The default configuration includes examples for common PII/PHI entities such as PERSON, EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, CREDIT_CARD, MEDICAL_LICENSE, and more.
**You can customize the prompts and examples to detect any entity types relevant to your use case**. **You can customize the prompts and examples to detect any entity types relevant to your use case**.
For the default entity mappings and examples, see the [default configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_ollama.yaml). For the default entity mappings and examples, see the [default configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_ollama.yaml).
## Supported Language Model Providers ## Supported Language Model Providers
@@ -52,9 +52,9 @@ Presidio provides a hierarchy of recognizers for language model-based PII/PHI de
- **`LMRecognizer`**: Abstract base class for all language model recognizers (LLMs, SLMs, etc.) - **`LMRecognizer`**: Abstract base class for all language model recognizers (LLMs, SLMs, etc.)
- **`LangExtractRecognizer`**: Abstract base class for LangExtract library integration (model-agnostic) - **`LangExtractRecognizer`**: Abstract base class for LangExtract library integration (model-agnostic)
- **`AzureOpenAILangExtractRecognizer`**: Concrete implementation for Azure OpenAI Service - **`AzureOpenAILangExtractRecognizer`**: Concrete implementation for Azure OpenAI Service
- [Implementation](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/azure_openai_langextract_recognizer.py) - [Implementation](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/azure_openai_langextract_recognizer.py)
- **`BasicLangExtractRecognizer`**: Concrete implementation where ModelConfig is configured from YAML (supporting Ollama, OpenAI, Gemini, and other providers) - **`BasicLangExtractRecognizer`**: Concrete implementation where ModelConfig is configured from YAML (supporting Ollama, OpenAI, Gemini, and other providers)
- [Implementation](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/basic_langextract_recognizer.py) - [Implementation](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/basic_langextract_recognizer.py)
--- ---
@@ -107,13 +107,13 @@ You have two options to set up Ollama:
> This option provides better performance with GPU acceleration (e.g., on Mac with Metal Performance Shaders or systems with NVIDIA GPUs). > This option provides better performance with GPU acceleration (e.g., on Mac with Metal Performance Shaders or systems with NVIDIA GPUs).
> The model must be pulled and run before using the recognizer. The default model is `qwen2.5:1.5b`. > The model must be pulled and run before using the recognizer. The default model is `qwen2.5:1.5b`.
3. **Configuration** (optional): Create your own `ollama_config.yaml` or use the [default configuration](https://github.com/microsoft/presidio/blob/main//presidio-analyzer/presidio_analyzer/conf/langextract_config_ollama.yaml) 3. **Configuration** (optional): Create your own `ollama_config.yaml` or use the [default configuration](https://github.com/data-privacy-stack/presidio/blob/main//presidio-analyzer/presidio_analyzer/conf/langextract_config_ollama.yaml)
### Usage ### Usage
**Option 1: Enable in configuration file** **Option 1: Enable in configuration file**
Enable the recognizer in [`default_recognizers.yaml`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml): Enable the recognizer in [`default_recognizers.yaml`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml):
```yaml ```yaml
- name: BasicLangExtractRecognizer - name: BasicLangExtractRecognizer
enabled: true # Change from false to true enabled: true # Change from false to true
@@ -176,7 +176,7 @@ The `langextract_config_ollama.yaml` file supports the following options:
- **`entity_mappings`**: Map LangExtract entity classes to Presidio entity names - **`entity_mappings`**: Map LangExtract entity classes to Presidio entity names
- **`min_score`**: Minimum confidence score (default: `0.5`) - **`min_score`**: Minimum confidence score (default: `0.5`)
See the [configuration file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/ollama_config.yaml) for all options. See the [configuration file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/ollama_config.yaml) for all options.
## Troubleshooting ## Troubleshooting
@@ -215,10 +215,10 @@ Azure OpenAI provides cloud-based access to OpenAI models (GPT-4o, GPT-4, GPT-3.
```sh ```sh
# On macOS/Linux/PowerShell: # On macOS/Linux/PowerShell:
wget https://raw.githubusercontent.com/microsoft/presidio/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml wget https://raw.githubusercontent.com/data-privacy-stack/presidio/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml
# Or download manually from: # Or download manually from:
# https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml # https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml
``` ```
### Authentication Options ### Authentication Options
@@ -347,7 +347,7 @@ For basic usage, just pass `model_id` as a parameter (see examples above).
1. **Download** the default config: 1. **Download** the default config:
```sh ```sh
wget https://raw.githubusercontent.com/microsoft/presidio/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml wget https://raw.githubusercontent.com/data-privacy-stack/presidio/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml
``` ```
2. **Customize** entities, prompts, or other settings in the file 2. **Customize** entities, prompts, or other settings in the file
@@ -382,7 +382,7 @@ The config file contains two main sections:
- `examples_file`: Path to few-shot examples file - `examples_file`: Path to few-shot examples file
- `entity_mappings`: Map LangExtract entity classes to Presidio entity names - `entity_mappings`: Map LangExtract entity classes to Presidio entity names
See the [full config file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml) for details. See the [full config file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/langextract_config_azureopenai.yaml) for details.
--- ---

2
docs/samples/python/ner_model_configuration.ipynb
View File

@@ -16,7 +16,7 @@
"id": "cb8e0bdb-3138-44ad-8d87-d9d549c51ce5", "id": "cb8e0bdb-3138-44ad-8d87-d9d549c51ce5",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/ner_model_configuration.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/ner_model_configuration.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/ner_model_configuration.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/ner_model_configuration.ipynb)"
] ]
}, },
{ {

16
docs/samples/python/no_code_config.ipynb
View File

@@ -21,13 +21,13 @@
"\n", "\n",
"In this example, we'll show how to create a no-code configuration in Presidio.\n", "In this example, we'll show how to create a no-code configuration in Presidio.\n",
"We start by creating YAML configuration files that are based on the default ones. \n", "We start by creating YAML configuration files that are based on the default ones. \n",
"Te default configuration files for Presidio can be found here:\n", "The default configuration files for Presidio can be found here:\n",
"- [Analyzer configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)\n", "- [Analyzer configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)\n",
"- [Recognizer registry configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)\n", "- [Recognizer registry configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)\n",
"- [NLP engine configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)\n", "- [NLP engine configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)\n",
"\n", "\n",
"Alternatively, one can create one configuration file for all three components.\n", "Alternatively, one can create one configuration file for all three components.\n",
"In this example, we'll tweak the configuration to reduce the number of predefinedrecognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish).\n" "In this example, we'll tweak the configuration to reduce the number of predefined recognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish).\n"
] ]
}, },
{ {
@@ -61,7 +61,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"### General Analyzer parameters\n", "### General Analyzer parameters\n",
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml))" "([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml))"
] ]
}, },
{ {
@@ -85,7 +85,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"### Recognizer Registry parameters\n", "### Recognizer Registry parameters\n",
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml))" "([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml))"
] ]
}, },
{ {
@@ -173,7 +173,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"### NLP Engine parameters\n", "### NLP Engine parameters\n",
"([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml))" "([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml))"
] ]
}, },
{ {

4
docs/samples/python/plot_custom_bboxes.ipynb
View File

@@ -5,7 +5,7 @@
"id": "0fa6b428", "id": "0fa6b428",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/plot_custom_bboxes.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/plot_custom_bboxes.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/plot_custom_bboxes.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/plot_custom_bboxes.ipynb)"
] ]
}, },
{ {
@@ -18,7 +18,7 @@
"\n", "\n",
"\n", "\n",
"## Prerequisites\n", "## Prerequisites\n",
"Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation)." "Before getting started, make sure presidio and the latest version of Tesseract OCR are installed. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation)."
] ]
}, },
{ {

2
docs/samples/python/presidio_notebook.ipynb
View File

@@ -17,7 +17,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/presidio_notebook.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/presidio_notebook.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/presidio_notebook.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/presidio_notebook.ipynb)"
] ]
}, },
{ {

2
docs/samples/python/pseudonymization.ipynb
View File

@@ -15,7 +15,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/pseudonomyzation.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/pseudonomyzation.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/pseudonymization.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/pseudonymization.ipynb)"
] ]
}, },
{ {

2
docs/samples/python/streamlit/flair_recognizer.py
View File

@@ -1,4 +1,4 @@
## Taken from https://github.com/microsoft/presidio/blob/main/docs/samples/python/flair_recognizer.py ## Taken from https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py
import logging import logging
from typing import Optional, List, Tuple, Set from typing import Optional, List, Tuple, Set

2
docs/samples/python/streamlit/index.md
View File

@@ -33,4 +33,4 @@ ALLOW_OTHER_MODELS=true #true if the user could download new models
``` ```
## Output ## Output
Output should be similar to this screenshot: Output should be similar to this screenshot:
![image](https://github.com/microsoft/presidio/assets/3776619/7d0eadf1-e750-4747-8b59-8203aa43cac8) ![image](https://github.com/data-privacy-stack/presidio/assets/3776619/7d0eadf1-e750-4747-8b59-8203aa43cac8)

22
docs/samples/python/streamlit/presidio_streamlit.py
View File

@@ -25,7 +25,7 @@ st.set_page_config(
layout="wide", layout="wide",
initial_sidebar_state="expanded", initial_sidebar_state="expanded",
menu_items={ menu_items={
"About": "https://microsoft.github.io/presidio/", "About": "https://data-privacy-stack.github.io/presidio/",
}, },
) )
@@ -39,7 +39,7 @@ allow_other_models = os.getenv("ALLOW_OTHER_MODELS", False)
# Sidebar # Sidebar
st.sidebar.header( st.sidebar.header(
""" """
PII De-Identification with [Microsoft Presidio](https://microsoft.github.io/presidio/) PII De-Identification with [Presidio](https://data-privacy-stack.github.io/presidio/)
""" """
) )
@@ -208,7 +208,7 @@ st_return_decision_process = st.sidebar.checkbox(
"Add analysis explanations to findings", "Add analysis explanations to findings",
value=False, value=False,
help="Add the decision process to the output table. " help="Add the decision process to the output table. "
"More information can be found here: https://microsoft.github.io/presidio/analyzer/decision_process/", "More information can be found here: https://data-privacy-stack.github.io/presidio/analyzer/decision_process/",
) )
# Allow and deny lists # Allow and deny lists
@@ -236,10 +236,10 @@ with st_deny_allow_expander:
with st.expander("About this demo", expanded=False): with st.expander("About this demo", expanded=False):
st.info( st.info(
"""Presidio is an open source customizable framework for PII detection and de-identification. """Presidio is an open source customizable framework for PII detection and de-identification.
\n\n[Code](https://aka.ms/presidio) | \n\n[Code](https://github.com/data-privacy-stack/presidio) |
[Tutorial](https://microsoft.github.io/presidio/tutorial/) | [Tutorial](https://data-privacy-stack.github.io/presidio/tutorial/) |
[Installation](https://microsoft.github.io/presidio/installation/) | [Installation](https://data-privacy-stack.github.io/presidio/installation/) |
[FAQ](https://microsoft.github.io/presidio/faq/) | [FAQ](https://data-privacy-stack.github.io/presidio/faq/) |
[Feedback](https://forms.office.com/r/9ufyYjfDaY) |""" [Feedback](https://forms.office.com/r/9ufyYjfDaY) |"""
) )
@@ -248,11 +248,11 @@ with st.expander("About this demo", expanded=False):
Use this demo to: Use this demo to:
- Experiment with different off-the-shelf models and NLP packages. - Experiment with different off-the-shelf models and NLP packages.
- Explore the different de-identification options, including redaction, masking, encryption and more. - Explore the different de-identification options, including redaction, masking, encryption and more.
- Generate synthetic text with Microsoft Presidio and OpenAI. - Generate synthetic text with Presidio and OpenAI.
- Configure allow and deny lists. - Configure allow and deny lists.
This demo website shows some of Presidio's capabilities. This demo website shows some of Presidio's capabilities.
[Visit our website](https://microsoft.github.io/presidio) for more info, [Visit our website](https://data-privacy-stack.github.io/presidio) for more info,
samples and deployment options. samples and deployment options.
""" """
) )
@@ -260,7 +260,7 @@ with st.expander("About this demo", expanded=False):
st.markdown( st.markdown(
"[![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-analyzer.svg)](https://img.shields.io/pypi/dm/presidio-analyzer.svg)" "[![Pypi Downloads](https://img.shields.io/pypi/dm/presidio-analyzer.svg)](https://img.shields.io/pypi/dm/presidio-analyzer.svg)"
"[![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)" "[![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)"
"![GitHub Repo stars](https://img.shields.io/github/stars/microsoft/presidio?style=social)" "![GitHub Repo stars](https://img.shields.io/github/stars/data-privacy-stack/presidio?style=social)"
) )
analyzer_load_state = st.info("Starting Presidio analyzer...") analyzer_load_state = st.info("Starting Presidio analyzer...")
@@ -289,7 +289,7 @@ try:
default=list(get_supported_entities(*analyzer_params)), default=list(get_supported_entities(*analyzer_params)),
help="Limit the list of PII entities detected. " help="Limit the list of PII entities detected. "
"This list is dynamic and based on the NER model and registered recognizers. " "This list is dynamic and based on the NER model and registered recognizers. "
"More information can be found here: https://microsoft.github.io/presidio/analyzer/adding_recognizers/", "More information can be found here: https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/",
) )
# Before # Before

4
docs/samples/python/synth_data_with_openai.ipynb
View File

@@ -21,7 +21,7 @@
"id": "3345f1c4", "id": "3345f1c4",
"metadata": {}, "metadata": {},
"source": [ "source": [
"###### Path to notebook: [https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/synth_data_with_openai.ipynb](https://www.github.com/microsoft/presidio/blob/main/docs/samples/python/synth_data_with_openai.ipynb)" "###### Path to notebook: [https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/synth_data_with_openai.ipynb](https://www.github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/synth_data_with_openai.ipynb)"
] ]
}, },
{ {
@@ -325,7 +325,7 @@
"\n", "\n",
"templates = []\n", "templates = []\n",
"\n", "\n",
"url = \"https://raw.githubusercontent.com/microsoft/presidio-research/master/presidio_evaluator/data_generator/raw_data/templates.txt\"\n", "url = \"https://raw.githubusercontent.com/data-privacy-stack/presidio-research/master/presidio_evaluator/data_generator/raw_data/templates.txt\"\n",
"for line in urllib.request.urlopen(url):\n", "for line in urllib.request.urlopen(url):\n",
" templates.append(line.decode('utf-8')) " " templates.append(line.decode('utf-8')) "
] ]

2
docs/samples/python/text_analytics/index.md
View File

@@ -22,7 +22,7 @@ for instructions. The key and endpoint, generated once the resource is created,
will be used when integrating with Text Analytics, using a Presidio Text Analytics recognizer. will be used when integrating with Text Analytics, using a Presidio Text Analytics recognizer.
## Azure AI Language Recognizer ## Azure AI Language Recognizer
[The implementation of a `AzureAILanguage` recognizer can be found here](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/azure_ai_language.py). [The implementation of a `AzureAILanguage` recognizer can be found here](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/third_party/azure_ai_language.py).
## How to integrate Azure AI Language into Presidio ## How to integrate Azure AI Language into Presidio

6
docs/samples/python/transformers_recognizer/index.md
View File

@@ -5,7 +5,7 @@
This example demonstrates how to create a **Presidio Recognizer**. This example demonstrates how to create a **Presidio Recognizer**.
To integrate a transformers model as a **Presidio NLP Engine**, see [this documentation](../../../analyzer/nlp_engines/transformers.md). To integrate a transformers model as a **Presidio NLP Engine**, see [this documentation](../../../analyzer/nlp_engines/transformers.md).
We allow these two options, as a user might want to have multiple NER models running in parallel. In this case, one can create multiple `EntityRecognizer` instances, each serving a different model. If you only plan to use one NER model, consider creating a [`TransformersNlpEngine`](../../../analyzer/nlp_engines/transformers.md) instead of the [`TransformersRecognizer`](https://github.com/microsoft/presidio/blob/main/docs/samples/python/transformers_recognizer/transformer_recognizer.py) described in this document. We allow these two options, as a user might want to have multiple NER models running in parallel. In this case, one can create multiple `EntityRecognizer` instances, each serving a different model. If you only plan to use one NER model, consider creating a [`TransformersNlpEngine`](../../../analyzer/nlp_engines/transformers.md) instead of the [`TransformersRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/transformers_recognizer/transformer_recognizer.py) described in this document.
When initializing the `TransformersRecognizer`, choose from the following options: When initializing the `TransformersRecognizer`, choose from the following options:
@@ -16,14 +16,14 @@ When initializing the `TransformersRecognizer`, choose from the following option
3. Provide the path to your own local custom trained model. 3. Provide the path to your own local custom trained model.
!!! note "Note" !!! note "Note"
For each combination of model & dataset, it is recommended to create a configuration object which includes setting necessary parameters for getting the correct results. Please reference this [configuration.py](https://github.com/microsoft/presidio/blob/main/configuration.py) file for examples. For each combination of model & dataset, it is recommended to create a configuration object which includes setting necessary parameters for getting the correct results. Please reference this [configuration.py](https://github.com/data-privacy-stack/presidio/blob/main/configuration.py) file for examples.
## Example Code ## Example Code
This example code uses a `TransformersRecognizer` for NER, and removes the default `SpacyRecognizer`. This example code uses a `TransformersRecognizer` for NER, and removes the default `SpacyRecognizer`.
In order to be able to use spaCy features such as lemmas, we introduce the small (and faster) `en_core_web_sm` model. In order to be able to use spaCy features such as lemmas, we introduce the small (and faster) `en_core_web_sm` model.
[link to full TransformersRecognizer code](https://github.com/microsoft/presidio/blob/main/docs/samples/python/transformers_recognizer/transformer_recognizer.py) [link to full TransformersRecognizer code](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/transformers_recognizer/transformer_recognizer.py)
```python ```python
from presidio_analyzer import AnalyzerEngine, RecognizerRegistry from presidio_analyzer import AnalyzerEngine, RecognizerRegistry

8
docs/structured/index.md
View File

@@ -108,7 +108,7 @@ print(anonymized_complex_json)
A more detailed sample can be found here: A more detailed sample can be found here:
- <https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_structured.ipynb> - <https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_structured.ipynb>
#### Selection Strategy for Entity Detection in Tabular Data #### Selection Strategy for Entity Detection in Tabular Data
@@ -138,11 +138,11 @@ tabular_analysis = PandasAnalysisBuilder().generate_analysis(sample_df, selectio
- PySpark implementation - PySpark implementation
- Integration of additional anonymization techniques such as K-Anonymity and Differential Privacy. - Integration of additional anonymization techniques such as K-Anonymity and Differential Privacy.
Contributions are welcome! Please refer to the [Contributing Guide](https://github.com/microsoft/presidio/blob/main/CONTRIBUTING.md). Contributions are welcome! Please refer to the [Contributing Guide](https://github.com/data-privacy-stack/presidio/blob/main/CONTRIBUTING.md).
#### More information #### More information
- [API documentation](../api/structured_python.md) - [API documentation](../api/structured_python.md)
- [Sample code](../samples/python/example_structured.ipynb) - [Sample code](../samples/python/example_structured.ipynb)
- [Join the discussion](https://github.com/microsoft/presidio/discussions?discussions_q=structured) - [Join the discussion](https://github.com/data-privacy-stack/presidio/discussions?discussions_q=structured)
- [Relevant issues on Github](https://github.com/microsoft/presidio/issues?q=is%3Aissue+label%3Astructured-data) - [Relevant issues on GitHub](https://github.com/data-privacy-stack/presidio/issues?q=is%3Aissue+label%3Astructured-data)

2
docs/tutorial/00_getting_started.md
View File

@@ -2,7 +2,7 @@
## Installation ## Installation
First, let's install presidio using `pip`. For detailed documentation, see the [installation docs](https://microsoft.github.io/presidio/installation). First, let's install presidio using `pip`. For detailed documentation, see the [installation docs](https://data-privacy-stack.github.io/presidio/installation).
Install from PyPI: Install from PyPI:
```sh ```sh

2
docs/tutorial/02_regex.md
View File

@@ -28,4 +28,4 @@ print("Result:")
print(numbers_result) print(numbers_result)
``` ```
It's important to mention that recognizers are likely to have errors, both false-positive and false-negative, which would impact the entire performance of Presidio. Consider testing each recognizer on a representative dataset prior to integrating it into Presidio. For more info, see the [best practices for developing recognizers documentation](https://microsoft.github.io/presidio/analyzer/developing_recognizers/). It's important to mention that recognizers are likely to have errors, both false-positive and false-negative, which would impact the entire performance of Presidio. Consider testing each recognizer on a representative dataset prior to integrating it into Presidio. For more info, see the [best practices for developing recognizers documentation](https://data-privacy-stack.github.io/presidio/analyzer/developing_recognizers/).

2
docs/tutorial/03_rule_based.md
View File

@@ -4,7 +4,7 @@ Taking the numbers recognizer one step further, let's say we also would like to
Notes: Notes:
- In this example we would create a new class, which implements [`EntityRecognizer`](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py), the basic recognizer in Presidio. This abstract class requires us to implement the `load` method and `analyze` method. - In this example we would create a new class, which implements [`EntityRecognizer`](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/entity_recognizer.py), the basic recognizer in Presidio. This abstract class requires us to implement the `load` method and `analyze` method.
- Each recognizer accepts an object of type `NlpArtifacts`, which holds pre-computed attributes on the input text. - Each recognizer accepts an object of type `NlpArtifacts`, which holds pre-computed attributes on the input text.

6
docs/tutorial/04_external_services.md
View File

@@ -7,11 +7,11 @@ In a similar way to example 3, we can write logic to call external services for
## Calling a remote service ## Calling a remote service
1. [Documentation on remote recognizers](https://microsoft.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer). 1. [Documentation on remote recognizers](https://data-privacy-stack.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer).
2. [A sample implementation of a remote recognizer](https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_remote_recognizer.py). 2. [A sample implementation of a remote recognizer](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py).
## Calling a model in a different framework ## Calling a model in a different framework
- [This example](https://github.com/microsoft/presidio/blob/main/docs/samples/python/flair_recognizer.py) shows a Presidio wrapper for a Flair model. - [This example](https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py) shows a Presidio wrapper for a Flair model.
- Using a similar approach, we could create wrappers for HuggingFace models, Conditional Random Fields or any other framework. - Using a similar approach, we could create wrappers for HuggingFace models, Conditional Random Fields or any other framework.

4
docs/tutorial/05_languages.md
View File

@@ -7,7 +7,7 @@ Two main parts in Presidio handle the text, and should be adapted if a new langu
## Adapting the NLP engine ## Adapting the NLP engine
As its internal NLP engine, Presidio supports both spaCy and Stanza. Make sure you download the required models from spacy/stanza prior to using them. More details [here](https://microsoft.github.io/presidio/analyzer/languages/#configuring-the-nlp-engine). For example, to download the Spanish medium spaCy model: `python -m spacy download es_core_news_md` As its internal NLP engine, Presidio supports both spaCy and Stanza. Make sure you download the required models from spacy/stanza prior to using them. More details [here](https://data-privacy-stack.github.io/presidio/analyzer/languages/#configuring-the-nlp-engine). For example, to download the Spanish medium spaCy model: `python -m spacy download es_core_news_md`
In this example we will configure Presidio to use spaCy as its underlying NLP framework, with NLP models in English and Spanish: In this example we will configure Presidio to use spaCy as its underlying NLP framework, with NLP models in English and Spanish:
@@ -47,7 +47,7 @@ print("Results from English request:")
print(results_english) print(results_english)
``` ```
[See this documentation](https://microsoft.github.io/presidio/analyzer/languages/) for more details on setting up additional NLP models and languages. [See this documentation](https://data-privacy-stack.github.io/presidio/analyzer/languages/) for more details on setting up additional NLP models and languages.
## Using external models/frameworks ## Using external models/frameworks

2
docs/tutorial/07_decision_process.md
View File

@@ -9,7 +9,7 @@ Presidio-analyzer's decision process exposes information on why a specific PII w
- Confidence scores before and after each step - Confidence scores before and after each step
And more. And more.
For more information, refer to the [decision process documentation](https://microsoft.github.io/presidio/analyzer/decision_process/). For more information, refer to the [decision process documentation](https://data-privacy-stack.github.io/presidio/analyzer/decision_process/).
Let's use the decision process output to understand how the zip code value was detected: Let's use the decision process output to understand how the zip code value was detected:

14
docs/tutorial/08_no_code.md
View File

@@ -10,12 +10,12 @@ In this example, we'll show how to create a no-code configuration in Presidio.
We start by creating YAML configuration files that are based on the default ones. We start by creating YAML configuration files that are based on the default ones.
The default configuration files for Presidio can be found here: The default configuration files for Presidio can be found here:
- [Analyzer configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml) - [Analyzer configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)
- [Recognizer registry configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml) - [Recognizer registry configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)
- [NLP engine configuration](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml) - [NLP engine configuration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)
Alternatively, one can create one configuration file for all three components. Alternatively, one can create one configuration file for all three components.
In this example, we'll tweak the configuration to reduce the number of predefinedrecognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish). In this example, we'll tweak the configuration to reduce the number of predefined recognizers to only a few, and add a new custom one. We'll also adjust the context words to support the detection of a different language (Spanish).
```python ```python
import yaml import yaml
@@ -31,7 +31,7 @@ In this example we're going to create the yaml as a string for illustration purp
### General Analyzer parameters ### General Analyzer parameters
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml)) ([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_analyzer.yaml))
```python ```python
analyzer_config_yaml = """ analyzer_config_yaml = """
@@ -44,7 +44,7 @@ default_score_threshold: 0.4
### Recognizer Registry parameters ### Recognizer Registry parameters
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml)) ([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default_recognizers.yaml))
```python ```python
@@ -121,7 +121,7 @@ recognizer_registry:
### NLP Engine parameters ### NLP Engine parameters
([default file](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml)) ([default file](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/conf/default.yaml))
```python ```python
nlp_engine_yaml = """ nlp_engine_yaml = """

4
docs/tutorial/index.md
View File

@@ -1,6 +1,6 @@
# Tutorial: Customization in Microsoft Presidio # Tutorial: Customization in Presidio
This tutorials covers different customization use cases to: This tutorial covers different customization use cases to:
1. Adapt Presidio to detect new types of PII entities. 1. Adapt Presidio to detect new types of PII entities.
2. Adapt Presidio to detect PII entities in a new language. 2. Adapt Presidio to detect PII entities in a new language.

26
mkdocs.yml
View File

@@ -1,9 +1,9 @@
site_name: Presidio site_name: Presidio
site_url: https://microsoft.github.io/presidio site_url: https://data-privacy-stack.github.io/presidio
site_description: PII anonymization for text, images, and structured data. site_description: PII anonymization for text, images, and structured data.
site_author: Data Privacy Stack site_author: Data Privacy Stack
repo_url: https://github.com/microsoft/presidio/ repo_url: https://github.com/data-privacy-stack/presidio/
edit_uri: "" edit_uri: ""
nav: nav:
@@ -66,7 +66,7 @@ nav:
- Supported entities: supported_entities.md - Supported entities: supported_entities.md
- Project transition update: project_transition.md - Project transition update: project_transition.md
- Community: community.md - Community: community.md
- Change log: https://github.com/microsoft/presidio/blob/main/CHANGELOG.md - Change log: https://github.com/data-privacy-stack/presidio/blob/main/CHANGELOG.md
- Setting up a development environment: development.md - Setting up a development environment: development.md
- Build and release process: build_release.md - Build and release process: build_release.md
- Changes from V1 to V2: presidio_V2.md - Changes from V1 to V2: presidio_V2.md
@@ -76,7 +76,7 @@ nav:
- Presidio Anonymizer Python API: api/anonymizer_python.md - Presidio Anonymizer Python API: api/anonymizer_python.md
- Presidio Image Redactor Python API: api/image_redactor_python.md - Presidio Image Redactor Python API: api/image_redactor_python.md
- Presidio Structured Python API: api/structured_python.md - Presidio Structured Python API: api/structured_python.md
- REST API reference: https://microsoft.github.io/presidio/api-docs/api-docs.html" target="_blank - REST API reference: https://data-privacy-stack.github.io/presidio/api-docs/api-docs.html" target="_blank
- Recipes: - Recipes:
- Home: recipes/index.md - Home: recipes/index.md
- Contributing: recipes/CONTRIBUTING.md - Contributing: recipes/CONTRIBUTING.md
@@ -93,21 +93,21 @@ nav:
- Anonymizing known values: samples/python/Anonymizing known values.ipynb - Anonymizing known values: samples/python/Anonymizing known values.ipynb
- Keeping some entities from being anonymized: samples/python/keep_entities.ipynb - Keeping some entities from being anonymized: samples/python/keep_entities.ipynb
- Integrating with external services: samples/python/integrating_with_external_services.ipynb - Integrating with external services: samples/python/integrating_with_external_services.ipynb
- Remote Recognizer: https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_remote_recognizer.py - Remote Recognizer: https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_remote_recognizer.py
- Azure AI Language as a Remote Recognizer: samples/python/text_analytics/index.md - Azure AI Language as a Remote Recognizer: samples/python/text_analytics/index.md
- Azure Health DataServices De-identification as a Remote Recognizer: samples/python/ahds/index.md - Azure Health DataServices De-identification as a Remote Recognizer: samples/python/ahds/index.md
- Using Flair as an external PII model: https://github.com/microsoft/presidio/blob/main/docs/samples/python/flair_recognizer.py - Using Flair as an external PII model: https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/flair_recognizer.py
- Using Span Marker as an external PII model: https://github.com/microsoft/presidio/blob/main/docs/samples/python/span_marker_recognizer.py - Using Span Marker as an external PII model: https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/span_marker_recognizer.py
- Using Transformers as an external PII model: samples/python/transformers_recognizer/index.md - Using Transformers as an external PII model: samples/python/transformers_recognizer/index.md
- Using GLiNER as an external PII model: samples/python/gliner.md - Using GLiNER as an external PII model: samples/python/gliner.md
- Pseudonymization (replace PII values using mappings): samples/python/pseudonymization.ipynb - Pseudonymization (replace PII values using mappings): samples/python/pseudonymization.ipynb
- Passing a lambda as a Presidio anonymizer using Faker: https://github.com/microsoft/presidio/blob/main/docs/samples/python/example_custom_lambda_anonymizer.py - Passing a lambda as a Presidio anonymizer using Faker: https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/example_custom_lambda_anonymizer.py
- Synthetic data generation with OpenAI: samples/python/synth_data_with_openai.ipynb - Synthetic data generation with OpenAI: samples/python/synth_data_with_openai.ipynb
- YAML based no-code configuration: samples/python/no_code_config.ipynb - YAML based no-code configuration: samples/python/no_code_config.ipynb
- Data: - Data:
- Analyzing structured / semi-structured data in batch: samples/python/batch_processing.ipynb - Analyzing structured / semi-structured data in batch: samples/python/batch_processing.ipynb
- Presidio Structured Basic Usage Notebook: samples/python/example_structured.ipynb - Presidio Structured Basic Usage Notebook: samples/python/example_structured.ipynb
- Analyze and Anonymize CSV file: https://github.com/microsoft/presidio/blob/main/docs/samples/python/process_csv_file.py - Analyze and Anonymize CSV file: https://github.com/data-privacy-stack/presidio/blob/main/docs/samples/python/process_csv_file.py
- Images: - Images:
- Redacting Text PII from DICOM images: samples/python/example_dicom_image_redactor.ipynb - Redacting Text PII from DICOM images: samples/python/example_dicom_image_redactor.ipynb
- Using an allow list with image redaction: samples/python/image_redaction_allow_list_approach.ipynb - Using an allow list with image redaction: samples/python/image_redaction_allow_list_approach.ipynb
@@ -225,11 +225,13 @@ plugins:
extra: extra:
social: social:
- icon: fontawesome/brands/github - icon: fontawesome/brands/github
link: https://github.com/microsoft/presidio link: https://github.com/data-privacy-stack/presidio
- icon: fontawesome/brands/discord
link: https://discord.gg/MewtBGFquB
- icon: fontawesome/brands/docker - icon: fontawesome/brands/docker
link: https://hub.docker.com/_/microsoft-presidio link: https://github.com/orgs/data-privacy-stack/packages
- icon: fontawesome/solid/envelope - icon: fontawesome/solid/envelope
link: mailto:presidio@microsoft.com link: mailto:presidio@dataprivacystack.org
- icon: fontawesome/solid/meteor - icon: fontawesome/solid/meteor
link: https://huggingface.co/spaces/presidio/presidio_demo link: https://huggingface.co/spaces/presidio/presidio_demo