upstream-archive/presidio
1
0
Fork 0
mirror of https://github.com/data-privacy-stack/presidio.git synced 2026-09-21 13:38:05 -05:00
Code Issues Packages Projects Releases Wiki Activity

Integrating spacy-huggingface-pipelines and refactoring NlpEngine logic (#1159)

Browse Source
This commit is contained in:
Omri Mendels
2023-10-19 13:58:04 +03:00
committed by GitHub
parent 818c80f978
commit 3a7b8f67f5
43 changed files with 1522 additions and 476 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.pipelines/templates/build-analyzer.yml
+1
View File
@@ -18,6 +18,7 @@ steps:
set -eux # fail on error
pipenv install --deploy --dev
pipenv run python -m spacy download en_core_web_lg
pipenv run python -m spacy download en_core_web_sm
- template: ./build-python.yml
parameters:
NOTICE
+233 -1
View File
@@ -3,6 +3,238 @@ Do Not Translate or Localize
This project incorporates components from the projects listed below. The original copyright notices and the licenses under which Microsoft received such components are set forth below. Microsoft reserves all rights not expressly granted herein, whether by implication, estoppel or otherwise.
*******
transformers
Copyright 2018- The Hugging Face team. All rights reserved.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*******
stanza
Copyright 2019 The Board of Trustees of The Leland Stanford Junior University
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*******
spacy-huggingface-pipelines
MIT License
Copyright (c) 2023 ExplosionAI GmbH
*******
azure-ai-formrecognizer
@@ -20,7 +252,7 @@ furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
docs/analyzer/languages-config.yml
+21 -1
View File
@@ -3,6 +3,26 @@ models:
-
lang_code: en
model_name: en_core_web_lg
-
lang_code: de
model_name: de_core_news_md
-
lang_code: es
model_name: es_core_news_md
model_name: es_core_news_md
ner_model_configuration:
- model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
LOC: LOCATION
LOCATION: LOCATION
GPE: LOCATION
ORG: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
NORP: NRP
- low_confidence_score_multiplier: 0.4
- low_score_entity_names:
- ORGANIZATION
- ORG
- default_score: 0.85
mkdocs.yml
+36 -28
View File
@@ -8,6 +8,7 @@ edit_uri: ""
nav:
- Home: index.md
- Installation: installation.md
- Quickstart: getting_started.md
- Step by step tutorial:
- Home: tutorial/index.md
@@ -25,33 +26,38 @@ nav:
- Custom anonymization: tutorial/11_custom_anonymization.md
- Encryption/Decryption: tutorial/12_encryption.md
- Allow-lists: tutorial/13_allow_list.md
- Docs:
- Installation: installation.md
- Handling text:
- Home: text_anonymization.md
- Presidio Analyzer:
- Home: analyzer/index.md
- Developing PII recognizers:
- Tutorial : analyzer/adding_recognizers.md
- Best practices in developing recognizers : analyzer/developing_recognizers.md
- Multi-language support: analyzer/languages.md
- Customizing the NLP model: analyzer/customizing_nlp_models.md
- Tracing the decision process: analyzer/decision_process.md
- Presidio Anonymizer:
- Home: anonymizer/index.md
- Developing PII operators: anonymizer/adding_operators.md
- Handling images: image-redactor/index.md
- Supported entities: supported_entities.md
- Development and design:
- Design: design.md
- Setting up a development environment: development.md
- Build and release process: build_release.md
- Changes from V1 to V2: presidio_V2.md
- Python API reference:
- Presidio Analyzer Python API: api/analyzer_python.md
- Presidio Anonymizer Python API: api/anonymizer_python.md
- Presidio Image Redactor Python API: api/image_redactor_python.md
- REST API reference: https://microsoft.github.io/presidio/api-docs/api-docs.html" target="_blank
- Handling text:
- Home: text_anonymization.md
- Presidio Analyzer:
- Home: analyzer/index.md
- Developing PII recognizers:
- Tutorial : analyzer/adding_recognizers.md
- Best practices in developing recognizers : analyzer/developing_recognizers.md
- Multi-language support: analyzer/languages.md
- Customizing the NLP model:
- Home: analyzer/customizing_nlp_models.md
- Spacy/Stanza: analyzer/nlp_engines/spacy_stanza.md
- Transformers: analyzer/nlp_engines/transformers.md
- Tracing the decision process: analyzer/decision_process.md
- Presidio Anonymizer:
- Home: anonymizer/index.md
- Developing PII operators: anonymizer/adding_operators.md
- Handling images:
- Home: image-redactor/index.md
- Evaluating DICOM redaction: image-redactor/evaluating_dicom_redaction.md
- Supported entities: supported_entities.md
- Development and design:
- Design: design.md
- Setting up a development environment: development.md
- Build and release process: build_release.md
- Changes from V1 to V2: presidio_V2.md
- Python API reference:
- Home: api.md
- Presidio Analyzer Python API: api/analyzer_python.md
- Presidio Anonymizer Python API: api/anonymizer_python.md
- Presidio Image Redactor Python API: api/image_redactor_python.md
- REST API reference: https://microsoft.github.io/presidio/api-docs/api-docs.html" target="_blank
- Samples: samples/index.md
- Community: community.md
- FAQ: faq.md
@@ -72,6 +78,7 @@ theme:
favicon: assets/ms_icon.png
features:
- navigation.instant
- content.tabs.link
# - navigation.tabs
# - navigation.tabs.sticky
plugins:
@@ -102,4 +109,5 @@ markdown_extensions:
- pymdownx.highlight
- pymdownx.superfences
- pymdownx.pathconverter
- pymdownx.tabbed
- pymdownx.tabbed:
alternate_style: true
presidio-analyzer/Pipfile
+5 -1
View File
@@ -4,16 +4,20 @@ verify_ssl = true
name = "pypi"
[packages]
spacy = ">=3.4.4,<4.0.0"
spacy = ">=3.4.4, <4.0.0"
regex = "*"
tldextract = "*"
flask = ">=1.1"
pyyaml = "*"
phonenumbers = ">=8.12,<9.0.0"
typing-extensions = "*"
spacy-huggingface-pipelines = "*"
# stanza = "*"
# spacy-stanza = "*"
[dev-packages]
pytest = "*"
pytest-mock = "*"
flake8= {version = ">=3.7.9"}
pep8-naming = "*"
flake8-docstrings = "*"
presidio-analyzer/conf/default.yaml
+30
View File
@@ -3,3 +3,33 @@ models:
-
lang_code: en
model_name: en_core_web_lg
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
GPE: LOCATION
LOCATION: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
low_confidence_score_multiplier: 0.4
low_score_entity_names:
- ORG
- ORGANIZATION
labels_to_ignore:
- CARDINAL
- EVENT
- LANGUAGE
- LAW
- MONEY
- ORDINAL
- PERCENT
- PRODUCT
- QUANTITY
- WORK_OF_ART
presidio-analyzer/conf/spacy.yaml
+30
View File
@@ -3,3 +3,33 @@ models:
-
lang_code: en
model_name: en_core_web_lg
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
LOCATION: LOCATION
GPE: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
low_confidence_score_multiplier: 0.4
low_score_entity_names:
- ORG
- ORGANIZATION
labels_to_ignore:
- CARDINAL
- EVENT
- LANGUAGE
- LAW
- MONEY
- ORDINAL
- PERCENT
- PRODUCT
- QUANTITY
- WORK_OF_ART
presidio-analyzer/conf/spacy_multilingual.yaml
+19
View File
@@ -9,3 +9,22 @@ models:
-
lang_code: es
model_name: es_core_news_md
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
LOCATION: LOCATION
GPE: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
low_confidence_score_multiplier: 0.4
low_score_entity_names:
- ORG
- ORGANIZATION
presidio-analyzer/conf/stanza.yaml
+13
View File
@@ -4,3 +4,16 @@ models:
lang_code: en
model_name: en
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
LOCATION: LOCATION
GPE: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
presidio-analyzer/conf/stanza_multilingual.yaml
+13
View File
@@ -7,3 +7,16 @@ models:
lang_code: de
model_name: de
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
LOCATION: LOCATION
GPE: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
presidio-analyzer/conf/transformers.yaml
+38 -1
View File
@@ -4,4 +4,41 @@ models:
lang_code: en
model_name:
spacy: en_core_web_sm
transformers: elastic/distilbert-base-uncased-finetuned-conll03-english
transformers: StanfordAIMI/stanford-deidentifier-base
ner_model_configuration:
labels_to_ignore:
- O
aggregation_strategy: simple # "simple", "first", "average", "max"
stride: 16 # If stride >= 0, process long texts in
# overlapping windows of the model max
# length. The value is the length of the
# window overlap in transformer tokenizer
# tokens, NOT the length of the stride.
alignment_mode: expand # "strict", "contract", "expand"
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
LOC: LOCATION
LOCATION: LOCATION
GPE: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
NORP: NRP
AGE: AGE
ID: ID
EMAIL: EMAIL
PATIENT: PERSON
STAFF: PERSON
HOSP: ORGANIZATION
PATORG: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
PHONE: PHONE_NUMBER
HCW: PERSON
HOSPITAL: ORGANIZATION
FACILITY: LOCATION
low_confidence_score_multiplier: 0.4
low_score_entity_names:
- ID
presidio-analyzer/presidio_analyzer/analyzer_engine.py
+4 -1
View File
@@ -67,6 +67,9 @@ class AnalyzerEngine:
self.supported_languages = supported_languages
self.nlp_engine = nlp_engine
if not self.nlp_engine.is_loaded():
self.nlp_engine.load()
self.registry = registry
# load all recognizers
@@ -333,7 +336,7 @@ class AnalyzerEngine:
@staticmethod
def __add_recognizer_id_if_not_exists(
results: List[RecognizerResult], recognizer: EntityRecognizer
):
) -> None:
"""Ensure recognition metadata with recognizer id existence.
Ensure recognizer result list contains recognizer id inside recognition
presidio-analyzer/presidio_analyzer/nlp_engine/__init__.py
+2
View File
@@ -1,5 +1,6 @@
"""NLP engine package. Performs text pre-processing."""
from .ner_model_configuration import NerModelConfiguration
from .nlp_artifacts import NlpArtifacts
from .nlp_engine import NlpEngine
from .spacy_nlp_engine import SpacyNlpEngine
@@ -8,6 +9,7 @@ from .transformers_nlp_engine import TransformersNlpEngine
from .nlp_engine_provider import NlpEngineProvider
__all__ = [
"NerModelConfiguration",
"NlpArtifacts",
"NlpEngine",
"SpacyNlpEngine",
presidio-analyzer/presidio_analyzer/nlp_engine/ner_model_configuration.py
+121
View File
@@ -0,0 +1,121 @@
import logging
from dataclasses import dataclass
from typing import Dict, Optional, Collection, Type
logger = logging.getLogger("presidio-analyzer")
MODEL_TO_PRESIDIO_ENTITY_MAPPING = dict(
PER="PERSON",
PERSON="PERSON",
LOC="LOCATION",
LOCATION="LOCATION",
GPE="LOCATION",
ORG="ORGANIZATION",
DATE="DATE_TIME",
TIME="DATE_TIME",
NORP="NRP",
AGE="AGE",
ID="ID",
EMAIL="EMAIL",
PATIENT="PERSON",
STAFF="PERSON",
HOSP="ORGANIZATION",
PATORG="ORGANIZATION",
PHONE="PHONE_NUMBER",
HCW="PERSON",
HOSPITAL="ORGANIZATION",
)
LOW_SCORE_ENTITY_NAMES = {"ORG", "ORGANIZATION"}
LABELS_TO_IGNORE = {"O"}
@dataclass
class NerModelConfiguration:
"""NER model configuration.
:param labels_to_ignore: List of labels to not return predictions for.
:param aggregation_strategy:
See https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TokenClassificationPipeline.aggregation_strategy
:param stride:
See https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TokenClassificationPipeline.stride
:param alignment_mode: See https://spacy.io/api/doc#char_span
:param default_score: Default confidence score if the model does not provide one.
:param model_to_presidio_entity_mapping:
Mapping between the NER model entities and Presidio entities.
:param low_score_entity_names:
Set of entity names that are likely to have low detection accuracy that should be adjusted.
:param low_confidence_score_multiplier: A multiplier for the score given for low_score_entity_names.
Multiplier to the score given for low_score_entity_names.
""" # noqa E501
labels_to_ignore: Optional[Collection[str]] = None
aggregation_strategy: Optional[str] = "simple"
stride: Optional[int] = 14
alignment_mode: Optional[str] = "expand"
default_score: Optional[float] = 0.85
model_to_presidio_entity_mapping: Optional[Dict[str, str]] = None
low_score_entity_names: Optional[Collection] = None
low_confidence_score_multiplier: Optional[float] = 0.4
def __post_init__(self):
"""Validate the configuration and set defaults."""
if self.model_to_presidio_entity_mapping is None:
logger.warning(
"model_to_presidio_entity_mapping is missing from configuration, "
"using default"
)
self.model_to_presidio_entity_mapping = MODEL_TO_PRESIDIO_ENTITY_MAPPING
if self.low_score_entity_names is None:
logger.warning(
"low_score_entity_names is missing from configuration, " "using default"
)
self.low_score_entity_names = LOW_SCORE_ENTITY_NAMES
if self.labels_to_ignore is None:
logger.warning(
"labels_to_ignore is missing from configuration, " "using default"
)
self.labels_to_ignore = LABELS_TO_IGNORE
@classmethod
def _validate_input(cls, ner_model_configuration_dict: Dict) -> None:
key_to_type = {
"labels_to_ignore": list,
"aggregation_strategy": str,
"alignment_mode": str,
"model_to_presidio_entity_mapping": dict,
"low_confidence_score_multiplier": float,
"low_score_entity_names": list,
"stride": int,
}
for key, field_type in key_to_type.items():
cls.__validate_type(
config_dict=ner_model_configuration_dict, key=key, field_type=field_type
)
@staticmethod
def __validate_type(config_dict: Dict, key: str, field_type: Type) -> None:
if key in config_dict:
if not isinstance(config_dict[key], field_type):
raise ValueError(f"{key} must be of type {field_type}")
@classmethod
def from_dict(cls, nlp_engine_configuration: Dict) -> "NerModelConfiguration":
"""Load NLP engine configuration from dict.
:param nlp_engine_configuration: Dict with the configuration to load.
"""
cls._validate_input(nlp_engine_configuration)
return cls(**nlp_engine_configuration)
def to_dict(self) -> Dict:
"""Return the configuration as a dict."""
return self.__dict__
def __str__(self) -> str: # noqa D105
return str(self.to_dict())
def __repr__(self) -> str: # noqa D105
return str(self)
presidio-analyzer/presidio_analyzer/nlp_engine/nlp_artifacts.py
+14 -2
View File
@@ -1,5 +1,5 @@
import json
from typing import List
from typing import List, Optional
from spacy.tokens import Doc, Span
@@ -10,6 +10,14 @@ class NlpArtifacts:
processing over a given text, it holds attributes such as entities,
tokens and lemmas which can be used by any recognizer
:param entities: Identified entities
:param tokens: Tokenized text
:param tokens_indices: Indices of tokens
:param lemmas: List of lemmas in text
:param nlp_engine: NlpEngine object
:param language: Text language
:param scores: Entity confidence scores
"""
def __init__(
@@ -18,8 +26,9 @@ class NlpArtifacts:
tokens: Doc,
tokens_indices: List[int],
lemmas: List[str],
nlp_engine, # noqa ANN001
nlp_engine: "NlpEngine", # noqa F821
language: str,
scores: Optional[List[float]] = None,
):
self.entities = entities
self.tokens = tokens
@@ -27,6 +36,7 @@ class NlpArtifacts:
self.tokens_indices = tokens_indices
self.keywords = self.set_keywords(nlp_engine, lemmas, language)
self.nlp_engine = nlp_engine
self.scores = scores if scores else [0.85] * len(entities)
@staticmethod
def set_keywords(
@@ -70,5 +80,7 @@ class NlpArtifacts:
return_dict["tokens"] = [token.text for token in self.tokens]
if "entities" in return_dict:
return_dict["entities"] = [entity.text for entity in self.entities]
if "scores" in return_dict:
return_dict["scores"] = [float(score) for score in self.scores]
return json.dumps(return_dict)
presidio-analyzer/presidio_analyzer/nlp_engine/nlp_engine.py
+15 -2
View File
@@ -1,5 +1,5 @@
from abc import ABC, abstractmethod
from typing import Iterable, Iterator, Tuple
from typing import Iterable, Iterator, Tuple, List
from presidio_analyzer.nlp_engine import NlpArtifacts
@@ -12,13 +12,21 @@ class NlpEngine(ABC):
on tokens.
"""
@abstractmethod
def load(self) -> None:
"""Load the NLP model."""
@abstractmethod
def is_loaded(self) -> bool:
"""Return True if the model is already loaded."""
@abstractmethod
def process_text(self, text: str, language: str) -> NlpArtifacts:
"""Execute the NLP pipeline on the given text and language."""
@abstractmethod
def process_batch(
self, texts: Iterable[str], language: str, **kwargs
self, texts: Iterable[str], language: str, **kwargs # noqa ANN003
) -> Iterator[Tuple[str, NlpArtifacts]]:
"""Execute the NLP pipeline on a batch of texts.
@@ -40,3 +48,8 @@ class NlpEngine(ABC):
(within the given language)
"""
@abstractmethod
def get_supported_entities(self) -> List[str]:
"""Return the supported entities for this NLP engine."""
pass
presidio-analyzer/presidio_analyzer/nlp_engine/nlp_engine_provider.py
+21 -7
View File
@@ -9,6 +9,7 @@ from presidio_analyzer.nlp_engine import (
SpacyNlpEngine,
NlpEngine,
TransformersNlpEngine,
NerModelConfiguration,
)
logger = logging.getLogger("presidio-analyzer")
@@ -37,7 +38,6 @@ class NlpEngineProvider:
conf_file: Optional[Union[Path, str]] = None,
nlp_configuration: Optional[Dict] = None,
):
if not nlp_engines:
nlp_engines = (SpacyNlpEngine, StanzaNlpEngine, TransformersNlpEngine)
@@ -59,7 +59,7 @@ class NlpEngineProvider:
if conf_file:
self.nlp_configuration = self._read_nlp_conf(conf_file)
if not conf_file and not nlp_configuration:
if conf_file is None and nlp_configuration is None:
conf_file = self._get_full_conf_path()
logger.debug(f"Reading default conf file from {conf_file}")
self.nlp_configuration = self._read_nlp_conf(conf_file)
@@ -84,11 +84,20 @@ class NlpEngineProvider:
)
try:
nlp_engine_class = self.nlp_engines[nlp_engine_name]
nlp_engine_opts = {
m["lang_code"]: m["model_name"]
for m in self.nlp_configuration["models"]
}
engine = nlp_engine_class(nlp_engine_opts)
nlp_models = self.nlp_configuration["models"]
ner_model_configuration = self.nlp_configuration.get(
"ner_model_configuration"
)
if ner_model_configuration:
ner_model_configuration = NerModelConfiguration.from_dict(
ner_model_configuration
)
engine = nlp_engine_class(
models=nlp_models, ner_model_configuration=ner_model_configuration
)
engine.load()
logger.info(
f"Created NLP engine: {engine.engine_name}. "
f"Loaded models: {list(engine.nlp.keys())}"
@@ -114,6 +123,11 @@ class NlpEngineProvider:
else:
nlp_configuration = yaml.safe_load(open(conf_file))
if "ner_model_configuration" not in nlp_configuration:
logger.warning(
"configuration file is missing 'ner_model_configuration'. Using default"
)
return nlp_configuration
@staticmethod
presidio-analyzer/presidio_analyzer/nlp_engine/spacy_nlp_engine.py
+156 -15
View File
@@ -2,10 +2,11 @@ import logging
from typing import Optional, Dict, Iterator, Tuple, Union, List
import spacy
from spacy.language import Language
from spacy.tokens import Doc
from presidio_analyzer.nlp_engine import NlpArtifacts, NlpEngine
from spacy.language import Language
from spacy.tokens import Doc, Span
from presidio_analyzer.nlp_engine import NlpArtifacts, NlpEngine, NerModelConfiguration
logger = logging.getLogger("presidio-analyzer")
@@ -22,24 +23,74 @@ class SpacyNlpEngine(NlpEngine):
engine_name = "spacy"
is_available = bool(spacy)
def __init__(self, models: Optional[Dict[str, str]] = None):
def __init__(
self,
models: Optional[List[Dict[str, str]]] = None,
ner_model_configuration: Optional[NerModelConfiguration] = None,
):
"""
Initialize a wrapper on spaCy functionality.
:param models: Dictionary with the name of the spaCy model per language.
For example: models = {"en": "en_core_web_lg"}
For example: models = [{"lang_code": "en", "model_name": "en_core_web_lg"}]
:param ner_model_configuration: Parameters for the NER model.
See conf/spacy.yaml for an example
"""
if not models:
models = {"en": "en_core_web_lg"}
logger.debug(f"Loading SpaCy models: {models.values()}")
models = [{"lang_code": "en", "model_name": "en_core_web_lg"}]
self.models = models
self.nlp = {
lang_code: spacy.load(model_name, disable=["parser"])
for lang_code, model_name in models.items()
}
if not ner_model_configuration:
ner_model_configuration = NerModelConfiguration(self.engine_name)
self.ner_model_configuration = ner_model_configuration
self.nlp = None
def load(self) -> None:
"""Load the spaCy NLP model."""
logger.debug(f"Loading SpaCy models: {self.models}")
self.nlp = {}
# Download spaCy model if missing
for model in self.models:
self._validate_model_params(model)
self._download_spacy_model_if_needed(model["model_name"])
self.nlp[model["lang_code"]] = spacy.load(model["model_name"])
@staticmethod
def _download_spacy_model_if_needed(model_name: str) -> None:
if not spacy.util.is_package(model_name):
logger.warning(f"Model {model_name} is not installed. Downloading...")
spacy.cli.download(model_name)
logger.info(f"Finished downloading model {model_name}")
@staticmethod
def _validate_model_params(model: Dict) -> None:
if "lang_code" not in model:
raise ValueError("lang_code is missing from model configuration")
if "model_name" not in model:
raise ValueError("model_name is missing from model configuration")
if not isinstance(model["model_name"], str):
raise ValueError("model_name must be a string")
def get_supported_entities(self) -> List[str]:
"""Return the supported entities for this NLP engine."""
if not self.ner_model_configuration.model_to_presidio_entity_mapping:
raise ValueError(
"model_to_presidio_entity_mapping is missing from model configuration"
)
return list(
set(self.ner_model_configuration.model_to_presidio_entity_mapping.values())
)
def is_loaded(self) -> bool:
"""Return True if the model is already loaded."""
return self.nlp is not None
def process_text(self, text: str, language: str) -> NlpArtifacts:
"""Execute the SpaCy NLP pipeline on the given text and language."""
if not self.nlp:
raise ValueError("NLP engine is not loaded. Consider calling .load()")
doc = self.nlp[language](text)
return self._doc_to_nlp_artifact(doc, language)
@@ -50,7 +101,18 @@ class SpacyNlpEngine(NlpEngine):
language: str,
as_tuples: bool = False,
) -> Iterator[Optional[NlpArtifacts]]:
"""Execute the NLP pipeline on a batch of texts using spacy pipe."""
"""Execute the NLP pipeline on a batch of texts using spacy pipe.
:param texts: A list of texts to process.
:param language: The language of the texts.
:param as_tuples: If set to True, inputs should be a sequence of
(text, context) tuples. Output will then be a sequence of
(doc, context) tuples. Defaults to False.
"""
if not self.nlp:
raise ValueError("NLP engine is not loaded. Consider calling .load()")
texts = (str(text) for text in texts)
docs = self.nlp[language].pipe(texts, as_tuples=as_tuples)
for doc in docs:
@@ -76,15 +138,20 @@ class SpacyNlpEngine(NlpEngine):
"""
Return the language model loaded for a language.
:param language: Name of language
:return: Language model from spaCy
:param language: Language
:return: Model from spaCy
"""
return self.nlp[language]
def _doc_to_nlp_artifact(self, doc: Doc, language: str) -> NlpArtifacts:
lemmas = [token.lemma_ for token in doc]
tokens_indices = [token.idx for token in doc]
entities = doc.ents
entities = self._get_entities(doc)
scores = self._get_scores_for_entities(doc)
entities, scores = self._get_updated_entities(entities, scores)
return NlpArtifacts(
entities=entities,
tokens=doc,
@@ -92,4 +159,78 @@ class SpacyNlpEngine(NlpEngine):
lemmas=lemmas,
nlp_engine=self,
language=language,
scores=scores,
)
def _get_entities(self, doc: Doc) -> List[Span]:
"""
Extract entities out of a spaCy pipeline, depending on the type of pipeline.
For normal spaCy, this would be doc.ents
:param doc: the output spaCy doc.
:return: List of entities
"""
return doc.ents
def _get_scores_for_entities(self, doc: Doc) -> List[float]:
"""Extract scores for entities from the doc.
Since spaCy does not provide confidence scores for entities by default,
we use the default score from the ner model configuration.
:param doc: SpaCy doc
"""
entities = doc.ents
scores = [self.ner_model_configuration.default_score] * len(entities)
return scores
def _get_updated_entities(
self, entities: List[Span], scores: List[float]
) -> Tuple[List[Span], List[float]]:
"""
Get an updated list of entities based on the ner model configuration.
Remove entities that are in labels_to_ignore,
update entity names based on model_to_presidio_entity_mapping
:param entities: Entities that were extracted from a spaCy pipeline
:param scores: Original confidence scores for the entities extracted
:return: Tuple holding the entities and confidence scores
"""
if len(entities) != len(scores):
raise ValueError("Entities and scores must be the same length")
new_entities = []
new_scores = []
mapping = self.ner_model_configuration.model_to_presidio_entity_mapping
to_ignore = self.ner_model_configuration.labels_to_ignore
for ent, score in zip(entities, scores):
# Remove model labels in the ignore list
if ent.label_ in to_ignore:
continue
# Update entity label based on mapping
if ent.label_ in mapping:
ent.label_ = mapping[ent.label_]
else:
logger.warning(
f"Entity {ent.label_} is not mapped to a Presidio entity, "
f"but keeping anyway. "
f"Add to `NerModelConfiguration.labels_to_ignore` to remove."
)
# Remove presidio entities in the ignore list
if ent.label_ in to_ignore:
continue
new_entities.append(ent)
# Update score if entity is in low score entity names
if ent.label_ in self.ner_model_configuration.low_score_entity_names:
score *= self.ner_model_configuration.low_confidence_score_multiplier
new_scores.append(score)
return new_entities, new_scores
presidio-analyzer/presidio_analyzer/nlp_engine/stanza_nlp_engine.py
+28 -12
View File
@@ -1,4 +1,5 @@
import logging
from typing import Optional, Dict, List
try:
import stanza
@@ -6,7 +7,7 @@ try:
except ImportError:
stanza = None
from presidio_analyzer.nlp_engine import SpacyNlpEngine
from presidio_analyzer.nlp_engine import SpacyNlpEngine, NerModelConfiguration
logger = logging.getLogger("presidio-analyzer")
@@ -19,22 +20,37 @@ class StanzaNlpEngine(SpacyNlpEngine):
on tokens.
The StanzaNlpEngine uses spacy-stanza and stanza as its NLP module
:param models: Dictionary with the name of the stanza model per language.
For example: models = {"en": "en"}
:param models: Dictionary with the name of the spaCy model per language.
For example: models = [{"lang_code": "en", "model_name": "en"}]
:param ner_model_configuration: Parameters for the NER model.
See conf/stanza.yaml for an example
"""
engine_name = "stanza"
is_available = bool(stanza)
def __init__(self, models=None): # noqa ANN201
if not models:
models = {"en": "en"}
logger.debug(f"Loading Stanza models: {models.values()}")
def __init__(
self,
models: Optional[List[Dict[str, str]]] = None,
ner_model_configuration: Optional[NerModelConfiguration] = None,
download_if_missing: bool = True,
):
super().__init__(models, ner_model_configuration)
self.download_if_missing = download_if_missing
self.nlp = {
lang_code: spacy_stanza.load_pipeline(
model_name,
def load(self) -> None:
"""Load the NLP model."""
logger.debug(f"Loading Stanza models: {self.models}")
self.nlp = {}
for model in self.models:
self._validate_model_params(model)
self.nlp[model["lang_code"]] = spacy_stanza.load_pipeline(
model["model_name"],
processors="tokenize,pos,lemma,ner",
download_method="DOWNLOAD_RESOURCES"
if self.download_if_missing
else None,
)
for lang_code, model_name in models.items()
}
presidio-analyzer/presidio_analyzer/nlp_engine/transformers_nlp_engine.py
+90 -110
View File
@@ -1,96 +1,44 @@
import logging
from typing import Optional, Dict
from typing import Optional, Dict, List
import spacy
from spacy.language import Language
from spacy.tokens import Doc, Span
from presidio_analyzer.nlp_engine import SpacyNlpEngine
try:
import torch
import spacy_huggingface_pipelines
import transformers
from transformers import (
AutoTokenizer,
AutoModelForTokenClassification,
pipeline,
)
except ImportError:
torch = None
spacy_huggingface_pipelines = None
transformers = None
logger = logging.getLogger("presidio-analyzer")
@Language.factory(
"transformers",
default_config={"pretrained_model_name_or_path": "dslim/bert-base-NER"},
from presidio_analyzer.nlp_engine import (
SpacyNlpEngine,
NerModelConfiguration,
)
def create_transformer_component(nlp, name, pretrained_model_name_or_path: str):
"""Spacy Language factory for creating custom component."""
return TransformersComponent(
pretrained_model_name_or_path=pretrained_model_name_or_path
)
class TransformersComponent:
"""
Custom component to use in spacy pipeline.
Using HaggingFace transformers pretrained models for entity recognition.
:param pretrained_model_name_or_path: HaggingFace pretrained_model_name_or_path
"""
def __init__(self, pretrained_model_name_or_path: str) -> None:
Span.set_extension("confidence_score", default=1.0, force=True)
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)
model = AutoModelForTokenClassification.from_pretrained(
pretrained_model_name_or_path
)
self.nlp = pipeline(
"ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple"
)
def __call__(self, doc: Doc) -> Doc:
"""Write transformers results to doc entities."""
res = self.nlp(doc.text)
ents = []
for d in res:
span = doc.char_span(
d["start"], d["end"], label=d["entity_group"], alignment_mode="expand"
)
if span is not None:
span._.confidence_score = d["score"]
ents.append(span)
else:
logger.warning(
f"Transformers model returned {d} but no valid span was found."
)
doc.ents = ents
return doc
logger = logging.getLogger("presidio-analyzer")
class TransformersNlpEngine(SpacyNlpEngine):
"""
SpacyTransformersNlpEngine is a transformers based NlpEngine.
TransformersNlpEngine is a transformers based NlpEngine.
It comprises a spacy pipeline used for tokenization,
lemmatization, pos, and a transformers component for NER.
Both the underlying spacy pipeline and the transformers engine could be
configured by the user.
:param models: a dictionary containing the model names per language.
:param models: A dict holding the model's configuration.
:example:
{
"en": {
[{"lang_code": "en", "model_name": {
"spacy": "en_core_web_sm",
"transformers": "dslim/bert-base-NER"
}
}
}
}]
:param ner_model_configuration: Parameters for the NER model.
See conf/transformers.yaml for an example
Note that since the spaCy model is not used for NER,
we recommend using a simple model, such as en_core_web_sm for English.
@@ -98,57 +46,89 @@ class TransformersNlpEngine(SpacyNlpEngine):
https://huggingface.co/models?pipeline_tag=token-classification
It is further recommended to fine-tune these models
to the specific scenario in hand.
"""
engine_name = "transformers"
is_available = bool(spacy) and bool(transformers)
is_available = bool(spacy_huggingface_pipelines)
def __init__(self, models: Optional[Dict[str, Dict[str, str]]] = None):
# default models if not specified
def __init__(
self,
models: Optional[Dict] = None,
ner_model_configuration: Optional[NerModelConfiguration] = None,
):
if not models:
models = {
"en": {"spacy": "en_core_web_sm", "transformers": "dslim/bert-base-NER"}
}
# validate models type
elif type(models) is not dict:
logger.error(f"''models' argument must be dict, not {type(models)}")
raise KeyError(f"Expected 'models' argument to be dict, not {type(models)}")
# validate models[model_lang] type is dict for all model_lang
elif any(
[type(model_dict) is not dict for model_lang, model_dict in models.items()]
):
# elif type(models["model_name"]) is not dict:
logger.error(
"'models.model_name' argument must be dict,"
f"not {type(models['model_name'])}"
)
raise KeyError(
"Expected 'models.model_name' argument to be dict,"
f"not {type(models['model_name'])}"
)
# chack that model_name dict includes the keys: "spacy" and "transformers"
elif any(
[
any([key not in model_dict for key in ("spacy", "transformers")])
for model_lang, model_dict in models.items()
models = [
{
"lang_code": "en",
"model_name": {
"spacy": "en_core_web_sm",
"transformers": "obi/deid_roberta_i2b2",
},
}
]
):
logger.error(
"'models.model_name' must contains 'spacy' and 'transformers' keys"
)
raise KeyError(
"Expected keys ('spacy' and 'transformers') was not found in "
"models.model_name dict"
)
super().__init__(models=models, ner_model_configuration=ner_model_configuration)
self.entity_key = "bert-base-ner"
logger.debug(f"Loading SpaCy and transformers models: {models.values()}")
def load(self) -> None:
"""Load the spaCy and transformers models."""
logger.debug(f"Loading SpaCy and transformers models: {self.models}")
self.nlp = {}
for lang_code, model_name in models.items():
nlp = spacy.load(model_name["spacy"], disable=["parser", "ner"])
for model in self.models:
self._validate_model_params(model)
spacy_model = model["model_name"]["spacy"]
transformers_model = model["model_name"]["transformers"]
self._download_spacy_model_if_needed(spacy_model)
nlp = spacy.load(spacy_model, disable=["parser", "ner"])
nlp.add_pipe(
"transformers",
config={"pretrained_model_name_or_path": model_name["transformers"]},
last=True,
"hf_token_pipe",
config={
"model": transformers_model,
"annotate": "spans",
"stride": self.ner_model_configuration.stride,
"alignment_mode": self.ner_model_configuration.alignment_mode,
"aggregation_strategy": self.ner_model_configuration.aggregation_strategy, # noqa E501
"annotate_spans_key": self.entity_key,
},
)
self.nlp[lang_code] = nlp
self.nlp[model["lang_code"]] = nlp
@staticmethod
def _validate_model_params(model: Dict) -> None:
if "lang_code" not in model:
raise ValueError("lang_code is missing from model configuration")
if "model_name" not in model:
raise ValueError("model_name is missing from model configuration")
if not isinstance(model["model_name"], dict):
raise ValueError("model_name must be a dictionary")
if "spacy" not in model["model_name"]:
raise ValueError("spacy model name is missing from model configuration")
if "transformers" not in model["model_name"]:
raise ValueError(
"transformers model name is missing from model configuration"
)
def _get_entities(self, doc: Doc) -> List[Span]:
"""
Extract entities out of a spaCy pipeline, depending on the type of pipeline.
For spacy-huggingface-pipeline, this would be doc.spans[key]
:param doc: the output spaCy doc.
:return: List of entities
"""
return doc.spans[self.entity_key]
def _get_scores_for_entities(self, doc: Doc) -> List[float]:
"""Extract scores for entities from the doc.
While spaCy does not provide confidence scores,
the spacy-huggingface-pipeline flow adds confidence scores
as SpanGroup attributes.
:param doc: SpaCy doc
"""
return doc.spans[self.entity_key].attrs["scores"]
presidio-analyzer/presidio_analyzer/predefined_recognizers/__init__.py
+8 -7
View File
@@ -8,8 +8,14 @@ from .credit_card_recognizer import CreditCardRecognizer
from .crypto_recognizer import CryptoRecognizer
from .date_recognizer import DateRecognizer
from .email_recognizer import EmailRecognizer
from .es_nif_recognizer import EsNifRecognizer
from .iban_recognizer import IbanRecognizer
from .ip_recognizer import IpRecognizer
from .it_driver_license_recognizer import ItDriverLicenseRecognizer
from .it_fiscal_code_recognizer import ItFiscalCodeRecognizer
from .it_identity_card_recognizer import ItIdentityCardRecognizer
from .it_passport_recognizer import ItPassportRecognizer
from .it_vat_code import ItVatCodeRecognizer
from .medical_license_recognizer import MedicalLicenseRecognizer
from .phone_recognizer import PhoneRecognizer
from .sg_fin_recognizer import SgFinRecognizer
@@ -22,18 +28,13 @@ from .us_driver_license_recognizer import UsLicenseRecognizer
from .us_itin_recognizer import UsItinRecognizer
from .us_passport_recognizer import UsPassportRecognizer
from .us_ssn_recognizer import UsSsnRecognizer
from .es_nif_recognizer import EsNifRecognizer
from .au_abn_recognizer import AuAbnRecognizer
from .au_acn_recognizer import AuAcnRecognizer
from .au_tfn_recognizer import AuTfnRecognizer
from .au_medicare_recognizer import AuMedicareRecognizer
from .it_driver_license_recognizer import ItDriverLicenseRecognizer
from .it_fiscal_code_recognizer import ItFiscalCodeRecognizer
from .it_vat_code import ItVatCodeRecognizer
from .it_identity_card_recognizer import ItIdentityCardRecognizer
from .it_passport_recognizer import ItPassportRecognizer
from .in_pan_recognizer import InPanRecognizer
NLP_RECOGNIZERS = {
"spacy": SpacyRecognizer,
"stanza": StanzaRecognizer,
@@ -72,5 +73,5 @@ __all__ = [
"ItVatCodeRecognizer",
"ItIdentityCardRecognizer",
"ItPassportRecognizer",
"InPanRecognizer"
"InPanRecognizer",
]
presidio-analyzer/presidio_analyzer/predefined_recognizers/spacy_recognizer.py
+52 -46
View File
@@ -1,4 +1,5 @@
import logging
import warnings
from typing import Optional, List, Tuple, Set
from presidio_analyzer import (
@@ -14,29 +15,17 @@ class SpacyRecognizer(LocalRecognizer):
"""
Recognize PII entities using a spaCy NLP model.
Since the spaCy pipeline is ran by the AnalyzerEngine,
this recognizer only extracts the entities from the NlpArtifacts
and replaces their types to align with Presidio's.
Since the spaCy pipeline is ran by the AnalyzerEngine/SpacyNlpEngine,
this recognizer only extracts the entities from the NlpArtifacts
and returns them.
:param supported_language: Language this recognizer supports
:param supported_entities: The entities this recognizer can detect
:param ner_strength: Default confidence for NER prediction
:param check_label_groups: Tuple containing Presidio entity names
and spaCy entity names, for verifying that the right entity
is translated into a Presidio entity.
"""
ENTITIES = [
"DATE_TIME",
"NRP",
"LOCATION",
"PERSON",
# "ORGANIZATION" - Less accurate with the 'en_core_web_lg' model,
# can be used with more assurance when using 'en_core_web_trf'.
]
ENTITIES = ["DATE_TIME", "NRP", "LOCATION", "PERSON", "ORGANIZATION"]
DEFAULT_EXPLANATION = "Identified as {} by Spacy's Named Entity Recognition"
# deprecated, use MODEL_TO_PRESIDIO_MAPPING in NerModelConfiguration instead
CHECK_LABEL_GROUPS = [
({"LOCATION"}, {"GPE", "LOC"}),
({"PERSON", "PER"}, {"PERSON", "PER"}),
@@ -50,12 +39,30 @@ class SpacyRecognizer(LocalRecognizer):
supported_language: str = "en",
supported_entities: Optional[List[str]] = None,
ner_strength: float = 0.85,
default_explanation: Optional[str] = None,
check_label_groups: Optional[List[Tuple[Set, Set]]] = None,
context: Optional[List[str]] = None,
):
"""
:param supported_language: Language this recognizer supports
:param supported_entities: The entities this recognizer can detect
:param ner_strength: Default confidence for NER prediction
:param check_label_groups: (DEPRECATED) Tuple containing Presidio entity names
:param default_explanation: Default explanation for the results when using return_decision_process=True
""" # noqa E501
self.ner_strength = ner_strength
self.check_label_groups = (
check_label_groups if check_label_groups else self.CHECK_LABEL_GROUPS
if check_label_groups:
warnings.warn(
"check_label_groups is deprecated and isn't used;"
"entities are mapped in NerModelConfiguration",
DeprecationWarning,
2,
)
self.default_explanation = (
default_explanation if default_explanation else self.DEFAULT_EXPLANATION
)
supported_entities = supported_entities if supported_entities else self.ENTITIES
super().__init__(
@@ -69,7 +76,7 @@ class SpacyRecognizer(LocalRecognizer):
# preprocessed nlp artifacts
pass
def build_spacy_explanation(
def build_explanation(
self, original_score: float, explanation: str
) -> AnalysisExplanation:
"""
@@ -80,42 +87,43 @@ class SpacyRecognizer(LocalRecognizer):
:return:
"""
explanation = AnalysisExplanation(
recognizer=self.__class__.__name__,
recognizer=self.name,
original_score=original_score,
textual_explanation=explanation,
)
return explanation
def analyze(self, text, entities, nlp_artifacts=None): # noqa D102
def analyze(self, text: str, entities, nlp_artifacts=None): # noqa D102
results = []
if not nlp_artifacts:
logger.warning("Skipping SpaCy, nlp artifacts not provided...")
return results
ner_entities = nlp_artifacts.entities
ner_scores = nlp_artifacts.scores
for entity in entities:
if entity not in self.supported_entities:
for ner_entity, ner_score in zip(ner_entities, ner_scores):
if ner_entity.label_ not in entities:
logger.debug(
f"Skipping entity {ner_entity.label_} "
f"as it is not in the supported entities list"
)
continue
for ent in ner_entities:
if not self.__check_label(entity, ent.label_, self.check_label_groups):
continue
textual_explanation = self.DEFAULT_EXPLANATION.format(ent.label_)
explanation = self.build_spacy_explanation(
self.ner_strength, textual_explanation
)
spacy_result = RecognizerResult(
entity_type=entity,
start=ent.start_char,
end=ent.end_char,
score=self.ner_strength,
analysis_explanation=explanation,
recognition_metadata={
RecognizerResult.RECOGNIZER_NAME_KEY: self.name,
RecognizerResult.RECOGNIZER_IDENTIFIER_KEY: self.id,
},
)
results.append(spacy_result)
textual_explanation = self.DEFAULT_EXPLANATION.format(ner_entity.label_)
explanation = self.build_explanation(ner_score, textual_explanation)
spacy_result = RecognizerResult(
entity_type=ner_entity.label_,
start=ner_entity.start_char,
end=ner_entity.end_char,
score=ner_score,
analysis_explanation=explanation,
recognition_metadata={
RecognizerResult.RECOGNIZER_NAME_KEY: self.name,
RecognizerResult.RECOGNIZER_IDENTIFIER_KEY: self.id,
},
)
results.append(spacy_result)
return results
@@ -123,6 +131,4 @@ class SpacyRecognizer(LocalRecognizer):
def __check_label(
entity: str, label: str, check_label_groups: Tuple[Set, Set]
) -> bool:
return any(
[entity in egrp and label in lgrp for egrp, lgrp in check_label_groups]
)
raise DeprecationWarning("__check_label is deprecated")
presidio-analyzer/presidio_analyzer/predefined_recognizers/transformers_recognizer.py
+18 -56
View File
@@ -1,72 +1,34 @@
from typing import Tuple, Set
import logging
from presidio_analyzer.predefined_recognizers.spacy_recognizer import SpacyRecognizer
from presidio_analyzer import RecognizerResult
logger = logging.getLogger("presidio-analyzer")
class TransformersRecognizer(SpacyRecognizer):
"""
Recognize entities using the transformers package.
Recognize entities using the spacy-huggingface-pipeline package.
The recognizer doesn't run transformers models,
but loads the output from the NlpArtifacts
See https://huggingface.co/docs/transformers/main/en/index
Uses the transformers package
(https://huggingface.co/docs/transformers/main/en/installation) to align
transformers interface with spaCy
"""
See:
- https://huggingface.co/docs/transformers/main/en/index for transformer models
- https://github.com/explosion/spacy-huggingface-pipelines on the spaCy wrapper to transformers
""" # noqa E501
ENTITIES = [
"PERSON",
"LOCATION",
"ORGANIZATION",
"AGE",
"ID",
"EMAIL",
"DATE_TIME",
"PHONE_NUMBER",
]
def __init__(self, **kwargs): # noqa ANN003
self.DEFAULT_EXPLANATION = self.DEFAULT_EXPLANATION.replace(
"Spacy", "Transfromers"
"Spacy", "Transformers"
)
super().__init__(**kwargs)
def analyze(self, text, entities, nlp_artifacts=None): # noqa D102
results = []
if not nlp_artifacts:
logger.warning("Skipping SpaCy, nlp artifacts not provided...")
return results
ner_entities = nlp_artifacts.entities
for entity in entities:
if entity not in self.supported_entities:
continue
for ent in ner_entities:
if not self.__check_label(entity, ent.label_, self.check_label_groups):
continue
if not ent.has_extension("confidence_score"):
raise ValueError(
"confidence score not available as a spaCy span extension "
"(ent._.confidence_score)"
)
confidence_score = ent._.confidence_score
textual_explanation = self.DEFAULT_EXPLANATION.format(ent.label_)
explanation = self.build_spacy_explanation(
confidence_score, textual_explanation
)
spacy_result = RecognizerResult(
entity_type=entity,
start=ent.start_char,
end=ent.end_char,
score=confidence_score,
analysis_explanation=explanation,
recognition_metadata={
RecognizerResult.RECOGNIZER_NAME_KEY: self.name,
RecognizerResult.RECOGNIZER_IDENTIFIER_KEY: self.id,
},
)
results.append(spacy_result)
return results
@staticmethod
def __check_label(
entity: str, label: str, check_label_groups: Tuple[Set, Set]
) -> bool:
return any(
[entity in egrp and label in lgrp for egrp, lgrp in check_label_groups]
)
presidio-analyzer/presidio_analyzer/recognizer_registry/recognizer_registry.py
+14 -6
View File
@@ -1,7 +1,8 @@
import copy
import logging
from pathlib import Path
from typing import Optional, List, Iterable, Union, Type, Dict
from pathlib import Path
from presidio_analyzer.nlp_engine.transformers_nlp_engine import (
TransformersNlpEngine,
)
@@ -55,7 +56,6 @@ class RecognizerRegistry:
"""
def __init__(self, recognizers: Optional[Iterable[EntityRecognizer]] = None):
if recognizers:
self.recognizers = recognizers
else:
@@ -75,6 +75,7 @@ class RecognizerRegistry:
languages = ["en"]
nlp_recognizer = self._get_nlp_recognizer(nlp_engine)
recognizers_map = {
"en": [
UsBankRecognizer,
@@ -106,7 +107,6 @@ class RecognizerRegistry:
IbanRecognizer,
IpRecognizer,
MedicalLicenseRecognizer,
nlp_recognizer,
PhoneRecognizer,
UrlRecognizer,
],
@@ -118,11 +118,19 @@ class RecognizerRegistry:
rc(supported_language=lang) for rc in recognizers_map.get("ALL", [])
]
self.recognizers.extend(all_recognizers)
if nlp_engine:
nlp_recognizer_inst = nlp_recognizer(
supported_language=lang,
supported_entities=nlp_engine.get_supported_entities(),
)
else:
nlp_recognizer_inst = nlp_recognizer(supported_language=lang)
self.recognizers.append(nlp_recognizer_inst)
@staticmethod
def _get_nlp_recognizer(
nlp_engine: NlpEngine,
) -> Union[Type[SpacyRecognizer], Type[StanzaRecognizer]]:
) -> Type[SpacyRecognizer]:
"""Return the recognizer leveraging the selected NLP Engine."""
if isinstance(nlp_engine, StanzaNlpEngine):
@@ -231,7 +239,7 @@ class RecognizerRegistry:
)
self.recognizers = new_recognizers
def add_pattern_recognizer_from_dict(self, recognizer_dict: Dict):
def add_pattern_recognizer_from_dict(self, recognizer_dict: Dict) -> None:
"""
Load a pattern recognizer from a Dict into the recognizer registry.
@@ -246,7 +254,7 @@ class RecognizerRegistry:
recognizer = PatternRecognizer.from_dict(recognizer_dict)
self.add_recognizer(recognizer)
def add_recognizers_from_yaml(self, yml_path: Union[str, Path]):
def add_recognizers_from_yaml(self, yml_path: Union[str, Path]) -> None:
r"""
Read YAML file and load recognizers into the recognizer registry.
presidio-analyzer/setup.cfg
+1 -1
View File
@@ -7,4 +7,4 @@ exclude =
dist,
tests
docstring-convention = numpy
extend-ignore = E203 D100 D202 ANN101 ANN102 ANN204 ANN203
extend-ignore = E203 D100 D202 ANN101 ANN102 ANN204 ANN203 TC
presidio-analyzer/setup.py
+2 -1
View File
@@ -39,7 +39,8 @@ setuptools.setup(
"phonenumbers>=8.12",
],
extras_require={
'transformers': ['torch', 'transformers'],
"transformers": ["spacy_huggingface_pipelines"],
"stanza": ["stanza", "spacy_stanza"],
},
include_package_data=True,
license="MIT",
presidio-analyzer/tests/conf/default.yaml
+13
View File
@@ -4,3 +4,16 @@ models:
lang_code: en
model_name: en_core_web_lg
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: LOCATION
LOC: LOCATION
GPE: LOCATION
LOCATION: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
presidio-analyzer/tests/conf/test.yaml
+14
View File
@@ -6,3 +6,17 @@ models:
-
lang_code: bn
model_name: bn_test
ner_model_configuration:
model_to_presidio_entity_mapping:
PER: PERSON
PERSON: PERSON
NORP: NRP
FAC: FACILITY
LOC: LOCATION
GPE: LOCATION
LOCATION: LOCATION
ORG: ORGANIZATION
ORGANIZATION: ORGANIZATION
DATE: DATE_TIME
TIME: DATE_TIME
presidio-analyzer/tests/conf/test_transformers.yaml
+36
View File
@@ -0,0 +1,36 @@
nlp_engine_name: transformers
models:
-
lang_code: en
model_name:
spacy: en_core_web_lg
transformers: StanfordAIMI/stanford-deidentifier-base
ner_model_configuration:
labels_to_ignore:
- O
aggregation_strategy: simple # "simple", "first", "average", "max"
stride: 16 # If stride >= 0, process long texts in
# overlapping windows of the model max
# length. The value is the length of the
# window overlap in transformer tokenizer
# tokens, NOT the length of the stride.
alignment_mode: strict # "strict", "contract", "expand"
model_to_presidio_entity_mapping:
PER: PERSON
LOC: LOCATION
ORG: ORGANIZATION
AGE: AGE
ID: ID
EMAIL: EMAIL
PATIENT: PERSON
STAFF: PERSON
HOSP: ORGANIZATION
PATORG: ORGANIZATION
DATE: DATE_TIME
PHONE: PHONE_NUMBER
HCW: PERSON
HOSPITAL: ORGANIZATION
low_confidence_score_multiplier: 0.4
low_score_entity_names:
- ID
presidio-analyzer/tests/conftest.py
+49 -76
View File
@@ -3,7 +3,6 @@ from pathlib import Path
from typing import Dict
import pytest
import spacy
from presidio_analyzer import (
EntityRecognizer,
@@ -14,31 +13,15 @@ from presidio_analyzer import (
from presidio_analyzer import RecognizerRegistry
from presidio_analyzer.nlp_engine import NlpEngineProvider, NlpEngine
from presidio_analyzer.predefined_recognizers import NLP_RECOGNIZERS
from tests.mocks import RecognizerRegistryMock
def pytest_addoption(parser):
parser.addoption(
"--runfast", action="store_true", default=False, help="run fast tests"
)
from tests.mocks import RecognizerRegistryMock, NlpEngineMock
def pytest_configure(config):
config.addinivalue_line("markers", "slow: mark test as slow to run")
config.addinivalue_line(
"markers", "skip_engine(nlp_engine): skip test for given nlp engine"
)
def pytest_collection_modifyitems(config, items):
if config.getoption("--runfast"):
# --runfast given in cli: skip slow tests
skip_slow = pytest.mark.skip(reason="remove --runfast option to run")
for item in items:
if "slow" in item.keywords:
item.add_marker(skip_slow)
@pytest.fixture(scope="session")
def nlp_engine_provider() -> NlpEngineProvider:
return NlpEngineProvider()
@@ -50,19 +33,28 @@ def nlp_engines(request, nlp_engine_provider) -> Dict[str, NlpEngine]:
nlp_engines = nlp_engine_provider.nlp_engines
for name, engine_cls in nlp_engines.items():
if name == "spacy" and not request.config.getoption("--runfast"):
available_engines[f"{name}_en"] = engine_cls({"en": "en_core_web_lg"})
elif name == "transformers" and not request.config.getoption("--runfast"):
if name == "spacy":
available_engines[f"{name}_en"] = engine_cls(
{
"en": {
"spacy": "en_core_web_lg",
"transformers": "dslim/bert-base-NER",
models=[{"lang_code": "en", "model_name": "en_core_web_lg"}]
)
elif name == "stanza":
available_engines[f"{name}_en"] = engine_cls(
models=[{"lang_code": "en", "model_name": "en"}]
)
elif name == "transformers":
available_engines[f"{name}_en"] = engine_cls(
models=[
{
"lang_code": "en",
"model_name": {
"spacy": "en_core_web_sm",
"transformers": "StanfordAIMI/stanford-deidentifier-base",
},
}
}
]
)
else:
available_engines[f"{name}_en"] = engine_cls()
raise ValueError("Unsupported engine for tests")
return available_engines
@@ -76,6 +68,15 @@ def skip_by_engine(request, nlp_engines):
pytest.skip(f"skipped on this engine: {marker_arg}")
@pytest.mark.skip_engine("spacy_en")
@pytest.fixture(scope="session")
def spacy_nlp_engine(nlp_engines):
nlp_engine = nlp_engines.get("spacy_en", None)
if nlp_engine:
nlp_engine.load()
return nlp_engine
@pytest.fixture(scope="session")
def nlp_recognizers() -> Dict[str, EntityRecognizer]:
return {name: rec_cls() for name, rec_cls in NLP_RECOGNIZERS.items()}
@@ -91,46 +92,29 @@ def max_score() -> float:
return EntityRecognizer.MAX_SCORE
@pytest.fixture(scope="session")
def min_score() -> float:
return EntityRecognizer.MIN_SCORE
@pytest.fixture(scope="module")
def loaded_registry() -> RecognizerRegistry:
return RecognizerRegistry()
@pytest.fixture(scope="module")
def nlp_engine(nlp_engines) -> NlpEngine:
return nlp_engines["spacy_en"]
@pytest.fixture(scope="module")
def mock_registry() -> RecognizerRegistryMock:
return RecognizerRegistryMock()
@pytest.fixture(scope="module")
def analyzer_engine_simple(mock_registry, nlp_engine) -> AnalyzerEngine:
return AnalyzerEngine(registry=mock_registry, nlp_engine=nlp_engine)
def mock_nlp_engine() -> NlpEngineMock:
return NlpEngineMock()
@pytest.fixture(scope="session")
def mock_he_model():
"""
Create an empty Hebrew spaCy pipeline and save it to disk.
So that it could be loaded using spacy.load()
"""
he = spacy.blank("he")
he.to_disk("he_test")
@pytest.fixture(scope="session")
def mock_bn_model():
"""
Create an empty Bengali spaCy pipeline and save it to disk.
So that it could be loaded using spacy.load()
"""
bn = spacy.blank("bn")
bn.to_disk("bn_test")
@pytest.fixture(scope="module")
def analyzer_engine_simple(mock_registry, mock_nlp_engine) -> AnalyzerEngine:
return AnalyzerEngine(registry=mock_registry, nlp_engine=mock_nlp_engine)
@pytest.fixture(scope="session")
@@ -143,28 +127,17 @@ def zip_code_recognizer():
return zip_recognizer
@pytest.fixture(scope="session")
def zip_code_deny_list_recognizer():
regex = r"(\b\d{5}(?:\-\d{4})?\b)"
zipcode_pattern = Pattern(name="zip code (weak)", regex=regex, score=0.01)
zip_recognizer = PatternRecognizer(
supported_entity="ZIP", deny_list=["999"], patterns=[zipcode_pattern]
)
return zip_recognizer
def pytest_sessionfinish():
"""Remove files created during mock spaCy models creation."""
he_test_model_path = Path(Path(__file__).parent.parent, "he_test")
if he_test_model_path.exists():
try:
shutil.rmtree(he_test_model_path)
except OSError as e:
print("Failed to remove file: %s - %s." % (e.filename, e.strerror))
bn_test_model_path = Path(Path(__file__).parent.parent, "bn_test")
if bn_test_model_path.exists():
try:
shutil.rmtree(bn_test_model_path)
except OSError as e:
print("Failed to remove file: %s - %s." % (e.filename, e.strerror))
mock_models = ("he_test", "bn_test")
for mock_model in mock_models:
test_model_path1 = Path(Path(__file__).parent, mock_model)
test_model_path2 = Path(Path(__file__).parent.parent, mock_model)
for path in (test_model_path1, test_model_path2):
if path.exists():
try:
shutil.rmtree(path)
except OSError as e:
print("Failed to remove file: %s - %s." % (e.filename, e.strerror))
presidio-analyzer/tests/mocks/app_tracer_mock.py
-1
View File
@@ -5,7 +5,6 @@ from presidio_analyzer.app_tracer import AppTracer
class AppTracerMock(AppTracer):
def __init__(self, enable_decision_process=True):
logger = logging.getLogger("DecisionProcessMock")
if not logger.handlers:
ch = logging.StreamHandler()
presidio-analyzer/tests/mocks/nlp_engine_mock.py
+14 -2
View File
@@ -1,4 +1,4 @@
from typing import Iterable, Iterator, Tuple
from typing import Iterable, Iterator, Tuple, Dict, List
from presidio_analyzer.nlp_engine import NlpEngine, NlpArtifacts
@@ -8,10 +8,16 @@ class NlpEngineMock(NlpEngine):
self.stopwords = stopwords if stopwords else []
self.punct_words = punct_words if punct_words else []
if nlp_artifacts is None:
self.nlp_artifacts = NlpArtifacts([], [], [], [], None, "en")
self.nlp_artifacts = NlpArtifacts([], [], [], [], None, "en", [])
else:
self.nlp_artifacts = nlp_artifacts
def load(self):
pass
def is_loaded(self) -> bool:
return True
def is_stopword(self, word, language):
return word in self.stopwords
@@ -27,3 +33,9 @@ class NlpEngineMock(NlpEngine):
texts = list(texts)
for i in range(len(texts)):
yield texts[i], self.nlp_artifacts
def get_nlp_engine_configuration_as_dict(self) -> Dict:
return {}
def get_supported_entities(self) -> List[str]:
pass
presidio-analyzer/tests/test_analyzer_engine.py
+30 -21
View File
@@ -39,6 +39,16 @@ def loaded_analyzer_engine(loaded_registry, app_tracer):
return analyzer_engine
@pytest.fixture(scope="module")
def zip_code_deny_list_recognizer():
regex = r"(\b\d{5}(?:\-\d{4})?\b)"
zipcode_pattern = Pattern(name="zip code (weak)", regex=regex, score=0.01)
zip_recognizer = PatternRecognizer(
supported_entity="ZIP", deny_list=["999"], patterns=[zipcode_pattern]
)
return zip_recognizer
@pytest.fixture(scope="module")
def unit_test_guid():
return "00000000-0000-0000-0000-000000000000"
@@ -79,14 +89,14 @@ def test_when_analyze_with_predefined_recognizers_then_return_results(
def test_when_analyze_with_multiple_predefined_recognizers_then_succeed(
loaded_registry, unit_test_guid, nlp_engine, max_score
loaded_registry, unit_test_guid, spacy_nlp_engine, max_score
):
text = " Credit card: 4095-2609-9393-4932, my phone is 425 8829090"
language = "en"
entities = ["CREDIT_CARD", "PHONE_NUMBER"]
analyzer_engine_with_spacy = AnalyzerEngine(
registry=loaded_registry, nlp_engine=nlp_engine
registry=loaded_registry, nlp_engine=spacy_nlp_engine
)
results = analyzer_engine_with_spacy.analyze(
correlation_id=unit_test_guid,
@@ -134,8 +144,8 @@ def test_when_analyze_with_unsupported_language_then_fail(
)
def test_when_analyze_two_entities_embedded_then_return_results(nlp_engine):
analyzer = AnalyzerEngine(nlp_engine=nlp_engine)
def test_when_analyze_two_entities_embedded_then_return_results(spacy_nlp_engine):
analyzer = AnalyzerEngine(nlp_engine=spacy_nlp_engine)
# Name with driver license in it
text = "My name is John 1234567 Doe"
@@ -152,7 +162,6 @@ def test_when_analyze_added_pattern_recognizer_then_succeed(unit_test_guid):
)
mock_recognizer_registry = RecognizerRegistryMock()
# Make sure the analyzer doesn't get this entity
analyze_engine = AnalyzerEngine(
registry=mock_recognizer_registry,
@@ -319,10 +328,10 @@ def test_when_entities_is_none_then_return_all_fields(loaded_registry):
def test_when_entities_is_none_all_recognizers_loaded_then_return_all_fields(
nlp_engine,
spacy_nlp_engine,
):
analyze_engine = AnalyzerEngine(
registry=RecognizerRegistry(), nlp_engine=nlp_engine
registry=RecognizerRegistry(), nlp_engine=spacy_nlp_engine
)
threshold = 0
text = "My name is Sharon and I live in Seattle." "Domain: microsoft.com "
@@ -338,7 +347,7 @@ def test_when_entities_is_none_all_recognizers_loaded_then_return_all_fields(
def test_when_analyze_then_apptracer_has_value(
loaded_registry, unit_test_guid, nlp_engine
loaded_registry, unit_test_guid, spacy_nlp_engine
):
text = "My name is Bart Simpson, and Credit card: 4095-2609-9393-4932, my phone is 425 8829090" # noqa E501
language = "en"
@@ -348,7 +357,7 @@ def test_when_analyze_then_apptracer_has_value(
loaded_registry,
app_tracer=app_tracer_mock,
log_decision_process=True,
nlp_engine=nlp_engine,
nlp_engine=spacy_nlp_engine,
)
results = analyzer_engine_with_spacy.analyze(
correlation_id=unit_test_guid,
@@ -471,7 +480,7 @@ def test_when_get_supported_fields_then_return_all_languages(
def test_when_get_supported_fields_specific_language_then_return_single_result(
loaded_registry, unit_test_guid, nlp_engine
loaded_registry, unit_test_guid, spacy_nlp_engine
):
pattern = Pattern("rocket pattern", r"\W*(rocket)\W*", 0.8)
pattern_recognizer = PatternRecognizer(
@@ -481,7 +490,7 @@ def test_when_get_supported_fields_specific_language_then_return_single_result(
supported_language="ru",
)
analyzer = AnalyzerEngine(registry=loaded_registry, nlp_engine=nlp_engine)
analyzer = AnalyzerEngine(registry=loaded_registry, nlp_engine=spacy_nlp_engine)
analyzer.registry.add_recognizer(pattern_recognizer)
entities = analyzer.get_supported_entities(language="ru")
@@ -508,7 +517,7 @@ def test_when_get_recognizers_then_returns_supported_language():
assert len(response) == 1
def test_when_add_recognizer_then_also_outputs_others(nlp_engine):
def test_when_add_recognizer_then_also_outputs_others(spacy_nlp_engine):
pattern = Pattern("rocket pattern", r"\W*(rocket)\W*", 0.8)
pattern_recognizer = PatternRecognizer(
"ROCKET",
@@ -522,7 +531,7 @@ def test_when_add_recognizer_then_also_outputs_others(nlp_engine):
assert len(registry.recognizers) > 1
analyzer = AnalyzerEngine(registry=registry, nlp_engine=nlp_engine)
analyzer = AnalyzerEngine(registry=registry, nlp_engine=spacy_nlp_engine)
text = "Michael Jones has a rocket"
@@ -653,9 +662,9 @@ def test_entities_filter_for_ad_hoc_removes_recognizer(loaded_analyzer_engine):
assert "MR" not in [resp.entity_type for resp in responses2]
def test_ad_hoc_with_context_support_higher_confidence(nlp_engine, zip_code_recognizer):
def test_ad_hoc_with_context_support_higher_confidence(spacy_nlp_engine, zip_code_recognizer):
text = "Mr. John Smith's zip code is 10023"
analyzer_engine = AnalyzerEngine(nlp_engine=nlp_engine)
analyzer_engine = AnalyzerEngine(nlp_engine=spacy_nlp_engine)
responses1 = analyzer_engine.analyze(
text=text, language="en", ad_hoc_recognizers=[zip_code_recognizer]
@@ -687,7 +696,7 @@ def test_ad_hoc_when_no_other_recognizers_are_requested_returns_only_ad_hoc_resu
assert "ZIP" in [resp.entity_type for resp in responses]
def test_when_recognizer_doesnt_return_recognizer_name_no_exception(nlp_engine):
def test_when_recognizer_doesnt_return_recognizer_name_no_exception(spacy_nlp_engine):
class MockRecognizer1(EntityRecognizer, ABC):
def analyze(self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts):
return [RecognizerResult("TEST1", 10, 30, 0.5)]
@@ -704,7 +713,7 @@ def test_when_recognizer_doesnt_return_recognizer_name_no_exception(nlp_engine):
registry.add_recognizer(mock_recognizer1)
registry.add_recognizer(mock_recognizer2)
analyzer_engine = AnalyzerEngine(nlp_engine=nlp_engine, registry=registry)
analyzer_engine = AnalyzerEngine(nlp_engine=spacy_nlp_engine, registry=registry)
results = analyzer_engine.analyze("ABC", language="en")
assert len(results) == 2
@@ -736,7 +745,7 @@ def test_when_recognizer_doesnt_return_recognizer_name_no_exception(nlp_engine):
)
def test_when_recognizer_overrides_enhance_score_then_it_get_boosted_once(nlp_engine):
def test_when_recognizer_overrides_enhance_score_then_it_get_boosted_once(spacy_nlp_engine):
class MockRecognizer(EntityRecognizer, ABC):
def analyze(self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts):
return [
@@ -769,7 +778,7 @@ def test_when_recognizer_overrides_enhance_score_then_it_get_boosted_once(nlp_en
registry = RecognizerRegistry()
registry.add_recognizer(mock_recognizer)
analyzer_engine = AnalyzerEngine(nlp_engine=nlp_engine, registry=registry)
analyzer_engine = AnalyzerEngine(nlp_engine=spacy_nlp_engine, registry=registry)
recognizer_results = analyzer_engine.analyze("ABC", language="en")
assert len(recognizer_results) == 2
@@ -810,7 +819,7 @@ def test_when_recognizer_overrides_enhance_score_then_it_get_boosted_once(nlp_en
]
def test_when_multiple_nameless_recognizers_context_is_correct(nlp_engine):
def test_when_multiple_nameless_recognizers_context_is_correct(spacy_nlp_engine):
rocket_recognizer = PatternRecognizer(
supported_entity="ROCKET",
context=["cool"],
@@ -826,7 +835,7 @@ def test_when_multiple_nameless_recognizers_context_is_correct(nlp_engine):
registry.add_recognizer(rocket_recognizer)
registry.add_recognizer(rocket_recognizer2)
analyzer_engine = AnalyzerEngine(nlp_engine=nlp_engine, registry=registry)
analyzer_engine = AnalyzerEngine(nlp_engine=spacy_nlp_engine, registry=registry)
recognizer_results = analyzer_engine.analyze(
"I have a cool rocket and a fast missile.", language="en"
)
presidio-analyzer/tests/test_batch_analyzer_engine.py
+1 -1
View File
@@ -158,7 +158,7 @@ def test_analyze_dict_on_nested_dict(batch_analyzer_engine_simple):
key="key_a1",
value=nested_dict["key_a"]["key_a1"],
recognizer_results=[
RecognizerResult("PHONE_NUMBER", start=19, end=31, score=0.75)
RecognizerResult("PHONE_NUMBER", start=19, end=31, score=0.4)
],
)
],
presidio-analyzer/tests/test_context_support.py
+9 -14
View File
@@ -46,11 +46,6 @@ def recognizers_list(recognizers_map):
return rec_list
@pytest.fixture(scope="module")
def nlp_engine(nlp_engines):
return nlp_engines["spacy_en"]
@pytest.fixture(scope="module")
def dataset(recognizers_map):
"""Loads up a group of sentences with relevant context words and creates
@@ -95,7 +90,7 @@ def us_license_recognizer():
def test_when_text_with_aditional_context_lemma_based_context_enhancer_then_analysis_explanation_include_correct_supportive_context_word( # noqa: E501
nlp_engine, lemma_context, us_license_recognizer
spacy_nlp_engine, lemma_context, us_license_recognizer
):
"""This test checks that LemmaContextAwareEnhancer uses supportive context
word from analyze input as if it was in the text itself.
@@ -105,7 +100,7 @@ def test_when_text_with_aditional_context_lemma_based_context_enhancer_then_anal
return that word as supportive_context_word instead of other recognizer context word
"""
text = "John Smith license is AC432223"
nlp_artifacts = nlp_engine.process_text(text, "en")
nlp_artifacts = spacy_nlp_engine.process_text(text, "en")
recognizer_results = us_license_recognizer.analyze(text, nlp_artifacts)
results_without_additional_context = lemma_context.enhance_using_context(
text, recognizer_results, nlp_artifacts, [us_license_recognizer]
@@ -130,8 +125,8 @@ def test_when_text_with_aditional_context_lemma_based_context_enhancer_then_anal
)
def test_when_text_with_only_aditional_context_lemma_based_context_enhancer_then_analysis_explanation_include_correct_supportive_context_word( # noqa: E501
nlp_engine, lemma_context, us_license_recognizer
def test_when_text_with_only_additional_context_lemma_based_context_enhancer_then_analysis_explanation_include_correct_supportive_context_word( # noqa: E501
spacy_nlp_engine, lemma_context, us_license_recognizer
):
"""This test checks that LemmaContextAwareEnhancer uses supportive context
word from analyze input as if it was in the text itself but no other words apear
@@ -143,7 +138,7 @@ def test_when_text_with_only_aditional_context_lemma_based_context_enhancer_then
return that word as supportive_context_word and raise the score.
"""
text = "John Smith D.R is AC432223"
nlp_artifacts = nlp_engine.process_text(text, "en")
nlp_artifacts = spacy_nlp_engine.process_text(text, "en")
recognizer_results = us_license_recognizer.analyze(text, nlp_artifacts)
results_without_additional_context = lemma_context.enhance_using_context(
text, recognizer_results, nlp_artifacts, [us_license_recognizer]
@@ -171,11 +166,11 @@ def test_when_text_with_only_aditional_context_lemma_based_context_enhancer_then
def test_when_text_with_context_then_improves_score(
dataset, nlp_engine, mock_nlp_artifacts, lemma_context, recognizers_list
dataset, spacy_nlp_engine, mock_nlp_artifacts, lemma_context, recognizers_list
):
for item in dataset:
text, recognizer, entities = item
nlp_artifacts = nlp_engine.process_text(text, "en")
nlp_artifacts = spacy_nlp_engine.process_text(text, "en")
results_without_context = recognizer.analyze(text, entities, mock_nlp_artifacts)
results_with_context = recognizer.analyze(text, entities, nlp_artifacts)
@@ -194,7 +189,7 @@ def test_when_text_with_context_then_improves_score(
assert res_wo.score <= res_w.score
def test_when_context_custom_recognizer_then_succeed(nlp_engine, mock_nlp_artifacts):
def test_when_context_custom_recognizer_then_succeed(spacy_nlp_engine, mock_nlp_artifacts):
"""This test checks that a custom recognizer is also enhanced by context.
However this test also verifies a specific case in which the pattern also
@@ -211,7 +206,7 @@ def test_when_context_custom_recognizer_then_succeed(nlp_engine, mock_nlp_artifa
text = "hi, this is a cool ROCKET"
recognizer = rocket_recognizer
entities = ["ROCKET"]
nlp_artifacts = nlp_engine.process_text(text, "en")
nlp_artifacts = spacy_nlp_engine.process_text(text, "en")
results_without_context = recognizer.analyze(text, entities, mock_nlp_artifacts)
results_with_context = recognizer.analyze(text, entities, nlp_artifacts)
assert len(results_without_context) == len(results_with_context)
presidio-analyzer/tests/test_ner_model_configuration.py
+57
View File
@@ -0,0 +1,57 @@
from pathlib import Path
import pytest
import yaml
from presidio_analyzer.nlp_engine import NerModelConfiguration
@pytest.fixture(scope="module")
def ner_model_configuration_dict():
this_path = Path(__file__).parent.absolute()
conf_file = Path(this_path, "conf/test_transformers.yaml")
with open(conf_file) as f:
configuration_dict = yaml.safe_load(f)
return configuration_dict["ner_model_configuration"]
@pytest.mark.parametrize(
"key, original_value, expected_value",
[
("labels_to_ignore", [], []),
("labels_to_ignore", ["A", "B"], ["A", "B"]),
("aggregation_strategy", "X", "X"),
("alignment_mode", "Y", "Y"),
("stride", 51, 51),
("model_to_presidio_entity_mapping", {"A": "B"}, {"A": "B"}),
("low_score_entity_names", ["A", "C"], ["A", "C"]),
("low_confidence_score_multiplier", 12.0, 12.0),
],
)
def test_from_dict_happy_path(
ner_model_configuration_dict, key, original_value, expected_value
):
ner_model_configuration_dict[key] = original_value
result = NerModelConfiguration.from_dict(ner_model_configuration_dict)
assert result.to_dict()[key] == expected_value
@pytest.mark.parametrize(
"key, value",
[
("stride", []),
("stride", "X"),
("stride", None),
("alignment_mode", 5),
("alignment_mode", None),
("low_confidence_score_multiplier", "X"),
],
)
def test_from_dict_wrong_types(ner_model_configuration_dict, key, value):
new_config = ner_model_configuration_dict.copy()
new_config[key] = value
with pytest.raises(ValueError):
NerModelConfiguration.from_dict(new_config)
presidio-analyzer/tests/test_nlp_engine_provider.py
+122 -41
View File
@@ -1,4 +1,5 @@
from pathlib import Path
from typing import Dict
import pytest
import spacy
@@ -11,6 +12,52 @@ from presidio_analyzer.nlp_engine import (
from presidio_analyzer.nlp_engine.transformers_nlp_engine import TransformersNlpEngine
@pytest.fixture(scope="module")
def mock_he_model():
"""
Create an empty Hebrew spaCy pipeline and save it to disk.
So that it could be loaded using spacy.load()
"""
he = spacy.blank("he")
he.to_disk("he_test")
@pytest.fixture(scope="module")
def mock_bn_model():
"""
Create an empty Bengali spaCy pipeline and save it to disk.
So that it could be loaded using spacy.load()
"""
bn = spacy.blank("bn")
bn.to_disk("bn_test")
@pytest.fixture(scope="session")
def nlp_configuration_dict() -> Dict:
nlp_configuration = {
"lang_code": "en",
"model_name": {
"spacy": "en_core_web_lg",
"transformers": "StanfordAIMI/stanford-deidentifier-base",
},
}
return nlp_configuration
@pytest.fixture(scope="session")
def ner_model_configuration_dict() -> Dict:
ner_model_configuration = {
"nlp_engine_name": "transformers",
"aggregation_strategy": "simple",
"alignment_mode": "strict",
"low_score_entity_names": ["O"],
}
return ner_model_configuration
def test_when_create_nlp_engine__then_return_default_configuration():
provider = NlpEngineProvider()
engine = provider.create_engine()
@@ -18,7 +65,12 @@ def test_when_create_nlp_engine__then_return_default_configuration():
assert engine.nlp is not None
def test_when_create_nlp_engine_then_simple_config_succeeds(mock_he_model):
def test_when_create_nlp_engine_then_simple_config_succeeds(mocker, mock_he_model):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
nlp_configuration = {
"nlp_engine_name": "spacy",
"models": [{"lang_code": "he", "model_name": "he_test"}],
@@ -31,7 +83,14 @@ def test_when_create_nlp_engine_then_simple_config_succeeds(mock_he_model):
assert isinstance(engine.nlp["he"], spacy.lang.he.Hebrew)
def test_when_create_nlp_engine_then_two_models_succeeds(mock_he_model, mock_bn_model):
def test_when_create_nlp_engine_then_two_models_succeeds(
mocker, mock_he_model, mock_bn_model
):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
nlp_configuration = {
"nlp_engine_name": "spacy",
"models": [
@@ -51,7 +110,11 @@ def test_when_create_nlp_engine_then_two_models_succeeds(mock_he_model, mock_bn_
assert isinstance(engine.nlp["bn"], spacy.lang.bn.Bengali)
def test_when_create_nlp_engine_from_wrong_conf_then_fail():
def test_when_create_nlp_engine_from_wrong_conf_then_fail(mocker):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
with pytest.raises(OSError):
nlp_configuration = {
"nlp_engine_name": "spacy",
@@ -61,7 +124,11 @@ def test_when_create_nlp_engine_from_wrong_conf_then_fail():
provider.create_engine()
def test_when_unsupported_nlp_engine_then_fail():
def test_when_unsupported_nlp_engine_then_fail(mocker):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
with pytest.raises(ValueError) as e:
unsupported_engine_name = "not exists"
nlp_configuration = {
@@ -70,11 +137,17 @@ def test_when_unsupported_nlp_engine_then_fail():
}
provider = NlpEngineProvider(nlp_configuration=nlp_configuration)
provider.create_engine()
assert (f"NLP engine '{unsupported_engine_name}' is not available. "
"Make sure you have all required packages installed") == e.value.args[0]
assert (
f"NLP engine '{unsupported_engine_name}' is not available. "
"Make sure you have all required packages installed"
) == e.value.args[0]
def test_when_read_test_nlp_conf_file_then_returns_spacy_nlp_engine():
def test_when_read_test_nlp_conf_file_then_returns_spacy_nlp_engine(mocker):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
test_conf_file = Path(Path(__file__).parent, "conf", "test.yaml")
provider = NlpEngineProvider(conf_file=test_conf_file)
nlp_engine = provider.create_engine()
@@ -93,7 +166,12 @@ def test_when_read_test_nlp_conf_file_then_returns_stanza_nlp_engine():
assert nlp_engine.nlp is not None
def test_when_both_conf_and_config_then_fail():
def test_when_both_conf_and_config_then_fail(mocker):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
nlp_configuration = {
"nlp_engine_name": "spacy",
"models": [{"lang_code": "he", "model_name": "he_test"}],
@@ -105,15 +183,19 @@ def test_when_both_conf_and_config_then_fail():
@pytest.mark.skip_engine("transformers_en")
def test_when_create_transformers_nlp_engine_then_succeeds():
nlp_configuration={
def test_when_create_transformers_nlp_engine_then_succeeds(mocker):
mocker.patch(
"presidio_analyzer.nlp_engine.TransformersNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
nlp_configuration = {
"nlp_engine_name": "transformers",
"models": [
{
"lang_code": "en",
"model_name": {
"spacy": "en_core_web_sm",
"transformers": "dslim/bert-base-NER",
"spacy": "en_core_web_lg",
"transformers": "StanfordAIMI/stanford-deidentifier-base",
},
}
],
@@ -124,8 +206,15 @@ def test_when_create_transformers_nlp_engine_then_succeeds():
assert isinstance(engine.nlp["en"], spacy.lang.en.English)
def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_not_dict_then_fail():
nlp_configuration={
@pytest.mark.skip_engine("transformers_en")
def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_not_dict_then_fail(
mocker,
):
mocker.patch(
"presidio_analyzer.nlp_engine.SpacyNlpEngine._download_spacy_model_if_needed",
return_value=None,
)
nlp_configuration = {
"nlp_engine_name": "transformers",
"models": [
{
@@ -138,35 +227,27 @@ def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_not
NlpEngineProvider(nlp_configuration=nlp_configuration).create_engine()
def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_keys_not_include_spacy_then_fail():
nlp_configuration={
"nlp_engine_name": "transformers",
"models": [
{
"lang_code": "en",
"model_name": { # keys should contain transformers and spacy
"not_spacy": "en_core_web_sm",
"transformers": "dslim/bert-base-NER",
},
}
],
}
def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_keys_not_include_spacy_then_fail(
nlp_configuration_dict,
):
nlp_configuration = nlp_configuration_dict.copy()
del nlp_configuration["model_name"]["spacy"]
nlp_configuration["model_name"]["not_spacy"] = "ERROR"
with pytest.raises(ValueError):
NlpEngineProvider(nlp_configuration=nlp_configuration).create_engine()
def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_keys_not_include_transformers_then_fail():
nlp_configuration={
"nlp_engine_name": "transformers",
"models": [
{
"lang_code": "en",
"model_name": { # keys should contain transformers and spacy
"spacy": "en_core_web_sm",
"not_transformers": "dslim/bert-base-NER",
},
}
],
}
def test_when_create_transformers_nlp_engine_from_wrong_conf_with_model_name_keys_not_include_transformers_then_fail(
nlp_configuration_dict,
):
nlp_configuration = nlp_configuration_dict.copy()
del nlp_configuration["model_name"]["transformers"]
nlp_configuration["model_name"]["not_transformers"] = "ERROR"
with pytest.raises(ValueError):
NlpEngineProvider(nlp_configuration=nlp_configuration).create_engine()
NlpEngineProvider(nlp_configuration=nlp_configuration).create_engine()
def test_nlp_engine_provider_init_through_nlp_engine_configuration():
engine = NlpEngineProvider().create_engine()
assert isinstance(engine, SpacyNlpEngine)
assert engine.engine_name == "spacy"
presidio-analyzer/tests/test_phone_recognizer.py
+2 -6
View File
@@ -9,10 +9,6 @@ def recognizer():
return PhoneRecognizer()
@pytest.fixture(scope="module")
def nlp_engine(nlp_engines):
return nlp_engines["spacy_en"]
@pytest.mark.parametrize(
"text, expected_len, entities, expected_positions, score",
@@ -32,7 +28,7 @@ def nlp_engine(nlp_engines):
],
)
def test_when_all_phones_then_succeed(
nlp_engine,
spacy_nlp_engine,
text,
expected_len,
entities,
@@ -40,7 +36,7 @@ def test_when_all_phones_then_succeed(
score,
recognizer,
):
nlp_artifacts = nlp_engine.process_text(text, "en")
nlp_artifacts = spacy_nlp_engine.process_text(text, "en")
results = recognizer.analyze(text, entities, nlp_artifacts=nlp_artifacts)
assert len(results) == expected_len
for i, (res, (st_pos, fn_pos)) in enumerate(zip(results, expected_positions)):
presidio-analyzer/tests/test_spacy_nlp_engine.py
+26 -5
View File
@@ -1,18 +1,20 @@
from typing import Iterator
import pytest
def test_simple_process_text(nlp_engine):
from presidio_analyzer.nlp_engine import SpacyNlpEngine
nlp_artifacts = nlp_engine.process_text("simple text", language="en")
def test_simple_process_text(spacy_nlp_engine):
nlp_artifacts = spacy_nlp_engine.process_text("simple text", language="en")
assert len(nlp_artifacts.tokens) == 2
assert not nlp_artifacts.entities
assert nlp_artifacts.lemmas[0] == "simple"
assert nlp_artifacts.lemmas[1] == "text"
def test_process_batch_strings(nlp_engine):
nlp_artifacts_batch = nlp_engine.process_batch(
def test_process_batch_strings(spacy_nlp_engine):
nlp_artifacts_batch = spacy_nlp_engine.process_batch(
["simple text", "simple text"], language="en"
)
assert isinstance(nlp_artifacts_batch, Iterator)
@@ -21,3 +23,22 @@ def test_process_batch_strings(nlp_engine):
for text, nlp_artifacts in nlp_artifacts_batch:
assert text == "simple text"
assert len(nlp_artifacts.tokens) == 2
def test_nlp_not_loaded_value_error():
unloaded_spacy_nlp = SpacyNlpEngine()
with pytest.raises(ValueError):
unloaded_spacy_nlp.process_text(
"This should fail as the NLP model isn't loaded", language="en"
)
def test_validate_model_params_missing_fields():
model = {"lang_code": "en", "model_name": "en_core_web_lg"}
for key in model.keys():
new_model = model.copy()
del new_model[key]
with pytest.raises(ValueError):
SpacyNlpEngine._validate_model_params(new_model)
presidio-analyzer/tests/test_spacy_recognizer.py
+11 -4
View File
@@ -1,5 +1,6 @@
import pytest
from presidio_analyzer.predefined_recognizers import SpacyRecognizer
from tests import assert_result_within_score_range
@@ -47,13 +48,13 @@ def test_when_using_spacy_then_all_spacy_result_found(
expected_len,
expected_positions,
entity_num,
nlp_engine,
spacy_nlp_engine,
nlp_recognizer,
entities,
ner_strength,
max_score,
):
results = prepare_and_analyze(nlp_engine, nlp_recognizer, text, entities)
results = prepare_and_analyze(spacy_nlp_engine, nlp_recognizer, text, entities)
assert len(results) == expected_len
entity_to_check = entities[entity_num]
for res, (st_pos, fn_pos) in zip(results, expected_positions):
@@ -63,10 +64,10 @@ def test_when_using_spacy_then_all_spacy_result_found(
def test_when_person_in_text_then_person_full_name_complex_found(
nlp_engine, nlp_recognizer, entities
spacy_nlp_engine, nlp_recognizer, entities
):
text = "William Bill Alexander"
results = prepare_and_analyze(nlp_engine, nlp_recognizer, text, entities)
results = prepare_and_analyze(spacy_nlp_engine, nlp_recognizer, text, entities)
assert len(results) > 0
@@ -77,3 +78,9 @@ def test_when_person_in_text_then_person_full_name_complex_found(
covered_text += text[sl]
assert len(text) - len(covered_text) < 5
def test_analyze_no_nlp_artifacts():
spacy_recognizer = SpacyRecognizer()
res = spacy_recognizer.analyze(text="text", nlp_artifacts=None, entities=["PERSON"])
assert len(res) == 0
presidio-analyzer/tests/test_stanza_recognizer.py
+11 -7
View File
@@ -10,8 +10,11 @@ def entities():
@pytest.mark.skip_engine("stanza_en")
@pytest.fixture(scope="module")
def nlp_engine(nlp_engines):
return nlp_engines.get("stanza_en", None)
def stanza_nlp_engine(nlp_engines):
nlp_engine = nlp_engines.get("stanza_en", None)
if nlp_engine:
nlp_engine.load()
return nlp_engine
@pytest.mark.skip_engine("stanza_en")
@@ -21,6 +24,7 @@ def nlp_recognizer(nlp_recognizers):
def prepare_and_analyze(nlp, recognizer, text, ents):
nlp.load()
nlp_artifacts = nlp.process_text(text, "en")
results = recognizer.analyze(text, ents, nlp_artifacts)
return results
@@ -50,18 +54,18 @@ def prepare_and_analyze(nlp, recognizer, text, ents):
# fmt: on
],
)
def test_when_using_stanze_then_all_stanza_result_correct(
def test_when_using_stanza_then_all_stanza_result_correct(
text,
expected_len,
expected_positions,
entity_num,
nlp_engine,
stanza_nlp_engine,
nlp_recognizer,
entities,
ner_strength,
max_score,
):
results = prepare_and_analyze(nlp_engine, nlp_recognizer, text, entities)
results = prepare_and_analyze(stanza_nlp_engine, nlp_recognizer, text, entities)
assert len(results) == expected_len
entity_to_check = entities[entity_num]
for res, (st_pos, fn_pos) in zip(results, expected_positions):
@@ -72,10 +76,10 @@ def test_when_using_stanze_then_all_stanza_result_correct(
@pytest.mark.skip_engine("stanza_en")
def test_when_person_in_text_then_person_full_name_complex_found(
nlp_engine, nlp_recognizer, entities
spacy_nlp_engine, nlp_recognizer, entities
):
text = "Richard (Rick) C. Henderson"
results = prepare_and_analyze(nlp_engine, nlp_recognizer, text, entities)
results = prepare_and_analyze(spacy_nlp_engine, nlp_recognizer, text, entities)
assert len(results) > 0
presidio-analyzer/tests/test_transformers_nlp_engine.py
+44
View File
@@ -0,0 +1,44 @@
import pytest
from presidio_analyzer.nlp_engine import TransformersNlpEngine
def test_default_models():
engine = TransformersNlpEngine()
assert len(engine.models) > 0
assert engine.models[0]["lang_code"] == "en"
assert isinstance(engine.models[0]["model_name"], dict)
def test_validate_model_params_happy_path():
model = {
"lang_code": "en",
"model_name": {
"spacy": "en_core_web_sm",
"transformers": "obi/deid_roberta_i2b2",
},
}
TransformersNlpEngine._validate_model_params(model)
@pytest.mark.parametrize(
"key",
[("lang_code"), ("model_name"), ("model_name.spacy"), ("model_name.transformers")],
)
def test_validate_model_params_missing_fields(key):
model = {
"lang_code": "en",
"model_name": {
"spacy": "en_core_web_sm",
"transformers": "obi/deid_roberta_i2b2",
},
}
keys = key.split(".")
if len(keys) == 1:
del model[keys[0]]
else:
del model[keys[0]][keys[1]]
with pytest.raises(ValueError):
TransformersNlpEngine._validate_model_params(model)
presidio-analyzer/tests/test_transformers_recognizer.py
+98
View File
@@ -0,0 +1,98 @@
import pytest
from tests import assert_result_within_score_range
@pytest.fixture(scope="module")
def entities():
return ["PERSON", "DATE_TIME"]
@pytest.mark.skip_engine("transformers_en")
@pytest.fixture(scope="module")
def nlp_recognizer(nlp_recognizers):
return nlp_recognizers.get("transformers", None)
@pytest.mark.skip_engine("transformers_en")
@pytest.fixture(scope="module")
def nlp_engine(nlp_engines):
nlp_engine = nlp_engines.get("transformers_en", None)
if nlp_engine:
nlp_engine.load()
return nlp_engine
def prepare_and_analyze(nlp, recognizer, text, entities):
nlp.load()
nlp_artifacts = nlp.process_text(text, "en")
results = recognizer.analyze(text, entities, nlp_artifacts)
return results
@pytest.mark.skip_engine("transformers_en")
@pytest.mark.parametrize(
"text, expected_len, expected_positions, entity_num",
[
# fmt: off
# Test PERSON entity
("my name is Dan", 1, ((11, 14),), 0),
("Dan Tailor", 1, ((0, 10),), 0),
("John Oliver is a comedian.", 1, ((0, 11),), 0),
("Richard Milhous Nixon", 1, ((0, 21),), 0),
("Richard M. Nixon", 1, ((0, 16),), 0),
("Dan May has a bank account.", 1, ((0, 7),), 0),
("his name is Mr. May", 1, ((12, 19),), 0),
("They call me Mr. May", 1, ((13, 20),), 0),
# Test DATE_TIME Entity
("year 1972", 1, ((0, 9),), 1),
("I bought my car in 1972.", 1, ((19, 23),), 1),
("I bought my car in May.", 1, ((19, 22),), 1),
("May 1st", 1, ((0, 7),), 1),
("May 1st, 1977", 1, ((0, 13),), 1),
("I bought my car on May 1st, 1977", 1, ((19, 32),), 1),
# fmt: on
],
)
def test_when_using_transformers_then_all_transformers_result_correct(
text,
expected_len,
expected_positions,
entity_num,
nlp_engine,
nlp_recognizer,
entities,
min_score,
max_score,
):
results = prepare_and_analyze(nlp_engine, nlp_recognizer, text, entities)
assert len(results) == expected_len
entity_to_check = entities[entity_num]
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result_within_score_range(
result=res,
expected_entity_type=entity_to_check,
expected_start=st_pos,
expected_end=fn_pos,
expected_score_min=min_score,
expected_score_max=max_score,
)
@pytest.mark.skip_engine("transformers_en")
def test_when_person_in_text_then_person_full_name_complex_found(
nlp_engine, nlp_recognizer, entities
):
text = "Richard (Rick) C. Henderson"
results = prepare_and_analyze(nlp_engine, nlp_recognizer, text, entities)
assert len(results) > 0
# check that most of the text is covered
covered_text = ""
for result in results:
sl = slice(result.start, result.end)
covered_text += text[sl]
assert len(text) - len(covered_text) < 5