* analyzer - multiple languages and nlp engines
Initially this was my attempt to use stanza, which is an nlp engine by
Stanford. But generally, it's an update to allow for one to add NLP
engines and custom recognizers more easily. Specifically, I
standardized the format of the recognizers, removed use of global
variables when possible, and removed a lot of hard-coding of defaults.
I am thinking of using presidio for several non-english projects at work
and these are several of the changes that I made.
Below is a list of the changes in list form:
* make spacy and/or stanza optional
* remove requirement of en_core_web_lg from install
* allow predefined recognizers to take parameters
* this allows for easily using these as non-english recognizers
* create config files for different NLP engines
* create tests for stanza
* make all spacy and stanza tests optional
* create a Dockerfile for an anaconda-based image
* pytorch is built with MKL and is much faster on cpu from conda
* completely rewrote the IBAN recognizer
* the current version only recognizes IBANs if they are the entirety
of the string. This version will find IBANs in sentences.
* fixed some tests
* created a `run.sh` file, so just run dockers without rebuilding them
"Breaking" Changes:
* I would like to use [black](https://github.com/psf/black), but it's
not super friendly with pylint. My suggestion is to drop pylint and
use black instead.
* Default spacy model is `en` rather than `en_core_web_lg` and no spacy
models are downloaded by default. The idea is to let the user choose
which models they want. For non-english users, it saves a lot of time
at installation because you don't need to install the large spacy
model that you aren't using.
Signed-off-by: David Pollack <d.pollack@solvemate.com>
* spacy required, spacy-stanza, update tests
* made spacy required
* using spacy-stanza for stanza models
* refactor tests to use pytest
* make one test reliant on big model optional
* refactor tests to pytest
All tests have been refactored to use pytest. Previously, there was a
mix of unittest, pytest and miscellaneous global initializations. This
commit moves everything to pytest. There is now extensive use of
fixtures instead of global variables and parametrized tests instead of
duplicated code for each test. The major difference is that
parametrized tests are not individually named.
* changes based on PR comments
* fixes to Dockerfiles
* remove sys.path.append
* fix pipeline errors (i.e. install spacy model)
this installs the big spacy model by default in the Docker and the Azure
pipeline.
* fix rebase errors
* use Pattern class
* update docs
* use PresidioLogger
* linting fixes
* move imports to top level
* edits based on PR-review
* add documentation and doc strings
* change yaml field names to be more logical
* fix pipelines based on PR comments
* python and tag
* full version
* pythonver
* .
* .
* version
* continueOnError
* .
* .
* .
* indent
* pyton version mark unused tasks
* only golangdockerfile
* python some back
* git diff or
* or
* or
* change python steps.
* .
* version again
* .
* .
* .
* .
* version
* script
* merge python
* '
* new build_deps
* debug
* debug
* .
* .
* debug
* .
* .
* .
* .
* publish condition
* .
* .
* fixes
* .
* pull right label
* fix labels
* .
* .
* cache from for golang base
* remove temp code
* paralllel deps
* fix vars
* .
* change from source to .
* file docs
* build release docs
* .
* .
* new order
* SetBuildDep
* fixes
* .
* .
* ץ
* depends on for base container
* fix
* .
* fix golang
* fix branch names
* test
* display name
* version
* change analyzer folder name only
* add credscan
* update docs
* change package name
* fix imports
* variable group name
* lint errs
* fix
* lint
* adding tests dockerfile
* .
* fixes PR
* golang env
* w display name
* .
* fix
* names first
* indent
* .
* .
* fix all names and inline script
* fix targettype
* .
* .
* golang env
* .
* .
* fix last PR
* import
* .
* build and use tests container
* .
* fix python code
* functional tests
* try separate twine in CI
* fix dockerfile
* with template
* indent
* .
* remove dep ensure
* indent
* no working folder
* remove TTY from docker
* fix main.py
* try publish
* fix if
* fix
* names
* publish functional test results
* fix authenticate to twine
* fix name
* display name
* smaller dockerfile
* colon
* update to v1
* parallel runs
* without param
* wait for tasks
* fix extra s
* wait by name
* only wait for analyzer
* temp upload wheel without pararm
* try with home
* add verbose
* wait for analyzer
* opt out w. TODO
* output test results
* .
* export results
* test
* with param upload only
* with all
* fix sc
* without wheel
* start build with security analysis.
* move to right place
* update makefile
* fix makefile
* fix makefile
* only python
* only python
* fix publish
* .
* dist
* fix dist
* artifact_is_ado
* fix dist
* presidio-analyzer*
* maintain wheel
* copyt wheel
* PythonStage
* .
* fix
* FIX
* .
* dont build container
* fix
* fix get wheel
* without copy
* mv
* display name
* fix dist publish
* with all
* with all
* remove hard coded values
* maintain go tests in local environment
* fix test results file
* run presidio makefile
* workingDirectory
* remove registry when local build
* with registry
* fix indent
* tagging base image
* fix python env and go test folder
* fix
* dep ensure
* install dep
* with doc and pr stage only
* with stages
* with docs and stages
* set fail on error
* with tesseract-ocr
* sudo
* without sudo
* with sudo on install and build
* try fix build
* copy after dep ensure
* shopt
* .
* try with export
* remove extra files
* new order
* new order
* test
* .
* .
* go get
* with ls
* ls
* debug locally
* debug
* fix
* debug
* debug
* remove go get
* mod vendor
* dep specific
* remove mod flag
* echo
* debugs
* unset goroot
* .
* with gobin
* fix dep
* without sudo
* ocr dev
* libtesseract-dev
* remove all debug
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Fixed typo on string #74 (#103)
* Default transformation (#94)
* This commit introduce the support of default transformation for undeclared fields in the anonymizer, which can be overrided by the user if supplied another default within the anonymizer template.
* Streams bug fixes (#92)
* Deployment script (#95)
New simplified deployment scripts + unified tags
* Analyzer redesign + supporting custom recognizers
This commit is the first part of the redesign of the analyzer service, and contains the following:
1. Separates spacy and recognizers logic to different files.
2. Implements a base class for all the recognizers,(which in future custom recognizers will inherit)
3. Moves the analyzer logic from main to analyzer_engine class
4. Removes the detected text from the analyzer result.
Future commits will contain the following:
1. Dynamic loading of the pre-defined recognizers. [link](https://dev.azure.com/csedevil/Presidio-internal/_sprints/taskboard/Presidio%20Crew/Presidio-internal/02%20-%20Testable%20custom%20models)
2. Add new pattern recognizer via api call, [work item](https://dev.azure.com/csedevil/Presidio-internal/_sprints/taskboard/Presidio%20Crew/Presidio-internal/02%20-%20Testable%20custom%20models)
3. Improve remove duplicates logic [bug](https://dev.azure.com/csedevil/Presidio-internal/_workitems/edit/597) and [bug](https://dev.azure.com/csedevil/Presidio-internal/_workitems/edit/596/)
4. Re-support context model. [work item](https://dev.azure.com/csedevil/Presidio-internal/_sprints/taskboard/Presidio%20Crew/Presidio-internal/02%20-%20Testable%20custom%20models)
Current Design:
* Presidio support for language code in template (#98)
Configure a language code on the request level and not the field level. All requests should have one language
* Fix Bug #604 - Refactor test assertions + some pylint fixes (#100)
* Fix Bug #604 - Refactor test assertions + some pylint fixes
* fix spaces in spacy_recognizer.py
* Update test_spacy_recognizer.py
Fix PR comment regarding Bug 617
* fixed bug: changed 'push' to 'pull' in Makefile (#102)
* Bug666 - Adding pylint to the Analyzer microservice (#105)
* Adding pylint for the analyzer microservice
* Ll bug610 - Fix bug in IBAN recognizers + additional test fixes (#101)
* Fix Bug 610: iban recognizer and fix additional tests
* Ignoring 0 score patterns, removed predefined 0 score patterns from code.
* All fields (#107)
support for requesting all fields (entities) and language refactor
* New functionality: Support custom pattern recognizers (#104)
Adding a new service for persisting recognizers
Adding API support for adding, removing and listing recognizers
Analyzer service now calls the recognizers store to get new recognizers to be used during analysis
* Entities list for all fields true (#111)
Bug fix with all_fields = True
* (Re)Enable context support in analyzer (#114)
This PR introduces estimation of surrounding words (aka context). It also introduces generic NLP classes for accessing metadata extracted by an NLP engine (specifically tokens, lemmas, NER, stopwords and punctuation). SpacyRecognizer no longer runs spacy but only processes the metadata processed by the NLP Engine. In the current implementation, Spacy is the implementation of the NLP engine.
* Add public path to makefile (#121)
* updated docker file to support correct spacy version
* added public path for push release
* Update Dockerfile.python.deps
* replaced registry with public mcr
* updated docs and samples based on new changes in dev (#120)
* updated docs and samples based on new changes in dev
* updated text change suggestion
* reverted registry change
* a post-upgrade e2e test (#112)
Runs e2e tests on the newly deployed code
* Build deps - adding build number to python&golang dependency containers (#122)
* adding presidio-dependency images versioning
* updating installation notes with dependency label
* add CI as code
* presidio deps ci yaml file
* add triggers to deps CI yaml
* separate CIs for golang and python
* adding presidio-dependency images versioning
* updating installation notes with dependency label
* Bug fixes - based on bugbash (#126)
* Bug 906 - CONTEXT_SUFFIX_COUNT is not used and PREFIX is used instead
* Bug 909 - Context not working with upper case / mixed case
* Bug 908 - support context with substrings
* Bug 907 - Context window size is off by 1 index
* Revert yaml (#137)
* Revert "Build deps - adding build number to python&golang dependency containers (#122)"
This reverts commit c054770582.
* Change deployment script to mcr (#130)
* updated docker file to support correct spacy version
* changed registry to mcr
* fixed the mcr url
* Updated readme and documentation(#125)
* Update README.MD
* Bug 911 fix (#131)
* Bug 911 fix
When the persistent recognizers store is empty there should be no error log line when requesting for the hash value
* changed genproto branch to master.
* Updating genproto to master branch
* Updated CI badge for new Presidio-CI pipeline
* bug fix - mcr latest tag was not correct
* adding presidio-dependency images versioning
* updating installation notes with dependency label
* add CI as code
* presidio deps ci yaml file
* add triggers to deps CI yaml
* separate CIs for golang and python
* adding presidio-dependency images versioning
* updating installation notes with dependency label
