Commit Graph

47 Commits

Author SHA1 Message Date
Omri Mendels
dda4444fda Updates to the TransformersRecognizer sample (#1084) 2023-06-04 12:14:18 +03:00
Omri Mendels
3e84d9ef25 Updated demo website (#1078) 2023-06-01 16:53:03 +03:00
Nava Vaisman Levy
481811af11 Add batch anonymization (#993) 2023-05-01 09:18:00 +03:00
Paulo Costa
24a76a8f44 Added "keep" anonymizer (#1062) 2023-04-30 10:25:00 +03:00
Omri Mendels
61a5405611 New demo app for Presidio, based on Streamlit (#1054) 2023-04-24 14:35:03 +03:00
Omri Mendels
60911edf16 GPT-3 based synthetic data generation (#1051) 2023-04-16 12:41:36 +03:00
Andy
07b854dd7a Adding a sample for identifying PII in a PDF (#1023) 2023-01-30 15:46:36 +02:00
Nile Wilson
b0430cabbb Add evaluation code for DICOM de-identification (#979)
* Initial commit with old paths (will need to fix)

* Fixing paths

* Adding in notebook and updating documentation

* Adding in WIP notebook and docs

* Adding example notebook

* Clearing some output to reduce file size

* Linting fix v1

* Linting v2

* Add rounding to integration test to account for minor differences in real testing

* Fixing typo with wrong variable name

* Removing check for exact or rounded conf

* Updating logic for comparing ocr_results

* Minor fixes on DICOM verify code based on PR comments

* Changing results and gt format from dictionary to list of dictionaries

* Updating notebook

* Updating docstring format

* Linting fix

* Changing fixture scope for mock_engine

* Update docs/image-redactor/evaluating_dicom_redaction.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Docstring updates

* Improving efficiency of _remove_duplicate_entities

* Add handling for divide by zero in recall and precision calculations

* Making precision and recall calculation functions public

* Moving common fixtures to conftest

* PR comments

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
2023-01-05 08:01:46 -05:00
Nava Vaisman Levy
8993be09e9 fix access to start end positions (#994) 2023-01-03 14:50:45 +02:00
Nava Vaisman Levy
aab4d20ee7 Sample - analyze and anonymize csv file (#990) 2022-12-29 14:59:06 +02:00
Omri Mendels
77b86ce01d Rename transformer_recognizer.md to index.md (#984) 2022-12-19 14:18:27 +02:00
Omri Mendels
b0a4c72d28 Update example_text_analytics_recognizer.py (#981) 2022-12-19 12:28:23 +02:00
melmatlis
2107f051c6 add revised example for transformer recognizer (#962)
* add revised example for transformer recognizer

* update documentation index to point to new example

* Update docs/samples/python/transformers_recognizer/transformer_recognizer.py

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* linting, docstrings, readibility

* externalize text splitting parameters

* document class input variables

* add documentaiton to the sample

* Update docs/samples/python/transformers_recognizer/transformer_recognizer.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Update docs/samples/python/transformers_recognizer/transformer_recognizer.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Update docs/samples/python/transformers_recognizer/transformer_recognizer.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* update documentation

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
2022-12-19 10:18:40 +02:00
Nile Wilson
2730c6c00b Adding DICOM image redacting capability to presidio-image-redactor module (#960)
* Adding DicomImageRedactorEngine class without test data for now

* Updating and putting placeholder in package doc

* Updating docs

* Moving presidio-dicom-image-redactor module into presidio-image-redactor module as an extension

* Updating docs to reflect refactor from presidio-dicom-image-redactor module to extension of presidio-image-redactor module

* Docs fixes continued

* Reflecting move from new module to updating existing module

* Updating requirements

* Linting fixes

* Linting fixes. Function and variable name casing

* Fixing getting started code blocks

* Updating image redactor version

* Fix minor typo: phi -> PHI in comment

* Update presidio-image-redactor/presidio_image_redactor/utils/dicom_image_redact_utils.py

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Update presidio-image-redactor/presidio_image_redactor/utils/dicom_image_redact_utils.py

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Fixing new typing adding added from PR comment commit

* Moving all utils into the DicomImageRedactorEngine class

* Fixing integration test now that utils have been moved

* Moving to ad-hoc recognizer approach

* Updating redact to have similar format as in ImageRedactorEngine. Moved existing logic for redact into redact_from_file

* Updating requirements

* Updating docs with new getting started code

* Versioning and changelog updates

* Adding in data with citations

* Linting fixes

* Minor change: from multiple list append to list extend

* Adding pytest-mock to requirements

* Fixing paths to test data

* Making file extension search more flexible and work with both Windows and Unix systems

* Changing docstring style to match rest of repo

* Making statement about Tesseract version more general in one other doc (was caught in another earlier)

* Changing box_color_setting to fill for consistency

* Updating changelog based on PR comments

* Update docs/image-redactor/index.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Updating docs

* Adding **kwargs support and addressing minor comments

* Adjusting with clause duration

* Adding note about custom ImageAnalyzerEngine support

* Changing minor comment in docstring

* Splitting redact_from_file into redact_from_file and redact_from_directory

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
2022-12-13 08:47:30 -05:00
Matthew
fa9a3703cc Add missing comma (#946) 2022-11-15 14:34:07 +02:00
Eran
fb01d2374e Update index.md (#897) 2022-07-21 10:52:51 +03:00
Akshara Ramakrishnan
72d225c5f8 added allow list functionality to analyzer engine (#865) 2022-06-19 16:16:57 +03:00
Omri Mendels
f3519e78c8 Batch analyze - Support for lists and dicts (partial) (#878) 2022-06-19 10:35:05 +03:00
Omri Mendels
b4b1b97f98 Added sample notebook for anonymizing known values (#864) 2022-06-13 11:59:11 +03:00
Eran
63ca86f819 Add transformers recognizer sample (#866) 2022-06-08 17:20:51 +03:00
Guy Bertental
307d28563e Feature/custom entity recognizer context enhancement (#842) 2022-03-08 17:07:39 +02:00
Omri Mendels
33a580058f Linting/formatting samples (#827) 2022-02-15 22:16:35 +02:00
Ignasi Domingo Gonzalez
827358bd3f Including entity tag equivalences with Presidio (#818) 2022-01-23 11:35:31 +02:00
Ignasi Domingo Gonzalez
9ee7997cb2 Create Presidio_with_Flair_as_a_3rd_party_recognizer (#810) 2021-12-28 19:03:52 +02:00
Omri Mendels
4b0baec8f3 Documentation for creating a Streamlit demo app (#809) 2021-12-23 20:02:21 +02:00
Sharon Hart
cedcb4cb4c Remove AnonymizerResult in favor of OperatorResult (#795)
* Remove AnonymizerResult in favor of OperatorResult

* fix test

* documentation changes + align rest api

* operator_name

Co-authored-by: sharon <sharon.hart@microsoft.com>
2021-11-16 11:17:07 +02:00
Sharon Hart
88a55496ee Add Encryption and Decryption sample (#791)
* Add Encryption and Decryption sample

* Add standalone decryption

Co-authored-by: sharon <sharon.hart@microsoft.com>
2021-11-15 12:11:59 +02:00
Omri Mendels
31e8675930 Changes to Anonymizer packages/imports + added repr to classes. (#763)
* modules changes, doc changes

* updates on notebook and e2e

* backward compatibility of OperatorConfig

* revert file structure

* revert customize_presidio_analyzer, doesn't belong in this PR

* update to spark notebook

* updates to notebook

* Update getting_entity_values.ipynb

* Update getting_entity_values.ipynb

* grouped imports

* Update test_context_support.py
2021-09-26 14:49:48 +03:00
Omri Mendels
67cb019ff0 Two new samples + docs refactoring (#744) 2021-07-14 12:17:35 +03:00
Omri Mendels
36ee232d70 Sample notebook for remote recognizer (using Text Analytics) (#684) 2021-05-05 11:55:10 +03:00
Omri Mendels
d5d9551d57 Ad-hoc recognizers (#677) 2021-05-05 10:18:01 +03:00
Nava Vaisman Levy
cc9a6012ca Custom anonymizer with lambda type (#654) 2021-05-04 10:07:28 +03:00
Omri Mendels
1ac45f89ae Some changes and bug fixes in our docs (#683) 2021-05-03 11:01:39 +03:00
Nava Vaisman Levy
d8c7d2e74c Documentation code and samples regression test pipeline (#679) 2021-05-03 09:56:12 +03:00
Sharon Hart
bdd2bd6c53 Text Analytics Integration Sample (#678)
* Text Analytics Integration Sample

* Load file, change categories

* change filename

* Add md docs

* Move docs to sample

* Rephrased docs, add to recognizers in the example

Co-authored-by: sharon <sharon.hart@microsoft.com>
2021-04-29 11:34:56 +03:00
Shiran Rubin
3afec0c958 Poc deanonymize docs (#652)
* Add decryption mechanism.
On purpose we separate it from the anonymizer/

* Fix tests

* Remove unneeded params
fix docs and tests

* Add decryption engine to match the anonymizer engine.
Refactor code so the text builder and text manipulator will be mutual.

* Change engine result to be unified with different items inheriting from result item.

* Fix decrypt elements and implmentations

* Add hints

* Fix naming

* Change manipulators to operators.

* Change manipulators to operators.

* Move initialization of text manipulator
Change the analyzer results json to receive the list and not the entire content.
Fix tests

* Move getting operators classes into the operator factory.

* Fix entities.
Make sure all engines can use the same entities by using factories, builders and inheritance more.

* revert unneeded change

* Code cleanup

* Add tests and clean up code.

* Move anonymizer configuration to inherit from operator metadata and add decryptor config

* More cleanup

* More cleanup

* revert

* cleanup

* Fix imports and renaming

* Start working on linting.

* Start working on linting.

* add tests

* add tests

* add tests

* add tests

* remove unneeded formating.

* fix linting

* try to understand why it fails in pipe

* try to understand why it fails in pipe

* Fix equal check for recognizer result

* revert change for test investigation

* fix imports and add test

* fix dictionary

* add test for dictionary fix

* Fix example

* Fix E2E entities

* Fix PR comments

* Fix PR comments

* Fix PR comments

* Fix PR comments

* Fix PR comments

* Fix PR comments - use comprehensions

* Fix PR comments - Change names and move classes accordingly.

* fix linting

* fix linting

* use dictionary comprehensions

* Fix documentation and nameing.

* Fix documentation and nameing.

* remove unused code.

* PR comments

* fix linting

* PR review -> be background compatible.

* fix linting

* fix linting due to merge

* Fix PR comments

* linting

* remove my name from code.

* Add engine test for encrypt decrypt in integration tests
Add helper method to change anonymized entity into encrypt result.

* add docs

* fix import

* Remove unneeded field

* Replace decrypt with deanonymize.

* Replace decrypt with deanonymize.

* revert

* Change to anonymizer_results

* Remove entities and merge them together.

* Remove entities and merge them together + renaming

* Remove entities and merge them together + renaming

* fix linting

* fix documentation and naming

* fix documentation and naming

* fix documentation and naming

* fix documentation and naming

* fix documentation and naming

* fix documentation and naming

* fix documentation and naming

* fix documentation and naming

* fix tests

* fix tests

* fix tests

* rename

* fix docs

* Change docs to the new deanonymization functionality.

* Change docs to the new deanonymization functionality.

* Fix PR comments

* Fix linting

* Fix linting

* fux PR comments

* fux PR comments

* fix PR comments

* fix PR comments

* add tests

* add tests

* fix PR comments

* fix PR comments

* fix PR comments

* fix PR comments

* fix PR comments

* bump version for breaking changes

* fix API doc

* Update docs/anonymizer/adding_operators.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Update docs/anonymizer/index.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Update docs/anonymizer/index.md

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* Update docs/api-docs/api-docs.yml

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>

* PR comments

Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
2021-04-07 11:01:39 +03:00
Nava Vaisman Levy
6aada038e2 print anonymized results json (#642) 2021-03-25 12:57:53 +02:00
Omri Mendels
ea85b71ee5 Additional analyzer examples (#605) 2021-03-11 17:02:58 +02:00
Sharon Hart
9e2bbb1dab Anonymizer - Encrypt anonymizer (#596)
* Add AES Encryption Logics

* Update NOTICE

* Key as method argument

* Anonymizer - Add encrypt anonymizer

* Anonymizer - Add encrypt anonymizer

* space

* Add documentation, tests

* Revert "Add AES Encryption Logics"

This reverts commit 18d991bc

* rename private method

* add key

* V1 to V2

* Add AES Encryption Logics

* Rebase to AES API change

* Add assertion on encryption

* change test names

Co-authored-by: sharon <sharon.hart@microsoft.com>
2021-03-08 11:39:45 +02:00
Omri Mendels
a6d720eb86 Align analyzer and anonymizer RecognizerResult objects (#591) 2021-03-03 18:28:47 +02:00
Nava Vaisman Levy
a229ec79da Add usage samples (#551)
* usage samples

* add curl sample, make postman collection downloadable
2021-02-24 19:37:09 +02:00
Nava Vaisman Levy
4652c51891 update notebook sample (#544) 2021-02-23 19:35:14 +02:00
Omri Mendels
800975f6c0 Added mkdocs for docs generation (#531) 2021-02-18 13:39:26 +02:00
Shiran Rubin
7093281062 V2 transformation to anonymizer (#526)
* Clean the engine a little by moving the text handling to another entity.
First draft.
2021-02-17 13:51:52 +02:00
Omri Mendels
0dda32f076 docs on adding recognizers (#508) 2021-02-16 17:10:19 +02:00
Nava Vaisman Levy
4c2496702f Sample jupiter notebook (#513) 2021-02-15 14:05:30 +02:00
omri374
8114524124 updates 2021-01-31 18:14:18 +02:00