* presidio-structured
changelog
Static analysis
docstrings, types
preliminary tests engine
static analysis
isort
Minor refactorings
Update README.md
Fix late binding issues and example
removal of old samples
Refactoring, adding example
pre-clean-break-commit
broken commit, fixing TabularConfigBuilder
Rename TabularConfig
pre-breaking replace commit
removal of some old experimental files
rename tabular to structured
restructuring presidio tabular - pre del commit
Add project TODOs
testing dump presidio tabular
* Add unit tests
* rename engine, add buildfile
* Update setup.py
* lint-build-test
* Update lint-build-test.yml
* Add packages to setup.py
* Update presidio-structured to alpha version
* Update Presidio structured README.md
* Add logging configuration to presidio-structured
module
* Refactor AnalysisBuilder constructor to accept an
optional AnalyzerEngine parameter
* Fix entity mapping in JsonAnalysisBuilder
* Drop type in docstring in analysis builder classes
* Refactor TabularAnalysisBuilder to use
BatchAnalyzerEngine for all columns
* Update data_reader.py with type hints for file
paths
* Update data_reader.py to include additional
keyword arguments in read() method
* Update Transformer to Processor term in
StructuredEngine
* Add PandasDataProcessor as default to StructuredEngine
init
* Move structured sample files to the docs
* Add Presidio Structured Notebook to samples index
* Remove unnecessary imports in structured sample
* Update to processors in structured __init__ files
* Add explanation for structured table sample
* Delete unnecessary __init__s in structured test
* Fix bug in JsonAnalysisBuilder entity mapping
* pr comments, nits, minor tests
* README
* Add TabularAnalysisBuilder
* Some basic logging
* linting
* Fix typo in logger variable name
* Refactor analysis builder to include score
threshold
* Linting, continued
* Update Pipfile
* Refactor JsonAnalysisBuilder to support language
parameter
* Fix not camel case in TabularAnalysisBuilder
* Add score_threshold parameter to AnalysisBuilder
* Refactor JSON analysis builder to gain consistency
* Remove low score results in JsonAnalysisBuilder
* Add tests to json analysis with score threshold
* Fix bug in JSON analysis to update map with
nested_mappings
* Fix bug in JSON analysis to take only entity types
* Fix typos in test anl json names and assert values
* Update build-structured.yml
* Create __init__.py
* Type hint fix python <3.10, loggger typo
* Update setup.py
* PR comments variety
* further pr comments
* readme, refactor score, refactor tabular analysis
* Update test_analysis_builder.py
* lint
---------
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Co-authored-by: enrique.botia <enrique.botia@netzima.com>
* Adding methods to enable plotting of custom bboxes
* Adding test for get_pii_bboxes
* Linting fixes and addition of test for add_custom_bboxes
* Adding use of custom bbox into DICOM verification engine
* Fixing tests for the DICOM verification engine
* Adding in ocr kwargs compatibility and updating tests
* Adding example notebook
* Linting fixes
* Remaining linting fix
---------
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Enabling allow list approach
* Adding in empty list for allow_list in existing unit tests
* Added unit tests for newly introduced methods
* Adding unit test for allow list functionality
* Linting fixes
* Removing spaces in empty lines
* Fix integration test not accounting for empty space removal
* Updating notebook with more examples and adding ad_hoc_recognizers approach to standard image redactor engine as well
* Linting fixes
* Removing incomplete example code
* Fixing section header numbers
* Removing duplicate comment
---------
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Initial commit with old paths (will need to fix)
* Fixing paths
* Adding in notebook and updating documentation
* Adding in WIP notebook and docs
* Adding example notebook
* Clearing some output to reduce file size
* Linting fix v1
* Linting v2
* Add rounding to integration test to account for minor differences in real testing
* Fixing typo with wrong variable name
* Removing check for exact or rounded conf
* Updating logic for comparing ocr_results
* Minor fixes on DICOM verify code based on PR comments
* Changing results and gt format from dictionary to list of dictionaries
* Updating notebook
* Updating docstring format
* Linting fix
* Changing fixture scope for mock_engine
* Update docs/image-redactor/evaluating_dicom_redaction.md
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Docstring updates
* Improving efficiency of _remove_duplicate_entities
* Add handling for divide by zero in recall and precision calculations
* Making precision and recall calculation functions public
* Moving common fixtures to conftest
* PR comments
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Adding DicomImageRedactorEngine class without test data for now
* Updating and putting placeholder in package doc
* Updating docs
* Moving presidio-dicom-image-redactor module into presidio-image-redactor module as an extension
* Updating docs to reflect refactor from presidio-dicom-image-redactor module to extension of presidio-image-redactor module
* Docs fixes continued
* Reflecting move from new module to updating existing module
* Updating requirements
* Linting fixes
* Linting fixes. Function and variable name casing
* Fixing getting started code blocks
* Updating image redactor version
* Fix minor typo: phi -> PHI in comment
* Update presidio-image-redactor/presidio_image_redactor/utils/dicom_image_redact_utils.py
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Update presidio-image-redactor/presidio_image_redactor/utils/dicom_image_redact_utils.py
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Fixing new typing adding added from PR comment commit
* Moving all utils into the DicomImageRedactorEngine class
* Fixing integration test now that utils have been moved
* Moving to ad-hoc recognizer approach
* Updating redact to have similar format as in ImageRedactorEngine. Moved existing logic for redact into redact_from_file
* Updating requirements
* Updating docs with new getting started code
* Versioning and changelog updates
* Adding in data with citations
* Linting fixes
* Minor change: from multiple list append to list extend
* Adding pytest-mock to requirements
* Fixing paths to test data
* Making file extension search more flexible and work with both Windows and Unix systems
* Changing docstring style to match rest of repo
* Making statement about Tesseract version more general in one other doc (was caught in another earlier)
* Changing box_color_setting to fill for consistency
* Updating changelog based on PR comments
* Update docs/image-redactor/index.md
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Updating docs
* Adding **kwargs support and addressing minor comments
* Adjusting with clause duration
* Adding note about custom ImageAnalyzerEngine support
* Changing minor comment in docstring
* Splitting redact_from_file into redact_from_file and redact_from_directory
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Add decryption mechanism.
On purpose we separate it from the anonymizer/
* Fix tests
* Remove unneeded params
fix docs and tests
* Add decryption engine to match the anonymizer engine.
Refactor code so the text builder and text manipulator will be mutual.
* Change engine result to be unified with different items inheriting from result item.
* Fix decrypt elements and implmentations
* Add hints
* Fix naming
* Change manipulators to operators.
* Change manipulators to operators.
* Move initialization of text manipulator
Change the analyzer results json to receive the list and not the entire content.
Fix tests
* Move getting operators classes into the operator factory.
* Fix entities.
Make sure all engines can use the same entities by using factories, builders and inheritance more.
* revert unneeded change
* Code cleanup
* Add tests and clean up code.
* Move anonymizer configuration to inherit from operator metadata and add decryptor config
* More cleanup
* More cleanup
* revert
* cleanup
* Fix imports and renaming
* Start working on linting.
* Start working on linting.
* add tests
* add tests
* add tests
* add tests
* remove unneeded formating.
* fix linting
* try to understand why it fails in pipe
* try to understand why it fails in pipe
* Fix equal check for recognizer result
* revert change for test investigation
* fix imports and add test
* fix dictionary
* add test for dictionary fix
* Fix example
* Fix E2E entities
* Fix PR comments
* Fix PR comments
* Fix PR comments
* Fix PR comments
* Fix PR comments
* Fix PR comments - use comprehensions
* Fix PR comments - Change names and move classes accordingly.
* fix linting
* fix linting
* use dictionary comprehensions
* Fix documentation and nameing.
* Fix documentation and nameing.
* remove unused code.
* PR comments
* fix linting
* PR review -> be background compatible.
* fix linting
* fix linting due to merge
* Fix PR comments
* linting
* remove my name from code.
* Add engine test for encrypt decrypt in integration tests
Add helper method to change anonymized entity into encrypt result.
* add docs
* fix import
* Remove unneeded field
* Replace decrypt with deanonymize.
* Replace decrypt with deanonymize.
* revert
* Change to anonymizer_results
* Remove entities and merge them together.
* Remove entities and merge them together + renaming
* Remove entities and merge them together + renaming
* fix linting
* fix documentation and naming
* fix documentation and naming
* fix documentation and naming
* fix documentation and naming
* fix documentation and naming
* fix documentation and naming
* fix documentation and naming
* fix documentation and naming
* fix tests
* fix tests
* fix tests
* rename
* fix docs
* Change docs to the new deanonymization functionality.
* Change docs to the new deanonymization functionality.
* Fix PR comments
* Fix linting
* Fix linting
* fux PR comments
* fux PR comments
* fix PR comments
* fix PR comments
* add tests
* add tests
* fix PR comments
* fix PR comments
* fix PR comments
* fix PR comments
* fix PR comments
* bump version for breaking changes
* fix API doc
* Update docs/anonymizer/adding_operators.md
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Update docs/anonymizer/index.md
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Update docs/anonymizer/index.md
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Update docs/api-docs/api-docs.yml
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* PR comments
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
