mirror of
https://github.com/data-privacy-stack/presidio.git
synced 2026-08-02 08:10:46 -05:00
08219e7663
* Add LangExtract recognizer for PII extraction - Introduced LangExtract recognizer to enhance PII detection capabilities. - Added configuration files for LangExtract prompts and examples. - Implemented LangExtractRecognizer class to handle PII extraction using LangExtract. - Created tests for LangExtract recognizer to ensure functionality and reliability. - Added a simple standalone test script for quick validation of LangExtract setup. - Updated pyproject.toml to include langextract as a dependency. * refine the docs * narrow support for oollama only * Refactor LangExtract tests to use Ollama; remove API key dependency * adding first draft of docker compose * Update model_id in tests to use 'gemma2:2b' instead of 'gemini-2.5-flash' * Refactor LangExtract documentation to focus on Ollama support; remove references to other LLM providers * Update README to remove Ollama setup instructions and clarify integration guide reference * Enhance Ollama installation script with progress messages and error handling; update model download method for better user feedback * auto ruff fixes * Enhance LangExtractRecognizer tests with real Ollama integration - Updated `langextract_recognizer_class` fixture to create a test-specific configuration for LangExtractRecognizer, enabling it for testing. - Refactored tests in `test_langextract_recognizer.py` to utilize the new configuration and validate the recognizer's behavior with real Ollama. - Removed mock-based tests for LangExtract and replaced them with integration tests that check the recognizer's functionality against a running Ollama instance. - Added tests to verify the recognizer's initialization, entity detection, and error handling when the Ollama server is unreachable. - Ensured that only requested entities are returned and that results include analysis explanations. * Remove unnecessary line breaks in LLM-based PII detection section of README * Add LangExtract LLM-based PII detection test and configuration * Improve Ollama availability check with setup attempt message * Increase wait time for services and update healthcheck parameters for Ollama service * Add Ollama setup for Analyzer tests and improve availability check * Set timeout for Ollama setup in Analyzer tests to 8 minutes * Enhance Ollama setup for Analyzer tests with improved installation and readiness checks * Update Ollama model references from gemma2:2b to llama3.2:1b across configuration and test files * Update model references from llama3.2:1b to gemma2:2b across configuration, scripts, and tests * Remove 'enabled' configuration from LangExtract settings in YAML files and update tests accordingly * Update Ollama service configuration: change port mapping and modify healthcheck command * Refactor logging in LangExtractRecognizer: reduce verbosity and improve clarity of extraction results * Update Ollama service configuration: modify port mapping and healthcheck command * Update CI workflow and tests: reduce sleep duration and add environment variable for LangExtract recognizer * Reduce sleep duration in CI workflow from 150 to 60 seconds * Update LangExtract model references from gemma2:2b to gemma3:1b and remove obsolete installation script * docs and prompt fixes * finalizing the pr * Update Ollama image to latest version and add LangExtract PII/PHI extraction examples and prompts * fix bad example * fix unit-tests * refactor: clean up .env file and simplify skip_engine logic in tests * chore: add a new line to .env file for better readability * chore: remove unnecessary blank line from .env file * revert .env * intial commit * Remove skip marker for spacy_nlp_engine fixture * Remove skip markers for stanza and transformers NLP engine fixtures * Remove Ollama recognizer test and update default recognizers configuration * Remove unused Ollama recognizer configuration and update prompt file references * Add end-to-end tests for API anonymization and redaction features - Implemented tests for the anonymization API in `test_api_anonymizer.py`, covering various scenarios including valid requests, empty inputs, malformed requests, and custom anonymizers. - Created integration tests in `test_api_e2e_integration_flows.py` to validate the analyze and anonymize workflow with PII detection. - Added tests for image redaction functionality in `test_api_image_redactor.py`, ensuring proper handling of image data and error responses. - Developed package-level tests in `test_package_e2e_integration_flows.py` to verify the functionality of the analyzer and anonymizer engines, including support for third-party recognizers. * Remove unused Ollama recognizer imports and related tests * Update requirements and improve Ollama recognizer availability checks in e2e tests * Fix formatting in requirements.txt for analyzer and anonymizer dependencies * Update Ollama model ID from gemma3:1b to gemma2:2b in configuration and tests * gemma2:2b * finalizing the pr * Remove unused ABC import from lm_recognizer.py * Fix indentation in docker-compose.yml for volumes section * Fix line break for clarity in adding_recognizers.md * Add timeout settings for Ollama recognizer and test cases * Refactor timeout comment for clarity in OllamaLangExtractRecognizer * Update Ollama model version and add configuration for LangExtract recognizer tests * Update examples_file path in configuration for Ollama recognizer * Remove timeout decorator from Ollama recognizer * Add rerun settings to unit and E2E tests for improved stability * Remove rerun settings from unit and E2E test commands for simplification * Set max-parallel to 2 for local build and E2E tests * Remove max-parallel setting from local build and E2E tests * move poetry cache dir * pr changes * code review changes * ruff check * remove unused json import in test_ollama_recognizer.py * refactor test names for clarity and consistency in test_ollama_recognizer.py * finalizing the PR * self code review fixes * Refactor LangExtractRecognizer to use yaml for configuration loading * ruff fixes * Update error messages in OllamaLangExtractRecognizer tests for clarity * CR comment addressed * Remove unused variables from Jinja2 prompt rendering in LangExtractRecognizer * exporting functionality to helpers enlarging composition * composition * Refactor entity mapper and langextract utilities for improved clarity and consistency * Refactor tests to use get_langextract_module for mocking LangExtract availability * Refactor langextract utilities to improve clarity and error handling; remove deprecated functions and update tests accordingly * Refactor LLM utilities by simplifying docstrings and consolidating imports for improved readability and maintainability * Update error message for missing Jinja2 installation to include poetry installation instructions * Add Ollama recognizer configuration and tests for YAML integration - Introduced `test_ollama_enabled_recognizers.yaml` to define recognizers including OllamaLangExtractRecognizer. - Enhanced `test_package_e2e_integration_flows.py` with a test to validate loading of Ollama recognizer from YAML configuration. - Updated `OllamaLangExtractRecognizer` to support configuration path and language parameters. - Improved handling of relative paths for configuration files. * Refactor docstrings in OllamaLangExtractRecognizer for improved clarity and formatting * Enhance OllamaLangExtractRecognizer initialization docstring to clarify kwargs usage * pr comments * Refactor OllamaLangExtractRecognizer to streamline config path handling and remove redundant comments * Refactor Ollama recognizer test to improve clarity and enhance entity detection validation * Refactor tests for Ollama recognizer and LMRecognizer to improve exception handling and configuration validation * Update config path for Ollama recognizer in test configuration * Update config paths for Ollama recognizer and add test configuration for LangExtract * Remove test configuration for Ollama LangExtract * Remove test configuration for Ollama LangExtract recognizer * Fix formatting in resolve_config_path function for improved readability * Enable UsLangExtractRecognizer and update its config path * change all configs to use gemma3:1b * Disable Ollama LangExtract recognizer and update its configuration path * Update langextract configuration paths to use absolute paths for prompt and examples files * Remove test script for Ollama recognizer configuration loading * Refactor config loading in examples and prompt loaders to use resolve_config_path; update logging level in LMRecognizer; add langextract availability check in OllamaLangExtractRecognizer. * Refactor parameter description in load_yaml_examples and clean up imports in prompt_loader * Update langextract paths to use repo-root-relative paths in tests and prompt loader * Enhance documentation for Ollama setup and improve __init__.py imports for clarity and maintainability * code review changes * pr comments & align to main --------- Co-authored-by: Tamir Kamara <26870601+tamirkamara@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>