feat(analyzer): add nine South African predefined recognizers (#2069)

* feat(analyzer): add nine South African predefined recognizers

Extend ZA coverage beyond ZA_ID_NUMBER with passport, tax, VAT, CIPC
registration, eNaTIS driver's licence and traffic register numbers,
licence plates, and mobile/telephone numbers split by line type.

All recognizers are disabled by default in default_recognizers.yaml.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(analyzer): update year validation logic in South African recognizers

Refactor the year validation logic in the ZaCompanyRegistrationRecognizer and ZaDriverLicenseRecognizer to ensure the current year is accurately checked without allowing for the next year. Remove unnecessary dependency on ZaIdNumberRecognizer in ZaDriverLicenseRecognizer to streamline the code. Update the validate_result method in ZaLicensePlateRecognizer to return a boolean type for consistency.

* fix(analyzer): address Copilot review feedback for ZA recognizers

Correct 08x NSN fallback classification, tighten driver licence validation,
and replace the unstable passport docstring reference.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(analyzer): align ZA driver licence docs and address Copilot round 3

Update driver licence length documentation to 10-14 characters to match
validation constraints, and use PhoneNumberMatcher.number directly instead
of re-parsing matched substrings.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Update CHANGELOG.md

* Update CHANGELOG.md

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
This commit is contained in:
Thato Mokoena
2026-07-23 11:36:59 +02:00
committed by GitHub
parent cc15735f99
commit f0dbdce29e
22 changed files with 1430 additions and 0 deletions

View File

@@ -6,6 +6,8 @@ All notable changes to this project will be documented in this file.
### Analyzer ### Analyzer
#### Added #### Added
- South African ID number (`ZA_ID_NUMBER`) recognizer for the 13-digit national identity number, using pattern matching, context words, birth-date validation, and Luhn checksum validation. Disabled by default.
- South African recognizers for `ZA_PASSPORT`, `ZA_INCOME_TAX_NUMBER`, `ZA_DRIVER_LICENSE`, `ZA_VAT_NUMBER`, `ZA_COMPANY_REGISTRATION`, `ZA_TRAFFIC_REGISTER_NUMBER`, `ZA_LICENSE_PLATE`, `ZA_MOBILE_NUMBER`, and `ZA_TELEPHONE_NUMBER`. All disabled by default.
- Added `NoOpNlpEngine` for configurations that do not require NLP engine artifacts, enabling standalone recognizers such as `HuggingFaceNerRecognizer` to run without a spaCy or Stanza model (#2071) (Thanks @ultramancode) - Added `NoOpNlpEngine` for configurations that do not require NLP engine artifacts, enabling standalone recognizers such as `HuggingFaceNerRecognizer` to run without a spaCy or Stanza model (#2071) (Thanks @ultramancode)
- Added per-recognizer and per-entity score threshold configuration in the recognizer registry YAML, with the analyzer's global `default_score_threshold` as the fallback (#2116) (Thanks @rodboev) - Added per-recognizer and per-entity score threshold configuration in the recognizer registry YAML, with the analyzer's global `default_score_threshold` as the fallback (#2116) (Thanks @rodboev)
- Added `PhUmidRecognizer` for Philippine Unified Multi-Purpose ID (UMID/CRN) numbers in dashed and plain 12-digit formats; disabled by default (#2045) (Thanks @Surya-5555) - Added `PhUmidRecognizer` for Philippine Unified Multi-Purpose ID (UMID/CRN) numbers in dashed and plain 12-digit formats; disabled by default (#2045) (Thanks @Surya-5555)

View File

@@ -141,6 +141,15 @@ For more information, refer to the [adding new recognizers documentation](analyz
| FieldType | Description | Detection Method | | FieldType | Description | Detection Method |
|------------|---------------------------------------------------------------------------------------------------------|------------------------------------------| |------------|---------------------------------------------------------------------------------------------------------|------------------------------------------|
| ZA_ID_NUMBER | The South African identity number is a 13-digit identifier in the `YYMMDDSSSSCAZ` format, where the trailing digit is validated with the Luhn algorithm. | Pattern match, context, and checksum. | | ZA_ID_NUMBER | The South African identity number is a 13-digit identifier in the `YYMMDDSSSSCAZ` format, where the trailing digit is validated with the Luhn algorithm. | Pattern match, context, and checksum. |
| ZA_PASSPORT | The South African passport number is a 9-character identifier with prefix letter A, D, M, or T followed by 8 digits. | Pattern match, context, and validation. |
| ZA_INCOME_TAX_NUMBER | The South African SARS income tax reference number is a 10-digit numeric identifier, commonly starting with 0, 1, 2, 3, or 9. | Pattern match, context, and validation. |
| ZA_DRIVER_LICENSE | The South African eNaTIS driver's licence number is an alphanumeric identifier of 1014 characters. | Pattern match, context, and validation. |
| ZA_VAT_NUMBER | The South African VAT registration number is a 10-digit identifier starting with 4. | Pattern match, context, and validation. |
| ZA_COMPANY_REGISTRATION | The South African CIPC company registration number uses modern `YYYY/NNNNNN/NN` format or legacy prefixed formats such as CK. | Pattern match, context, and validation. |
| ZA_TRAFFIC_REGISTER_NUMBER | The South African eNaTIS traffic register number is a 13-digit identifier for foreigners and organisations, disambiguated from ZA_ID_NUMBER via validation. | Pattern match, context, and validation. |
| ZA_LICENSE_PLATE | The South African vehicle licence plate uses provincial formats such as compact suffix forms (e.g. KD93GKGP) or spaced layouts (e.g. DK 28 LF GP). | Pattern match, context, and validation. |
| ZA_MOBILE_NUMBER | The South African mobile (cellular) number uses a 9-digit national significant number with country code +27 or domestic trunk prefix 0 (primarily 06x, 07x, and cellular 08x ranges). | `phonenumbers` match, line-type filter, and context. |
| ZA_TELEPHONE_NUMBER | The South African telephone number covers geographic landlines (01x05x) and non-mobile service lines such as toll-free (080), sharecall (086), and VoIP (087). | `phonenumbers` match, line-type filter, and context. |
### Thai ### Thai
| FieldType | Description | Detection Method | | FieldType | Description | Detection Method |

View File

@@ -318,6 +318,20 @@ recognizers:
enabled: false enabled: false
country_code: se country_code: se
- name: ZaCompanyRegistrationRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaDriverLicenseRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaIdNumberRecognizer - name: ZaIdNumberRecognizer
supported_languages: supported_languages:
- en - en
@@ -325,6 +339,55 @@ recognizers:
enabled: false enabled: false
country_code: za country_code: za
- name: ZaIncomeTaxNumberRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaLicensePlateRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaMobileNumberRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaPassportRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaTrafficRegisterNumberRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaTelephoneNumberRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ZaVatNumberRecognizer
supported_languages:
- en
type: predefined
enabled: false
country_code: za
- name: ThTninRecognizer - name: ThTninRecognizer
supported_languages: supported_languages:
- th - th

View File

@@ -87,9 +87,34 @@ from .country_specific.singapore.sg_fin_recognizer import SgFinRecognizer
from .country_specific.singapore.sg_uen_recognizer import SgUenRecognizer from .country_specific.singapore.sg_uen_recognizer import SgUenRecognizer
# South Africa recognizers # South Africa recognizers
from .country_specific.south_africa.za_company_registration_recognizer import (
ZaCompanyRegistrationRecognizer,
)
from .country_specific.south_africa.za_driver_license_recognizer import (
ZaDriverLicenseRecognizer,
)
from .country_specific.south_africa.za_id_number_recognizer import ( from .country_specific.south_africa.za_id_number_recognizer import (
ZaIdNumberRecognizer, ZaIdNumberRecognizer,
) )
from .country_specific.south_africa.za_income_tax_number_recognizer import (
ZaIncomeTaxNumberRecognizer,
)
from .country_specific.south_africa.za_license_plate_recognizer import (
ZaLicensePlateRecognizer,
)
from .country_specific.south_africa.za_passport_recognizer import (
ZaPassportRecognizer,
)
from .country_specific.south_africa.za_phone_number_recognizer import (
ZaMobileNumberRecognizer,
ZaTelephoneNumberRecognizer,
)
from .country_specific.south_africa.za_traffic_register_number_recognizer import (
ZaTrafficRegisterNumberRecognizer,
)
from .country_specific.south_africa.za_vat_number_recognizer import (
ZaVatNumberRecognizer,
)
# Spain recognizers # Spain recognizers
from .country_specific.spain.es_nie_recognizer import EsNieRecognizer from .country_specific.spain.es_nie_recognizer import EsNieRecognizer
@@ -250,7 +275,16 @@ __all__ = [
"TrLicensePlateRecognizer", "TrLicensePlateRecognizer",
"TrNationalIdRecognizer", "TrNationalIdRecognizer",
"SePersonnummerRecognizer", "SePersonnummerRecognizer",
"ZaCompanyRegistrationRecognizer",
"ZaDriverLicenseRecognizer",
"ZaIdNumberRecognizer", "ZaIdNumberRecognizer",
"ZaIncomeTaxNumberRecognizer",
"ZaLicensePlateRecognizer",
"ZaMobileNumberRecognizer",
"ZaPassportRecognizer",
"ZaTelephoneNumberRecognizer",
"ZaTrafficRegisterNumberRecognizer",
"ZaVatNumberRecognizer",
"LangExtractRecognizer", "LangExtractRecognizer",
"AzureOpenAILangExtractRecognizer", "AzureOpenAILangExtractRecognizer",
"BasicLangExtractRecognizer", "BasicLangExtractRecognizer",

View File

@@ -1,7 +1,27 @@
"""South Africa-specific recognizers.""" """South Africa-specific recognizers."""
from .za_company_registration_recognizer import ZaCompanyRegistrationRecognizer
from .za_driver_license_recognizer import ZaDriverLicenseRecognizer
from .za_id_number_recognizer import ZaIdNumberRecognizer from .za_id_number_recognizer import ZaIdNumberRecognizer
from .za_income_tax_number_recognizer import ZaIncomeTaxNumberRecognizer
from .za_license_plate_recognizer import ZaLicensePlateRecognizer
from .za_passport_recognizer import ZaPassportRecognizer
from .za_phone_number_recognizer import (
ZaMobileNumberRecognizer,
ZaTelephoneNumberRecognizer,
)
from .za_traffic_register_number_recognizer import ZaTrafficRegisterNumberRecognizer
from .za_vat_number_recognizer import ZaVatNumberRecognizer
__all__ = [ __all__ = [
"ZaCompanyRegistrationRecognizer",
"ZaDriverLicenseRecognizer",
"ZaIdNumberRecognizer", "ZaIdNumberRecognizer",
"ZaIncomeTaxNumberRecognizer",
"ZaLicensePlateRecognizer",
"ZaMobileNumberRecognizer",
"ZaPassportRecognizer",
"ZaTelephoneNumberRecognizer",
"ZaTrafficRegisterNumberRecognizer",
"ZaVatNumberRecognizer",
] ]

View File

@@ -0,0 +1,105 @@
from datetime import date
from typing import List, Optional
from presidio_analyzer import Pattern, PatternRecognizer
class ZaCompanyRegistrationRecognizer(PatternRecognizer):
"""
Recognize South African company registration numbers (CIPC).
Modern private and public companies use ``YYYY/NNNNNN/NN`` (year,
sequence, company-type suffix). Legacy formats include prefixed codes
such as ``CK`` (close corporation) and other CIPC entity prefixes.
Reference:
https://support.tradeshield.ai/support/solutions/articles/153000256853-cipc-company-codes-types-status-the-complete-guide
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
"""
COUNTRY_CODE = "za"
LEGACY_PREFIXES = frozenset({"CK", "K", "T", "W", "B", "M", "N", "NR"})
PATTERNS = [
Pattern(
"South African Company Registration (modern)",
r"\b(?:19|20)\d{2}/\d{6}/\d{2}\b",
0.4,
),
Pattern(
"South African Company Registration (legacy)",
r"\b(?:CK|K|T|W|B|M|N|NR)\d{4}/\d{6}\b",
0.3,
),
]
CONTEXT = [
"cipc",
"company registration",
"registration number",
"close corporation",
"company reg",
"enterprise number",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_COMPANY_REGISTRATION",
name: Optional[str] = None,
):
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
text = pattern_text.upper()
parts = text.split("/")
if len(parts) == 3 and parts[0].isdigit():
return self._validate_modern_format(text)
if len(parts) == 2:
return self._validate_legacy_format(text)
return False
def _validate_modern_format(self, text: str) -> bool:
parts = text.split("/")
if len(parts) != 3:
return False
year_part, sequence_part, type_part = parts
if not (
year_part.isdigit()
and sequence_part.isdigit()
and type_part.isdigit()
):
return False
if len(year_part) != 4 or len(sequence_part) != 6 or len(type_part) != 2:
return False
year = int(year_part)
return 1800 <= year <= date.today().year
def _validate_legacy_format(self, text: str) -> bool:
slash_index = text.index("/")
prefix = text[:slash_index]
sequence = text[slash_index + 1 :]
if not sequence.isdigit() or len(sequence) != 6:
return False
for legacy_prefix in sorted(self.LEGACY_PREFIXES, key=len, reverse=True):
if prefix.startswith(legacy_prefix):
year_part = prefix[len(legacy_prefix) :]
if len(year_part) == 4 and year_part.isdigit():
year = int(year_part)
return 1800 <= year <= date.today().year
return False

View File

@@ -0,0 +1,75 @@
import re
from typing import List, Optional
from presidio_analyzer import Pattern, PatternRecognizer
class ZaDriverLicenseRecognizer(PatternRecognizer):
"""
Recognize South African driver's licence numbers issued by eNaTIS.
eNaTIS licence numbers are alphanumeric strings of 1014
characters combining digit blocks with trailing letter groups.
Reference:
https://github.com/ugommirikwe/sa-license-decoder/blob/master/SPEC.md
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
"""
COUNTRY_CODE = "za"
MIN_LENGTH = 10
MAX_LENGTH = 14
PATTERNS = [
Pattern(
"South African Driver's Licence",
r"\b\d{6,10}[A-Z0-9]{2,5}\b",
0.3,
),
]
CONTEXT = [
"licence",
"license",
"driving licence",
"driving license",
"driver's licence",
"driver's license",
"drivers licence",
"drivers license",
"enatis",
"natis",
"licence number",
"license number",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_DRIVER_LICENSE",
name: Optional[str] = None,
):
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
text = pattern_text.upper()
if not self.MIN_LENGTH <= len(text) <= self.MAX_LENGTH:
return False
if re.fullmatch(r"\d{6,10}[A-Z0-9]{2,5}", text) is None:
return False
return bool(re.search(r"[A-Z]", text))

View File

@@ -0,0 +1,70 @@
from typing import List, Optional
from presidio_analyzer import Pattern, PatternRecognizer
class ZaIncomeTaxNumberRecognizer(PatternRecognizer):
"""
Recognize South African SARS income tax reference numbers.
Income tax reference numbers are 10-digit numeric identifiers issued
by SARS. They commonly start with ``0``, ``1``, ``2``, ``3``, or ``9``.
VAT numbers also have 10 digits but start with ``4`` — use
``ZA_VAT_NUMBER`` for those instead.
Reference:
https://www.taxtim.com/za/tax-guides/get-a-tax-number
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
"""
COUNTRY_CODE = "za"
TAX_NUMBER_LENGTH = 10
ALLOWED_LEADING_DIGITS = frozenset({"0", "1", "2", "3", "9"})
PATTERNS = [
Pattern(
"South African Income Tax Number",
r"\b[01239]\d{9}\b",
0.05,
),
]
CONTEXT = [
"sars",
"tax reference",
"income tax",
"tax number",
"itr",
"taxpayer",
"tax registration",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_INCOME_TAX_NUMBER",
name: Optional[str] = None,
):
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
return (
len(pattern_text) == self.TAX_NUMBER_LENGTH
and pattern_text.isdigit()
and pattern_text[0] in self.ALLOWED_LEADING_DIGITS
)

View File

@@ -0,0 +1,104 @@
from typing import List, Optional, Tuple
from presidio_analyzer import EntityRecognizer, Pattern, PatternRecognizer
class ZaLicensePlateRecognizer(PatternRecognizer):
"""
Recognize South African vehicle licence plates.
Provincial formats vary; common layouts include compact suffix forms
(e.g. ``KD93GKGP``), spaced 2023+ layouts (e.g. ``DK 28 LF GP``), and
prefix-digit forms (e.g. ``GET 103 WP``).
Reference:
https://en.wikipedia.org/wiki/Vehicle_registration_plates_of_South_Africa
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
:param replacement_pairs: List of tuples with potential replacement values
for different strings to be used during pattern matching.
"""
COUNTRY_CODE = "za"
PROVINCE_SUFFIXES = frozenset(
{"GP", "ZN", "WP", "EC", "NC", "FS", "LP", "MP", "NW"}
)
PATTERNS = [
Pattern(
"ZA Licence Plate (compact)",
r"\b[A-Z]{2,4}\d{2,4}[A-Z]{0,4}(?:GP|ZN|WP|EC|NC|FS|LP|MP|NW)\b",
0.3,
),
Pattern(
"ZA Licence Plate (spaced)",
r"\b[A-Z]{2}\s?\d{2}\s?[A-Z]{2}\s?(?:GP|ZN|WP|EC|NC|FS|LP|MP|NW)\b",
0.3,
),
Pattern(
"ZA Licence Plate (prefix digits)",
r"\b[A-Z]{2,3}\s?\d{2,3}\s?(?:GP|ZN|WP|EC|NC|FS|LP|MP|NW)\b",
0.3,
),
Pattern(
"ZA Licence Plate (EC numeric prefix)",
r"\b\d{2,3}\s?[A-Z]{2,3}\s?EC\b",
0.3,
),
]
CONTEXT = [
"licence plate",
"license plate",
"number plate",
"registration",
"vehicle registration",
"natis",
"enatis",
"plate number",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_LICENSE_PLATE",
replacement_pairs: Optional[List[Tuple[str, str]]] = None,
name: Optional[str] = None,
):
if replacement_pairs is not None:
self.replacement_pairs = replacement_pairs
else:
self.replacement_pairs = [("-", ""), (" ", "")]
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
sanitized = EntityRecognizer.sanitize_value(
pattern_text, self.replacement_pairs
).upper()
if len(sanitized) < 5:
return False
suffix = sanitized[-2:]
if suffix not in self.PROVINCE_SUFFIXES:
return False
body = sanitized[:-2]
if not body or not any(char.isalpha() for char in body):
return False
return True

View File

@@ -0,0 +1,68 @@
from typing import List, Optional
from presidio_analyzer import Pattern, PatternRecognizer
class ZaPassportRecognizer(PatternRecognizer):
"""
Recognize South African passport numbers using regex and validation.
South African passport numbers are 9 characters: one letter prefix
(A, D, M, or T) followed by 8 digits.
Reference:
https://en.wikipedia.org/wiki/South_African_passport
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
"""
COUNTRY_CODE = "za"
PASSPORT_LENGTH = 9
ALLOWED_PREFIXES = frozenset({"A", "D", "M", "T"})
PATTERNS = [
Pattern(
"South African Passport",
r"\b[ADMT]\d{8}\b",
0.2,
),
]
CONTEXT = [
"passport",
"passport number",
"travel document",
"dha",
"south african passport",
"rsa passport",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_PASSPORT",
name: Optional[str] = None,
):
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
text = pattern_text.upper()
if len(text) != self.PASSPORT_LENGTH:
return False
if text[0] not in self.ALLOWED_PREFIXES:
return False
return text[1:].isdigit()

View File

@@ -0,0 +1,197 @@
from typing import List, Literal, Optional
import phonenumbers
from phonenumbers import PhoneNumberType
from presidio_analyzer import EntityRecognizer
from presidio_analyzer.nlp_engine import NlpArtifacts
from presidio_analyzer.predefined_recognizers.generic.phone_recognizer import (
PhoneRecognizer,
)
LineClassification = Literal["mobile", "telephone"]
class ZaPhoneNumberRecognizer(PhoneRecognizer):
"""
Base recognizer for South African phone numbers using python-phonenumbers.
Splits detection into mobile and telephone line types. Subclasses set
``target_classification`` to return only one category.
Reference:
https://en.wikipedia.org/wiki/Telephone_numbers_in_South_Africa
:param supported_entity: The entity this recognizer can detect
:param target_classification: ``mobile`` or ``telephone`` filter to apply
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param leniency: Phone number matcher strictness (03)
:param name: Optional recognizer instance name
"""
COUNTRY_CODE = "za"
REGION = "ZA"
MOBILE_TYPES = frozenset(
{
PhoneNumberType.MOBILE,
PhoneNumberType.FIXED_LINE_OR_MOBILE,
}
)
TELEPHONE_TYPES = frozenset(
{
PhoneNumberType.FIXED_LINE,
PhoneNumberType.TOLL_FREE,
PhoneNumberType.PREMIUM_RATE,
PhoneNumberType.VOIP,
PhoneNumberType.SHARED_COST,
PhoneNumberType.PERSONAL_NUMBER,
PhoneNumberType.UAN,
PhoneNumberType.PAGER,
}
)
CONTEXT = PhoneRecognizer.CONTEXT + [
"cellular",
"handset",
"contact number",
"landline",
"tel",
"home number",
"work number",
"office number",
"sms",
"whatsapp",
]
def __init__(
self,
supported_entity: str,
target_classification: LineClassification,
context: Optional[List[str]] = None,
supported_language: str = "en",
leniency: Optional[int] = 1,
name: Optional[str] = None,
):
self.target_classification = target_classification
context = self.CONTEXT if context is None else context
super().__init__(
context=context,
supported_language=supported_language,
supported_entity=supported_entity,
supported_regions=(self.REGION,),
leniency=leniency,
name=name,
)
def analyze( # noqa: D102
self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts = None
):
results = []
for match in phonenumbers.PhoneNumberMatcher(
text, self.REGION, leniency=self.leniency
):
parsed_number = match.number
if phonenumbers.region_code_for_number(parsed_number) != self.REGION:
continue
classification = self._classify(parsed_number)
if classification != self.target_classification:
continue
results.append(
self._get_recognizer_result(match, text, self.REGION, nlp_artifacts)
)
return EntityRecognizer.remove_duplicates(results)
def _classify(self, parsed_number: phonenumbers.PhoneNumber) -> Optional[str]:
number_type = phonenumbers.number_type(parsed_number)
if number_type in self.MOBILE_TYPES:
return "mobile"
if number_type in self.TELEPHONE_TYPES:
return "telephone"
if number_type != PhoneNumberType.UNKNOWN:
return None
return self._classify_by_nsn_prefix(str(parsed_number.national_number))
@staticmethod
def _classify_by_nsn_prefix(nsn: str) -> Optional[LineClassification]:
if not nsn:
return None
first_digit = nsn[0]
if first_digit in "67":
return "mobile"
if nsn.startswith("80") or nsn.startswith(("86", "87")):
return "telephone"
if first_digit == "8":
return "mobile"
if first_digit in "123459":
return "telephone"
return None
class ZaMobileNumberRecognizer(ZaPhoneNumberRecognizer):
"""
Recognize South African mobile (cellular) numbers.
Covers national, international, and common formatted variants for
ICASA cellular ranges (primarily ``06x``, ``07x``, and cellular ``08x``).
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
:param leniency: Phone number matcher strictness (03)
:param name: Optional recognizer instance name
"""
def __init__(
self,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_MOBILE_NUMBER",
leniency: Optional[int] = 1,
name: Optional[str] = None,
):
super().__init__(
supported_entity=supported_entity,
target_classification="mobile",
context=context,
supported_language=supported_language,
leniency=leniency,
name=name,
)
class ZaTelephoneNumberRecognizer(ZaPhoneNumberRecognizer):
"""
Recognize South African telephone numbers.
Covers geographic landlines (``01x````05x``) and non-mobile service
lines such as toll-free (``080``), sharecall (``086``), and VoIP (``087``).
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
:param leniency: Phone number matcher strictness (03)
:param name: Optional recognizer instance name
"""
def __init__(
self,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_TELEPHONE_NUMBER",
leniency: Optional[int] = 1,
name: Optional[str] = None,
):
super().__init__(
supported_entity=supported_entity,
target_classification="telephone",
context=context,
supported_language=supported_language,
leniency=leniency,
name=name,
)

View File

@@ -0,0 +1,68 @@
from typing import List, Optional
from presidio_analyzer import Pattern, PatternRecognizer
from .za_id_number_recognizer import ZaIdNumberRecognizer
class ZaTrafficRegisterNumberRecognizer(PatternRecognizer):
"""
Recognize South African eNaTIS traffic register numbers (TRN).
Traffic register numbers are 13-digit identifiers assigned to
foreigners and organisations on the eNaTIS system. They share length
with South African ID numbers but use a different validation scheme.
Reference:
https://www.gov.za/services/driving-licence-driving/apply-traffic-register-number
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
"""
COUNTRY_CODE = "za"
TRN_LENGTH = 13
PATTERNS = [
Pattern(
"South African Traffic Register Number",
r"\b\d{13}\b",
0.05,
),
]
CONTEXT = [
"traffic register",
"traffic register number",
"trn",
"enatis",
"natis",
"vehicle register",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_TRAFFIC_REGISTER_NUMBER",
name: Optional[str] = None,
):
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
self._id_validator = ZaIdNumberRecognizer()
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
if len(pattern_text) != self.TRN_LENGTH or not pattern_text.isdigit():
return False
return not self._id_validator.validate_result(pattern_text)

View File

@@ -0,0 +1,66 @@
from typing import List, Optional
from presidio_analyzer import Pattern, PatternRecognizer
class ZaVatNumberRecognizer(PatternRecognizer):
"""
Recognize South African VAT registration numbers.
South African VAT numbers are 10 digits and always start with ``4``.
Reference:
https://meta.cdq.com/DataModel:CDQ/Business_Partner/Identifier/ZA_VAT
:param patterns: List of patterns to be used by this recognizer
:param context: List of context words to increase confidence in detection
:param supported_language: Language this recognizer supports
:param supported_entity: The entity this recognizer can detect
"""
COUNTRY_CODE = "za"
VAT_LENGTH = 10
VAT_PREFIX = "4"
PATTERNS = [
Pattern(
"South African VAT Number",
r"\b4\d{9}\b",
0.3,
),
]
CONTEXT = [
"vat",
"vat number",
"vat registration",
"tax invoice",
"sars",
"value added tax",
]
def __init__(
self,
patterns: Optional[List[Pattern]] = None,
context: Optional[List[str]] = None,
supported_language: str = "en",
supported_entity: str = "ZA_VAT_NUMBER",
name: Optional[str] = None,
):
patterns = self.PATTERNS if patterns is None else patterns
context = self.CONTEXT if context is None else context
super().__init__(
supported_entity=supported_entity,
patterns=patterns,
context=context,
supported_language=supported_language,
name=name,
)
def validate_result(self, pattern_text: str) -> bool: # noqa: D102
return (
len(pattern_text) == self.VAT_LENGTH
and pattern_text.isdigit()
and pattern_text.startswith(self.VAT_PREFIX)
)

View File

@@ -0,0 +1,53 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaCompanyRegistrationRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaCompanyRegistrationRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_COMPANY_REGISTRATION"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
("2009/199240/23", 1, ((0, 14),)),
("2014/256030/07", 1, ((0, 14),)),
("CIPC registration 2020/804826/07", 1, ((18, 32),)),
("CK2001/123456", 1, ((0, 13),)),
("Close corporation CK1998/654321 registered.", 1, ((18, 31),)),
("99/199240/23", 0, ()),
("2009/19924/23", 0, ()),
("2009/199240/234", 0, ()),
("hello world", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_company_registrations(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"registration_number, expected",
[
("2009/199240/23", True),
("2014/256030/07", True),
("CK2001/123456", True),
("K2010/654321", True),
("99/199240/23", False),
("2009/19924/23", False),
("AB2001/123456", False),
],
)
def test_validate_result(registration_number, expected, recognizer):
assert recognizer.validate_result(registration_number) is expected

View File

@@ -0,0 +1,53 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaDriverLicenseRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaDriverLicenseRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_DRIVER_LICENSE"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
("60390002CGBV", 1, ((0, 12),)),
("4024048D4P60", 1, ((0, 12),)),
("30040008X6Z6", 1, ((0, 12),)),
("4046048YPC9T", 1, ((0, 12),)),
("Driving licence number 114500482HFF on file.", 1, ((23, 35),)),
("eNaTIS licence 40260039Y068", 1, ((15, 27),)),
("8001015009087", 0, ()),
("60390002", 0, ()),
("ABCDEFGHIJK", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_drivers_licences(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"licence_number, expected",
[
("60390002CGBV", True),
("4024048D4P60", True),
("30040008X6Z6", True),
("8001015009087", False),
("60390002", False),
("ABCDEFGHIJK", False),
("ABC1234567890", False),
],
)
def test_validate_result(licence_number, expected, recognizer):
assert recognizer.validate_result(licence_number) is expected

View File

@@ -0,0 +1,51 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaIncomeTaxNumberRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaIncomeTaxNumberRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_INCOME_TAX_NUMBER"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
("0123456789", 1, ((0, 10),)),
("1234567890", 1, ((0, 10),)),
("9123456789", 1, ((0, 10),)),
("SARS tax reference 2987654321 on file.", 1, ((19, 29),)),
("4020269678", 0, ()),
("5123456789", 0, ()),
("012345678", 0, ()),
("01234567890", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_income_tax_numbers(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"tax_number, expected",
[
("0123456789", True),
("1234567890", True),
("9123456789", True),
("4020269678", False),
("5123456789", False),
("012345678", False),
],
)
def test_validate_result(tax_number, expected, recognizer):
assert recognizer.validate_result(tax_number) is expected

View File

@@ -0,0 +1,55 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaLicensePlateRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaLicensePlateRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_LICENSE_PLATE"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
("KD93GKGP", 1, ((0, 8),)),
("PMG017GP", 1, ((0, 8),)),
("BJ47HRZN", 1, ((0, 8),)),
("DK 28 LF GP", 1, ((0, 11),)),
("CC 75 CX ZN", 1, ((0, 11),)),
("GET 103 WP", 1, ((0, 10),)),
("015 SBZ EC", 1, ((0, 10),)),
("Licence plate MT77GJGP registered.", 1, ((14, 22),)),
("YES", 0, ()),
("1234567890", 0, ()),
("hello world", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_license_plates(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"plate, expected",
[
("KD93GKGP", True),
("DK28LFGP", True),
("CC75CXZN", True),
("GET103WP", True),
("015SBZEC", True),
("YES", False),
("1234GP", False),
],
)
def test_validate_result(plate, expected, recognizer):
assert recognizer.validate_result(plate) is expected

View File

@@ -0,0 +1,99 @@
"""Tests for South African mobile number (ZA_MOBILE_NUMBER) recognizer."""
import pytest
from presidio_analyzer.predefined_recognizers import ZaMobileNumberRecognizer
from presidio_analyzer.predefined_recognizers.country_specific.south_africa.za_phone_number_recognizer import (
ZaPhoneNumberRecognizer,
)
from tests import assert_result_within_score_range
@pytest.fixture(scope="module")
def recognizer():
"""Create a ZaMobileNumberRecognizer instance for testing."""
return ZaMobileNumberRecognizer()
@pytest.fixture(scope="module")
def entities():
"""Return the ZA_MOBILE_NUMBER entity type for testing."""
return ["ZA_MOBILE_NUMBER"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions, expected_score_ranges",
[
("+27632118258", 1, ((0, 12),), ((0.4, 1.0),)),
("+27615889091", 1, ((0, 12),), ((0.4, 1.0),)),
("063 211 8258", 1, ((0, 12),), ((0.4, 1.0),)),
("082 560 9352", 1, ((0, 12),), ((0.4, 1.0),)),
("+27825609352", 1, ((0, 12),), ((0.4, 1.0),)),
("0825609352", 1, ((0, 10),), ((0.4, 1.0),)),
(
"My mobile number is +27632118258.",
1,
((20, 32),),
((0.4, 1.0),),
),
(
"Cellphone: 082 560 9352",
1,
((11, 23),),
((0.4, 1.0),),
),
("011 262 5500", 0, (), ()),
("021 447 1234", 0, (), ()),
("0800 123 456", 0, (), ()),
("+14155550132", 0, (), ()),
("1234567890", 0, (), ()),
("hello world", 0, (), ()),
],
)
def test_when_mobile_in_text_then_all_mobile_numbers_found(
text,
expected_len,
expected_positions,
expected_score_ranges,
recognizer,
entities,
):
"""Test that ZA mobile recognizer correctly identifies cellular numbers."""
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos), (st_score, fn_score) in zip(
results, expected_positions, expected_score_ranges
):
assert_result_within_score_range(
res, entities[0], st_pos, fn_pos, st_score, fn_score
)
def test_supported_entity(recognizer):
"""Test that supported entity is correctly set."""
assert recognizer.supported_entities == ["ZA_MOBILE_NUMBER"]
def test_supported_language(recognizer):
"""Test that supported language is correctly set."""
assert recognizer.supported_language == "en"
@pytest.mark.parametrize(
"nsn, expected",
[
("632118258", "mobile"),
("825609352", "mobile"),
("881234567", "mobile"),
("891234567", "mobile"),
("800123456", "telephone"),
("861234567", "telephone"),
("871234567", "telephone"),
("112625500", "telephone"),
],
)
def test_classify_by_nsn_prefix(nsn, expected):
"""Test NSN fallback when python-phonenumbers returns UNKNOWN."""
assert ZaPhoneNumberRecognizer._classify_by_nsn_prefix(nsn) == expected

View File

@@ -0,0 +1,55 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaPassportRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaPassportRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_PASSPORT"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
("A34855903", 1, ((0, 9),)),
("D12345678", 1, ((0, 9),)),
("M87654321", 1, ((0, 9),)),
("T11223344", 1, ((0, 9),)),
("Passport number A19299317 on file.", 1, ((16, 25),)),
("DHA travel document T99887766", 1, ((20, 29),)),
("B12345678", 0, ()),
("A1234567", 0, ()),
("A123456789", 0, ()),
("X12345678", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_passports(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"passport_number, expected",
[
("A34855903", True),
("D12345678", True),
("M87654321", True),
("T11223344", True),
("B12345678", False),
("A1234567", False),
("A123456789", False),
("X12345678", False),
],
)
def test_validate_result(passport_number, expected, recognizer):
assert recognizer.validate_result(passport_number) is expected

View File

@@ -0,0 +1,78 @@
"""Tests for South African telephone number (ZA_TELEPHONE_NUMBER) recognizer."""
import pytest
from presidio_analyzer.predefined_recognizers import ZaTelephoneNumberRecognizer
from tests import assert_result_within_score_range
@pytest.fixture(scope="module")
def recognizer():
"""Create a ZaTelephoneNumberRecognizer instance for testing."""
return ZaTelephoneNumberRecognizer()
@pytest.fixture(scope="module")
def entities():
"""Return the ZA_TELEPHONE_NUMBER entity type for testing."""
return ["ZA_TELEPHONE_NUMBER"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions, expected_score_ranges",
[
("011 262 5500", 1, ((0, 12),), ((0.4, 1.0),)),
("021 447 1234", 1, ((0, 12),), ((0.4, 1.0),)),
("010 222 0057", 1, ((0, 12),), ((0.4, 1.0),)),
("(011) 390-9872", 1, ((0, 14),), ((0.4, 1.0),)),
("0800 123 456", 1, ((0, 12),), ((0.4, 1.0),)),
("0860 123 456", 1, ((0, 12),), ((0.4, 1.0),)),
("H(011)3909872", 1, ((1, 13),), ((0.4, 1.0),)),
(
"Landline (011) 262-5500 on file.",
1,
((9, 23),),
((0.4, 1.0),),
),
(
"H(011)3909872 B(011)4517333",
2,
((1, 13), (15, 27)),
((0.4, 1.0), (0.4, 1.0)),
),
("082 560 9352", 0, (), ()),
("+27632118258", 0, (), ()),
("+14155550132", 0, (), ()),
("1234567890", 0, (), ()),
("hello world", 0, (), ()),
],
)
def test_when_telephone_in_text_then_all_telephone_numbers_found(
text,
expected_len,
expected_positions,
expected_score_ranges,
recognizer,
entities,
):
"""Test that ZA telephone recognizer identifies landline and service numbers."""
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos), (st_score, fn_score) in zip(
results, expected_positions, expected_score_ranges
):
assert_result_within_score_range(
res, entities[0], st_pos, fn_pos, st_score, fn_score
)
def test_supported_entity(recognizer):
"""Test that supported entity is correctly set."""
assert recognizer.supported_entities == ["ZA_TELEPHONE_NUMBER"]
def test_supported_language(recognizer):
"""Test that supported language is correctly set."""
assert recognizer.supported_language == "en"

View File

@@ -0,0 +1,55 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaTrafficRegisterNumberRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaTrafficRegisterNumberRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_TRAFFIC_REGISTER_NUMBER"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
(
"Traffic register number 1234567890123 is on file.",
1,
((24, 37),),
),
(
"eNaTIS TRN 6001015000076 recorded.",
1,
((11, 24),),
),
("8001015009087", 0, ()),
("123456789012", 0, ()),
("12345678901234", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_traffic_register_numbers(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"trn, expected",
[
("1234567890123", True),
("6001015000076", True),
("8001015009087", False),
("123456789012", False),
("12345678901234", False),
],
)
def test_validate_result(trn, expected, recognizer):
assert recognizer.validate_result(trn) is expected

View File

@@ -0,0 +1,50 @@
import pytest
from presidio_analyzer.predefined_recognizers import ZaVatNumberRecognizer
from tests import assert_result
@pytest.fixture(scope="module")
def recognizer():
return ZaVatNumberRecognizer()
@pytest.fixture(scope="module")
def entities():
return ["ZA_VAT_NUMBER"]
@pytest.mark.parametrize(
"text, expected_len, expected_positions",
[
("4020269678", 1, ((0, 10),)),
("4170229407", 1, ((0, 10),)),
("VAT number 4250281542 on invoice.", 1, ((11, 21),)),
("SARS vat registration 4100168758", 1, ((22, 32),)),
("3020269678", 0, ()),
("402026967", 0, ()),
("40202696789", 0, ()),
("1234567890", 0, ()),
],
)
def test_analyze_valid_and_invalid_za_vat_numbers(
text, expected_len, expected_positions, recognizer, entities, max_score
):
results = recognizer.analyze(text, entities)
assert len(results) == expected_len
for res, (st_pos, fn_pos) in zip(results, expected_positions):
assert_result(res, entities[0], st_pos, fn_pos, max_score)
@pytest.mark.parametrize(
"vat_number, expected",
[
("4020269678", True),
("4170229407", True),
("3020269678", False),
("402026967", False),
("40202696789", False),
],
)
def test_validate_result(vat_number, expected, recognizer):
assert recognizer.validate_result(vat_number) is expected