pyveil detector evaluation: reproducible evidence

Current result

39 cases

33 positive and 6 negative synthetic regression cases.

Public corpus

36 expected findings

Across email, phone, card, JWT, credentials, URL, and key-value shapes.

Exact type counts

1.000 precision

Zero extra findings in this supported-shape corpus.

0 false positives

1.000 recall

Every labeled supported-shape finding was detected.

0 false negatives

Read this before quoting the numbers

This is a synthetic supported-shape regression corpus, not a representative real-world PII benchmark. It does not measure broad semantic names, addresses, languages, documents, images, or unknown provider formats.

What CI checks

python evaluation/evaluate.py --check exits non-zero if any case has:

a finding count or entity type different from the expected result
a labeled synthetic sensitive value left in redacted output
a non-empty Finding.raw
a negative input mutated unexpectedly

The gate runs alongside Ruff, mypy, the Python 3.8-3.14 test matrix, package build, and Twine metadata validation.

Reproduce it locally

git clone https://github.com/hyeonsangjeon/pyveil.git
cd pyveil
python -m pip install -e .

python evaluation/evaluate.py --check
python evaluation/evaluate.py --json
python evaluation/evaluate.py --benchmark

pyveil synthetic detector regression evaluation
cases: 39 (33 positive, 6 negative)
expected findings: 36
precision: 1.000
recall: 1.000
false positives: 0
false negatives: 0
mismatches: 0
scope: synthetic supported-shape regression only; not real-world PII recall

The optional benchmark reports a machine-dependent median for a fixed synthetic payload. It is diagnostic only and is not a latency service-level objective.

Built-in shapes covered by corpus v1

Finding	Covered examples	Important limit
`EMAIL`	Common addresses, plus tags, subdomains, multiples	Not a full RFC mailbox parser
`PHONE`	Korean, separated international, compact E.164	Not a country-aware phone library
`CREDIT_CARD`	Separated Luhn-valid test numbers	Luhn validation does not prove a real card
`JWT`	Compact three-segment shape	No signature or claims validation
`AUTH_HEADER`	Bearer, Basic, Token, ApiKey	Not every custom auth scheme
`PRIVATE_KEY`	PEM private-key block shape	No key validity check
`API_KEY`	High-signal OpenAI, GitHub, Slack, Google, AWS shapes	Not every provider format
`URL_QUERY_SECRET`	Token, key, and secret query parameters	Key-name driven
`KV_SECRET`	Password, cookie, secret, token pairs	Key-name driven

Source and contribution policy

The corpus and evaluator are standard-library Python and ship in the source distribution. Detector changes must add focused synthetic positive and negative cases, preserve the no-raw-value contract, and document provenance.

View corpus and evaluator Understand the limits