Data Security & PII Boundaries for Filing Systems

A PII boundary is the enforced line in a filing pipeline past which no raw policyholder identifier is allowed to travel. Valuation models, statutory reserve runs, and capital reports routinely ingest granular policy records — names, dates of birth, postal codes, direct policy identifiers — yet a regulatory submission almost never needs any of them in raw form. The engineering problem this page solves is how to strip, tokenize, and prove-the-absence of that data at a single deterministic control point, so that the output of your Regulatory Architecture & Compliance Mapping pipeline is both actuarially faithful and free of personally identifiable information (PII) before it reaches an examiner. This is a data-security obligation with statutory teeth: the NAIC Insurance Data Security Model Law (#668) Section 4 requires a documented information security program with data classification, the Gramm-Leach-Bliley Safeguards Rule requires access controls and encryption for nonpublic personal information, and NIST SP 800-122 defines the confidentiality controls a filing system must apply to PII at rest and in transit.

The Problem: Where PII Leaks in a Filing Pipeline #

Actuarial workspaces are hostile to naive data hygiene. The same policy extract that feeds a principle-based reserve also lands in scenario caches, Parquet spill files, worker logs, exception queues, and the audit trail — and every one of those surfaces is a place raw PII can escape. A reserve number is a cohort-level statistic; it does not require insured_name or ssn to be computed. Yet those columns arrive attached to every exposure row because policy administration systems export whole records, and unless a boundary actively removes them, they propagate by default.

The failure is rarely a dramatic breach. It is a date_of_birth column that survives into a filed XML because nobody classified it, a debug log that prints a full record on a validation exception, or a decrypted temporary file left on disk after an out-of-memory crash. Each is a reportable event under Model Law #668 Section 6 (notification of a cybersecurity event) and a finding under a GLBA examination. The discipline that prevents all of them is the same: treat PII handling as a fail-closed transformation that every record must pass through exactly once, driven by an explicit field classification matrix rather than ad-hoc column drops scattered through the code.

This boundary sits directly upstream of the Actuarial Audit Trail Architecture: direct identifiers must be tokenized before they enter the tamper-evident ledger, because an append-only trail that captures raw PII becomes an immutable liability you cannot purge. It also sits downstream of ingestion, where the typed data contracts from Schema Validation with Pydantic & Great Expectations first give every field a name and a type the boundary can key its rules against.

Architecture of the Boundary #

The boundary is a single-pass transformation with three responsibilities: classify every field against a matrix, apply the mandated handling (pass, pseudonymize, hash, or redact), and refuse to emit any record that contains a field it does not recognize. Referential integrity survives because pseudonymization is deterministic — the same policy_id always maps to the same surrogate token — so cohort joins, experience-study links, and cross-period reconciliations still work on tokenized data without ever exposing the raw key.

Determinism is what makes tokenization compatible with actuarial work. A random surrogate would break every join; a deterministic keyed token preserves the relationship structure of the portfolio while destroying the ability to recover the identity without the key. The key itself never lives in the pipeline — it is fetched from a KMS or HSM per run and held only in memory, satisfying the key-management expectation of NIST SP 800-53 Rev. 5 control SC-12 and the encryption-at-rest expectation of SC-28.

Prerequisites #

Before implementing the boundary, have the following in place:

• Python 3.11+ with the standard library hashlib, hmac, secrets, and enum. The cryptographic primitives for deterministic tokenization need no third-party library; add cryptography only if you introduce format-preserving encryption for numeric fields or detached signatures.
• A typed ingestion contract. Every field must already carry a stable name and dtype so the classification matrix can key against it. This comes from the Pydantic schema enforcement layer at the ingestion boundary; the matrix below assumes those field names.
• A vectorized data plane. For portfolio-scale runs the boundary applies per column, not per row — the DataFrame patterns in Pandas & NumPy for Actuarial Data Pipelines keep the transformation memory-safe over millions of exposures.
• A key source. A KMS/HSM endpoint or, at minimum, an environment-injected secret for the tokenization key and a per-jurisdiction pepper. Never hard-code either.
• Regulatory context. How your outputs map to statute, established in the parent Regulatory Architecture & Compliance Mapping framework, plus the residency and encryption constraints in OSFI Model Risk Management Guidelines for cross-border filers.

Core Implementation: A Classification-Driven PII Boundary #

The canonical pattern is a small module whose behaviour is entirely governed by a declarative classification matrix. The matrix is the single source of truth — auditors read it, tests assert against it, and the code does nothing a field’s rule does not authorize. Anything not in the matrix is rejected, which is what makes the boundary fail-closed.

from __future__ import annotations

import hashlib
import hmac
from dataclasses import dataclass
from enum import Enum


class Sensitivity(str, Enum):
    PUBLIC = "public"
    INTERNAL = "internal"
    CONFIDENTIAL = "confidential"
    RESTRICTED = "restricted"   # direct identifiers / PII


class Handling(str, Enum):
    PASS = "pass"               # plaintext allowed downstream
    PSEUDONYMIZE = "token"      # deterministic, key-reversible surrogate
    HASH = "hash"              # one-way digest, no recovery
    REDACT = "redact"          # dropped entirely


@dataclass(frozen=True)
class FieldRule:
    sensitivity: Sensitivity
    handling: Handling


# The classification matrix: the auditable contract the boundary enforces.
FILING_SCHEMA: dict[str, FieldRule] = {
    "policy_id":       FieldRule(Sensitivity.RESTRICTED,   Handling.PSEUDONYMIZE),
    "ssn":             FieldRule(Sensitivity.RESTRICTED,   Handling.REDACT),
    "insured_name":    FieldRule(Sensitivity.RESTRICTED,   Handling.REDACT),
    "date_of_birth":   FieldRule(Sensitivity.RESTRICTED,   Handling.HASH),
    "postal_code":     FieldRule(Sensitivity.CONFIDENTIAL, Handling.HASH),
    "issue_age":       FieldRule(Sensitivity.INTERNAL,     Handling.PASS),
    "gender":          FieldRule(Sensitivity.INTERNAL,     Handling.PASS),
    "face_amount":     FieldRule(Sensitivity.INTERNAL,     Handling.PASS),
    "lapse_rate":      FieldRule(Sensitivity.INTERNAL,     Handling.PASS),
    "mortality_table": FieldRule(Sensitivity.PUBLIC,       Handling.PASS),
    "valuation_date":  FieldRule(Sensitivity.PUBLIC,       Handling.PASS),
}


class PIIBoundaryError(ValueError):
    """Raised when a record cannot be sanitized safely (fail-closed)."""


def _tokenize(value: str, key: bytes, pepper: bytes) -> str:
    # Deterministic keyed surrogate: same input + same key -> same token,
    # but the raw value is unrecoverable without the key held in the KMS.
    mac = hmac.new(key, pepper + value.encode("utf-8"), hashlib.sha256)
    return "tok_" + mac.hexdigest()[:24]


def _hash(value: str, pepper: bytes) -> str:
    # One-way digest for fields we must compare but never reverse.
    return hashlib.sha256(pepper + value.encode("utf-8")).hexdigest()[:32]


def sanitize_record(
    record: dict[str, object],
    key: bytes,
    pepper: bytes,
) -> dict[str, object]:
    """Apply the classification matrix to one record. Fail-closed on any
    field the matrix does not recognize."""
    sanitized: dict[str, object] = {}
    for field, value in record.items():
        rule = FILING_SCHEMA.get(field)
        if rule is None:
            raise PIIBoundaryError(f"unclassified field at boundary: {field!r}")
        if value is None:
            sanitized[field] = None
            continue
        if rule.handling is Handling.PASS:
            sanitized[field] = value
        elif rule.handling is Handling.PSEUDONYMIZE:
            sanitized[field] = _tokenize(str(value), key, pepper)
        elif rule.handling is Handling.HASH:
            sanitized[field] = _hash(str(value), pepper)
        elif rule.handling is Handling.REDACT:
            continue  # field is dropped entirely
    return sanitized

The final guarantee is enforced not by the loop but by a schema on the output. A Pydantic model of the sanitized payload with extra="forbid" and no restricted raw fields turns any leak into a hard validation error at serialization time — a defense-in-depth check independent of the boundary code itself.

from datetime import date
from pydantic import BaseModel, ConfigDict


class SanitizedExposure(BaseModel):
    model_config = ConfigDict(extra="forbid")  # any stray field -> ValidationError

    policy_id: str          # tokenized surrogate, e.g. "tok_9f2c…"
    date_of_birth: str      # hashed digest, never a real date
    postal_code: str | None
    issue_age: int
    gender: str
    face_amount: float
    lapse_rate: float
    mortality_table: str
    valuation_date: date

    def assert_no_raw_pii(self) -> None:
        if not self.policy_id.startswith("tok_"):
            raise ValueError("policy_id is not tokenized")
        if len(self.date_of_birth) != 32:
            raise ValueError("date_of_birth is not a digest")

Because extra="forbid" rejects unknown keys, a leaked insured_name or ssn cannot round-trip into a filed payload even if the boundary loop is later edited incorrectly — the model raises before the record is serialized.

Configuration and Tuning #

The boundary’s security depends almost entirely on how keys, peppers, and chunk sizes are configured, not on the algorithm. The defaults below are annotated with the trade-off each parameter controls.

import os

BOUNDARY_CONFIG = {
    # Tokenization key: fetched per run from KMS/HSM, never persisted to disk.
    # Rotating it re-randomizes all surrogates, so pin one key per filing period
    # to keep tokens stable across the quarter's reconciliations.
    "token_key_arn": os.environ["FILING_TOKEN_KEY_ARN"],

    # Per-jurisdiction pepper: prevents a token minted for a US filing from
    # matching the same policyholder's token in a Canadian filing, satisfying
    # OSFI data-residency separation.
    "pepper": os.environb[b"FILING_PEPPER"],

    # Low-entropy fields (date_of_birth, postal_code) are vulnerable to
    # brute-force reversal of a plain hash. Stretch them with PBKDF2 instead
    # of a single SHA-256 pass.
    "stretch_low_entropy": True,
    "kdf_iterations": 200_000,

    # Chunk size for vectorized column-wise sanitization of large Parquet
    # partitions. Tune down if the worker RSS approaches its cgroup limit.
    "chunk_rows": 250_000,

    # Encrypt any temporary spill with AES-256-GCM and shred on cleanup.
    "encrypt_spill": True,
}

For the low-entropy case, replace the single-pass _hash with a stretched digest — a plain SHA-256 of a birth date is trivially reversible by hashing all plausible dates, so date_of_birth must be run through a slow KDF with the jurisdiction pepper as salt:

def _hash_low_entropy(value: str, pepper: bytes, iterations: int) -> str:
    return hashlib.pbkdf2_hmac(
        "sha256", value.encode("utf-8"), pepper, iterations
    ).hex()[:32]

Step-by-Step Walkthrough #

1. Author the classification matrix. Map every ingestion column to a Sensitivity tier and a Handling action in FILING_SCHEMA. This artifact is what a GLBA or Model Law #668 examiner reviews; keep it in version control and treat changes as reviewed events.
2. Fetch keys per run. Resolve token_key_arn and pepper from the KMS/HSM at the start of the job and hold them only in memory. Never write them to logs, config files, or the audit trail.
3. Sanitize column-wise. Apply sanitize_record (vectorized over chunk_rows partitions) at the extraction boundary, before any worker, cache, or log sees the raw record. Verify the raw frame is dereferenced and not persisted.
4. Validate the output shape. Load each sanitized record into SanitizedExposure and call assert_no_raw_pii. Any ValidationError here quarantines the batch — it means a raw field escaped the matrix.
5. Seal, then trail. Only after sanitization does the record enter the Actuarial Audit Trail Architecture ledger and the secure audit log packaging for transmission, so the immutable record contains only tokens and digests.
6. Route failures safely. On a PIIBoundaryError or validation failure, push the offending record to an encrypted dead-letter queue with the field name — never the value — attached, and alert through the compliance channel described in Automating NAIC Filing Deadline Alerts in Python so the miss surfaces before the deadline.

Validation and Testing #

Correctness for a PII boundary means two provable properties: no raw identifier survives, and referential integrity is preserved. Both are unit-testable and both should run as a mandatory gate in CI.

import pytest


def test_no_raw_pii_survives():
    key, pepper = b"unit-test-key-32-bytes-long!!", b"us-jurisdiction"
    raw = {
        "policy_id": "POL-100248",
        "ssn": "123-45-6789",
        "insured_name": "Jane Q. Policyholder",
        "date_of_birth": "1974-03-11",
        "issue_age": 50,
        "gender": "F",
        "face_amount": 500_000.0,
        "lapse_rate": 0.042,
        "mortality_table": "2017 CSO",
        "valuation_date": "2026-06-30",
    }
    clean = sanitize_record(raw, key, pepper)
    # Redacted identifiers are gone entirely.
    assert "ssn" not in clean and "insured_name" not in clean
    # Tokenized/hashed fields no longer contain the raw value.
    assert clean["policy_id"].startswith("tok_")
    assert "1974-03-11" not in str(clean.values())


def test_deterministic_referential_integrity():
    key, pepper = b"unit-test-key-32-bytes-long!!", b"us-jurisdiction"
    a = sanitize_record({"policy_id": "POL-100248"}, key, pepper)
    b = sanitize_record({"policy_id": "POL-100248"}, key, pepper)
    # Same policy -> same token, so cohort joins still work post-tokenization.
    assert a["policy_id"] == b["policy_id"]


def test_unclassified_field_is_rejected():
    with pytest.raises(PIIBoundaryError):
        sanitize_record({"beneficiary_email": "x@y.com"}, b"k" * 28, b"p")

Beyond unit tests, add a Great Expectations checkpoint on the sanitized dataset asserting a regex-based absence expectation (no SSN pattern, no date pattern) across all string columns — the same checkpoint discipline used for input contracts in Schema Validation with Pydantic & Great Expectations. Run a periodic re-identification drill: take the tokenized filing set and confirm that, without the KMS key, no direct join back to the source portfolio is possible.

Failure Modes and Gotchas #

• Fail-open on new columns. The most dangerous defect is a boundary that silently passes unrecognized fields. A policy admin system adds beneficiary_email next quarter, and without the PIIBoundaryError guard it flows straight into the filing. The matrix must be exhaustive and the default must be reject, not pass.
• Reversible low-entropy hashes. A single SHA-256 of date_of_birth or postal_code is not pseudonymization — the input space is small enough to enumerate. Always stretch low-entropy fields with a KDF and a secret pepper, or the “hash” is cosmetic.
• Non-deterministic serialization breaking tokens. If the value feeding _tokenize is not canonicalized (stray whitespace, "50" vs 50, timezone drift on timestamps), the same policyholder produces different tokens across runs and every reconciliation breaks. Normalize types before the boundary, keyed off the same typed contract the ingestion layer enforces.
• Decrypted spill files. Chunked processing of large Parquet partitions writes intermediates to disk. If those are unencrypted or not shredded on an out-of-memory crash, raw PII persists outside the boundary. Encrypt spill with AES-256-GCM and register a cleanup handler that runs even on abnormal exit.
• PII in exception logs. A validation error that logs the offending record re-introduces the identifier into an application log — a surface the boundary was meant to protect. Log field names and record IDs (already tokenized), never values.
• Cross-jurisdiction token collision. Reusing one pepper across US and Canadian filings lets an observer link a policyholder across regimes, which can breach residency separation under OSFI expectations. Scope the pepper per jurisdiction, as the configuration above does.

Related Guides #

• Actuarial Audit Trail Architecture — the tamper-evident ledger that must only ever see tokenized records
• Automating NAIC Filing Deadline Alerts in Python — surfacing quarantined batches before the deadline
• NAIC VM-20 Compliance Frameworks — the reserve pipeline this boundary feeds
• Validating Actuarial Input Schemas with Pydantic — the typed contract the classification matrix keys against
• OSFI Model Risk Management Guidelines — data-residency and encryption constraints for cross-border filers

Up one level: Regulatory Architecture & Compliance Mapping