Actuarial Validation & Filing Automation

Building Secure Audit Logs for Regulatory Submissions

Once a reserving run has been sealed into a hash-chained ledger, the remaining engineering problem is narrow but unforgiving: write that ledger to durable storage without exhausting memory on multi-gigabyte stochastic output, then transmit the examiner-ready package to a regulator so that a transient network failure, a rate limit, or a schema rejection can never leave the filing half-delivered or the cryptographic seal broken. This page shows the durable writer and the idempotent submission path that carry a completed trail across that last mile. It is the transmission stage of the Actuarial Audit Trail Architecture cluster, which owns the in-memory chaining logic; here we assume the chain already exists and focus on persisting and delivering it under production load.

Durable write and idempotent submission path for a sealed audit trail Stage one persists the hash-chained trail with memory-safe chunked hashing and crash-safe flushed appends, ending in a sealed jsonl package. Stage two delivers that package under a deterministic idempotency key with jittered exponential-backoff retries and a dead-letter fallback, so a transient network failure can never duplicate the filing or break the seal. Stage 1 — durable write (memory-safe seal) Stage 2 — idempotent submission (last mile) record() capture event + payload chunked SHA-256 4 MiB slices · O(1) memory append + flush buffering=0 · crash-safe filing_seal head_hash ↔ submission_id sealed .jsonl examiner-ready package sealed package idempotency_key SHA-256(bytes ‖ id) transport.send() header: Idempotency-Key 2xx ack? delivered Regulator records the filing once a retry is recognised as a duplicate attempts left? jittered backoff min(base·2ⁿ, 60 s) dead-letter queue metadata · manual replay, same key yes no ack yes no resend · same key

The Specific Problem: Memory-Safe Writes and Delivery That Cannot Lose the Seal

The in-memory ledger described upstream is elegant for reasoning about integrity, but a real valuation emits entries alongside deterministic and stochastic reserves computed over hundreds of thousands of policies. Buffering an entire scenario payload into RAM to hash it will trigger a garbage-collection pause or an out-of-memory kill mid-run — and a run that dies after computing the reserve but before sealing the trail is a filing you cannot defend. Two disciplines solve this. First, hash and write each entry in bounded chunks with zero-buffered, immediately flushed I/O, so peak memory never scales with payload size. Second, decouple the local seal from the remote submission with a write-ahead pattern and idempotent retries, so the network is allowed to fail without ever corrupting or duplicating the sealed record. Direct policyholder identifiers must already be tokenized before any of this runs, per Data Security & PII Boundaries for Filing Systems — an immutable, retained log is the worst possible place to bake in raw PII.

Minimal Working Example

The SecureAuditLogger below streams each entry to an append-only file, chains it into the running SHA-256, and rotates files at a size threshold. A separate submit_sealed_package function delivers the finished file with bounded, jittered retries keyed on a deterministic idempotency token.

import hashlib
import json
import time
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Optional

GENESIS_HASH = b"\x00" * 32


class SecureAuditLogger:
    def __init__(self, log_dir: Path, chunk_bytes: int = 4_194_304,
                 rotate_bytes: int = 100_000_000) -> None:
        self.log_dir = log_dir
        self.chunk_bytes = chunk_bytes
        self.rotate_bytes = rotate_bytes
        self.prev_hash = GENESIS_HASH
        self.sequence_id = 0
        self._file: Optional[Any] = None
        self.log_dir.mkdir(parents=True, exist_ok=True)
        self._open_next_file()

    def _open_next_file(self) -> None:
        if self._file:
            self._file.flush()
            self._file.close()
        stamp = time.strftime("%Y%m%d_%H%M%S")
        path = self.log_dir / f"audit_{stamp}_{self.sequence_id:06d}.jsonl"
        self._file = open(path, "ab", buffering=0)

    def _chunked_sha256(self, data: bytes) -> str:
        hasher = hashlib.sha256()
        for start in range(0, len(data), self.chunk_bytes):
            hasher.update(data[start:start + self.chunk_bytes])
        return hasher.hexdigest()

    def record(self, event_type: str, valuation_date: str,
               payload: dict, actor: str = "valuation-service") -> str:
        self.sequence_id += 1
        payload_bytes = json.dumps(
            payload, sort_keys=True, separators=(",", ":")
        ).encode("utf-8")
        entry = {
            "seq": self.sequence_id,
            "ts": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
            "event": event_type,
            "valuation_date": valuation_date,
            "actor": actor,
            "prev_hash": self.prev_hash.hex(),
            "payload_hash": self._chunked_sha256(payload_bytes),
            "size_bytes": len(payload_bytes),
        }
        entry_line = json.dumps(entry, separators=(",", ":")).encode("utf-8") + b"\n"
        if self._file.tell() > self.rotate_bytes:
            self._open_next_file()
        self._file.write(entry_line)
        self._file.flush()
        self.prev_hash = hashlib.sha256(entry_line).digest()
        return self.prev_hash.hex()

    def seal_filing(self, submission_id: str) -> str:
        """Bind the current chain head to a submission and close the file."""
        head = self.record(
            "filing_seal", datetime.now(timezone.utc).date().isoformat(),
            {"submission_id": submission_id, "head_hash": self.prev_hash.hex()},
        )
        self._file.flush()
        self._file.close()
        return head
import random


def idempotency_key(package_path: Path, submission_id: str) -> str:
    """Deterministic key so a retry is never treated as a new submission."""
    digest = hashlib.sha256()
    with open(package_path, "rb") as fh:
        for block in iter(lambda: fh.read(4_194_304), b""):
            digest.update(block)
    digest.update(submission_id.encode("utf-8"))
    return digest.hexdigest()


def submit_sealed_package(package_path: Path, submission_id: str, transport,
                          max_attempts: int = 6, base_delay: float = 1.5,
                          dead_letter_dir: Path = Path("./dlq")) -> bool:
    key = idempotency_key(package_path, submission_id)
    for attempt in range(1, max_attempts + 1):
        try:
            transport.send(package_path, headers={"Idempotency-Key": key})
            return True
        except (ConnectionError, TimeoutError) as exc:
            if attempt == max_attempts:
                dead_letter_dir.mkdir(parents=True, exist_ok=True)
                meta = {"submission_id": submission_id, "idempotency_key": key,
                        "last_error": repr(exc), "attempts": attempt}
                (dead_letter_dir / f"{submission_id}.json").write_text(
                    json.dumps(meta, sort_keys=True))
                return False
            sleep_s = min(base_delay * 2 ** (attempt - 1), 60.0)
            sleep_s += random.uniform(0, sleep_s * 0.25)  # jitter
            time.sleep(sleep_s)
    return False

Block-by-Block Walkthrough

  • Chunked hashing (_chunked_sha256). The payload is fed to the hasher in 4 MiB slices rather than hashed in a single call over a giant buffer. Peak memory is bounded by the chunk size, not by the size of a stochastic reserve payload, so a run that emits a 2 GB scenario summary hashes in constant memory. When a payload is genuinely large, store it in object storage and record only its digest and URI in the entry — keep the ledger line lean.
  • Zero-buffered append and flush. Opening with buffering=0 and calling flush() after every write means the entry is on disk the instant record() returns. If the process is killed on the next line, the sealed entry survives. This is the write-ahead property that makes the trail crash-safe.
  • Chain continuity (prev_hash). Each entry hashes the previous entry’s full serialized line, so the durable file carries the same tamper-evident chain as the in-memory ledger. seal_filing writes a terminal filing_seal event that binds the chain head to the submission identifier — this is the anchor an examiner reconciles against.
  • Deterministic idempotency key. The submission key is a SHA-256 over the package bytes plus the submission ID, not a random UUID. If the first send succeeds but the acknowledgement is lost and the client retries, the regulator’s endpoint sees the same key and treats the second call as a duplicate rather than a second filing.
  • Bounded jittered backoff and dead-letter fallback. Retries grow exponentially, cap at 60 seconds, and add random jitter so a fleet of clients recovering from an outage does not synchronize into a thundering herd. After the final attempt the package is moved to a dead-letter directory with full diagnostic metadata, so a sustained outage quarantines the filing for manual replay instead of silently dropping it. The realistic variable names — valuation_date, submission_id, head_hash — keep the log self-describing for the reviewer.

Edge Cases and Production Hardening

  • Non-deterministic serialization breaks the seal. Unsorted dictionary keys or locale-dependent float formatting make logically identical payloads hash differently, which surfaces later as a false tamper alarm. The sort_keys=True canonical dump and explicit UTC timestamps close this; never hash a Python dict’s default repr. This is the same contract enforced upstream by Schema Validation with Pydantic & Great Expectations.
  • Rotation mid-chain. When a file rotates at the size threshold, the chain must continue across the boundary — prev_hash is instance state, not file state, so it survives rotation. Verify this explicitly: concatenate the rotated files in sequence order and re-walk the chain end to end. If a downstream tool ever sorts files lexically, the zero-padded sequence in the filename keeps the order correct.
  • Concurrent writers fork the chain. Two workers sharing one SecureAuditLogger will both read the same prev_hash and append siblings, splitting the chain. For the parallel runs described in Async Batch Processing for Large Models, give each worker its own sub-log and chain the heads into a parent log at the join, so no global append lock is required.
  • A seal that never confirms. If submit_sealed_package exhausts its retries, the dead-letter entry — not a swallowed exception — is what lets an operator replay the exact same package under the exact same idempotency key once connectivity returns, with zero risk of a double filing. Pair this with the deadline monitoring in Automating NAIC Filing Deadline Alerts in Python so a quarantined filing raises an alarm before the statutory window closes.

Compliance Note

The durable writer and its seal exist to satisfy the reproducibility and non-repudiation expectations that VM-31 (the PBR Actuarial Report) attaches to principle-based reserves filed under NAIC VM-20 Compliance Frameworks: the filing_seal entry proves which sealed trail backs a given submission, and the chunked hashing keeps that proof affordable to generate even on large stochastic runs. The idempotent, dead-letter-backed transport maps to the change-control and record-retention discipline of OSFI E-23 and NIST SP 800-53 controls AU-9 (protection of audit information) and AU-10 (non-repudiation), and the six-year retention horizon of SEC Rule 17a-4(f) applies to the written package once it lands in Write-Once-Read-Many storage. Jurisdictional variants are covered under OSFI Model Risk Management Guidelines.

Up one level: Actuarial Audit Trail Architecture · Regulatory Architecture & Compliance Mapping