Actuarial Validation & Filing Automation

Checkpointing and Resuming Failed Actuarial Batch Runs

When a valuation batch dies at hour nine of a twelve-hour run, the fix is not to start over but to resume only the chunks that never finished — this page shows a per-chunk completion manifest that lets a crashed reserve run pick up exactly where it stopped, idempotently and reproducibly. It is a focused technique inside Async Batch Processing for Large Models, the discipline that splits a seriatim valuation into concurrent work, and it pairs naturally with Implementing Asyncio for High-Volume Actuarial Batch Jobs.

The Problem

A quarter-end valuation over several million policies is partitioned into chunks — reserve groups, issue-year bands, or product cohorts — and each chunk runs for minutes. Somewhere in that long sequence a worker hits an out-of-memory kill, a database timeout, or a bad assumption row, and the whole job aborts. Restarting from zero wastes the hours already spent and, worse, risks a second failure before the deadline. But a naive restart is also dangerous the other way: re-running a chunk that already wrote its reserves can double-count, corrupt aggregates, or silently change a figure an examiner will later reconcile. What the batch needs is a durable record of which chunks truly completed and what each produced, so a resumed run does exactly the outstanding work and nothing it has already done. VM-20 Section 7 expects a company to reproduce its principle-based reserves on demand; a batch that cannot say precisely which chunks contributed to a filed number cannot meet that expectation.

A Minimal Working Example

The pattern is a manifest: a JSON file mapping each chunk_id to its status and the hash of the output it produced. Before running a chunk the worker consults the manifest; after a chunk succeeds it records the result atomically. On resume, only chunks whose status is not done are executed. The code runs as-is against a stub valuation function:

from __future__ import annotations

import hashlib
import json
import os
import tempfile
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from pathlib import Path


@dataclass(frozen=True)
class ChunkRecord:
    chunk_id: str
    status: str          # "pending" | "running" | "done" | "failed"
    output_hash: str | None
    reserve_total: float | None
    completed_at_utc: str | None


class BatchManifest:
    """Durable per-chunk completion state for a resumable valuation run."""

    def __init__(self, path: Path):
        self.path = path
        self.records: dict[str, ChunkRecord] = {}
        if path.exists():
            raw = json.loads(path.read_text())
            self.records = {k: ChunkRecord(**v) for k, v in raw.items()}

    def is_done(self, chunk_id: str) -> bool:
        rec = self.records.get(chunk_id)
        return rec is not None and rec.status == "done"

    def mark(self, record: ChunkRecord) -> None:
        self.records[record.chunk_id] = record
        self._atomic_write()

    def _atomic_write(self) -> None:
        # Write to a temp file in the same directory, then atomically rename,
        # so a crash mid-write can never leave a half-serialized manifest.
        payload = {k: asdict(v) for k, v in self.records.items()}
        fd, tmp = tempfile.mkstemp(dir=self.path.parent, suffix=".tmp")
        with os.fdopen(fd, "w") as fh:
            json.dump(payload, fh, sort_keys=True, indent=2)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp, self.path)


def hash_reserves(rows: list[dict]) -> str:
    payload = json.dumps(rows, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()


def run_chunk(chunk_id: str, policies: list[dict]) -> list[dict]:
    # Stand-in for a real seriatim valuation over one reserve group.
    return [
        {"policy_id": p["policy_id"], "reserve": round(p["face_amount"] * 0.62, 2)}
        for p in policies
    ]


def process_batch(chunks: dict[str, list[dict]], manifest_path: Path) -> None:
    manifest = BatchManifest(manifest_path)
    for chunk_id in sorted(chunks):                    # deterministic order
        if manifest.is_done(chunk_id):
            continue                                   # already reconciled — skip
        try:
            rows = run_chunk(chunk_id, chunks[chunk_id])
            manifest.mark(ChunkRecord(
                chunk_id=chunk_id,
                status="done",
                output_hash=hash_reserves(rows),
                reserve_total=round(sum(r["reserve"] for r in rows), 2),
                completed_at_utc=datetime.now(timezone.utc).isoformat(),
            ))
        except Exception:
            manifest.mark(ChunkRecord(chunk_id, "failed", None, None, None))
            raise


if __name__ == "__main__":
    chunks = {
        "grp_2021_term": [{"policy_id": "P1", "face_amount": 500_000.0}],
        "grp_2022_term": [{"policy_id": "P2", "face_amount": 250_000.0}],
        "grp_2023_ul":   [{"policy_id": "P3", "face_amount": 750_000.0}],
    }
    process_batch(chunks, Path("valuation_manifest.json"))
Resumable valuation batch: manifest gate skips done chunks and reruns only the unfinished ones Deterministically ordered chunks reach a manifest gate that reads each chunk_id status. Done chunks are skipped; unfinished chunks run the valuation and are atomically committed back to the manifest, which then feeds a reconciled aggregate. Ordered chunks sorted chunk_id Manifest gate status lookup Skip status=done Run valuation atomic commit + hash Reconciled aggregate all chunks accounted
The manifest gate routes each chunk to a skip or a rerun, and only a fully covered manifest reaches aggregation.

How It Works, Block by Block

The manifest is the single source of truth about progress. BatchManifest loads whatever state survives from a prior run and exposes exactly one question that matters on resume — is_done(chunk_id) — and one mutation, mark. Progress lives in a file the operating system will still hold after a process is killed, not in memory that dies with the worker. Because the record for each chunk carries an output_hash and a reserve_total, the manifest is not merely a to-do list; it is evidence of what each completed chunk produced.

Completion is committed atomically or not at all. _atomic_write serializes the whole manifest to a temporary file in the same directory, flushes and fsyncs it, then uses os.replace to swap it into place. On POSIX filesystems that rename is atomic, so a crash during the write leaves either the old manifest or the new one — never a truncated JSON document that would fail to parse on the next start. This is what makes the “done” state trustworthy: a chunk is marked done only after its result is durably on disk.

Chunk ordering is deterministic on purpose. process_batch iterates sorted(chunks), so the first run and every resumed run visit chunks in the same sequence. Deterministic ordering is what lets a resumed run reconcile against the original: the set of done chunks is a stable prefix-plus-gaps of a known ordering, and the final aggregate does not depend on which worker happened to reach a chunk first. Combined with the output hash, a resumed run can prove that the chunks it did not re-execute produced the same figures the interrupted run recorded.

Skipping is how idempotency is enforced. The if manifest.is_done(chunk_id): continue guard means a chunk that already wrote its reserves is never run twice. Re-executing valuation logic is not free of side effects — it may append to an output table or increment a counter — so the manifest, not the valuation function, is where the batch decides whether work is needed. The result is that running process_batch again after any failure is safe: it converges on a complete run without redoing completed work.

Edge Cases and Production Hardening

Partial writes from the chunk itself, not just the manifest. The atomic manifest write protects the progress record, but the chunk’s own reserve output can still be half-written when a worker dies mid-flush. Guard against a “manifest says done, output is truncated” mismatch by writing chunk output atomically too and verifying the stored hash on resume:

def verify_or_reset(manifest: BatchManifest, chunk_id: str,
                    reload_rows) -> None:
    rec = manifest.records.get(chunk_id)
    if rec is None or rec.status != "done":
        return
    if hash_reserves(reload_rows(chunk_id)) != rec.output_hash:
        # Output on disk disagrees with the sealed hash — treat as unfinished.
        manifest.mark(ChunkRecord(chunk_id, "pending", None, None, None))

A chunk whose reloaded output no longer matches its recorded output_hash is demoted to pending and will be re-run, closing the gap between “recorded as done” and “actually reproducible.”

Deadline pressure and the failed-fast tradeoff. The example re-raises on the first chunk failure, which surfaces problems immediately but stops the batch. Near a filing deadline you may prefer to mark the chunk failed, continue with the rest, and quarantine the failures for a targeted retry — turning one poisoned reserve group into a small remediation task rather than a blocked run. Whichever policy you choose, record it: a chunk left in failed must never be silently treated as done by the aggregation step, or the filed total will be missing a cohort.

Non-deterministic chunk boundaries defeat resume. If the chunking itself is non-deterministic — for example, partitioning by a hash of a wall-clock timestamp, or reading policies in database-return order — then a resumed run may draw different chunk boundaries and the manifest’s chunk_id keys no longer line up. Derive each chunk_id from stable business keys (reserve group, issue-year band, product code) so the same policy always lands in the same chunk across runs. This is also what lets the durable record feed the Actuarial Audit Trail Architecture, where the manifest becomes evidence of exactly which cohorts contributed to a filed reserve.

Compliance Note

Resumability is not just an operational convenience; it is a reproducibility control. VM-20 Section 7 and the broader principle-based reserving framework require a company to reproduce its reserves and demonstrate that reported figures follow deterministically from a pinned set of assumptions and scenarios. A checkpoint manifest that records the output_hash and reserve_total of every completed chunk gives the appointed actuary a mechanical answer to “which pieces made this number, and can you regenerate them?” — the same reconstruction the VM-31 PBR Actuarial Report expects. ASOP No. 56 (Modeling) likewise calls for controls that make a model’s runs traceable and reconcilable rather than assumed; a manifest gate that refuses to double-count and can prove which chunks it skipped is exactly such a control. The wider batch orchestration this fits into is described in Actuarial Model Ingestion & Testing Workflows.

Frequently Asked Questions

How is a resumable batch different from simply retrying the whole job?

A retry re-runs every chunk, wasting the hours already spent and risking a second failure before the deadline. A resumable batch consults a durable manifest and executes only the chunks not yet marked done, so completed reserve groups are skipped and the run converges toward completion instead of restarting from zero.

Why store an output hash in the checkpoint manifest and not just a done flag?

A bare done flag records that a chunk finished but not what it produced. Storing the SHA-256 hash of the chunk’s reserve output lets a resumed run verify that the figures on disk still match what was recorded, detect a partial or corrupted write, and demonstrate to an examiner that the chunks it skipped are genuinely reproducible.

What makes the resume operation idempotent?

Every chunk is guarded by an is_done check against the manifest before it runs, so a chunk that already wrote its reserves is never executed a second time. Because completion is committed atomically only after the result is durably on disk, running the batch again after any crash performs exactly the outstanding work and nothing already done.

Why does chunk ordering need to be deterministic?

Deterministic ordering, derived from stable business keys rather than wall-clock or database-return order, guarantees that the same policy always lands in the same chunk across runs. That lets a resumed run reconcile its skipped chunks against the interrupted run and produce an aggregate that does not depend on which worker reached a chunk first.

Up a level: Async Batch Processing for Large Models · Actuarial Model Ingestion & Testing Workflows

Regulatory references: NAIC Valuation Manual VM-20 (Requirements for Principle-Based Reserves for Life Products) and VM-31 (PBR Actuarial Report); Actuarial Standards Board ASOP No. 56 (Modeling).