Policy Lapse & Surrender Assumption Engines

A policy lapse and surrender assumption engine is the deterministic subsystem that turns experience studies, surrender-charge schedules, and forward economic scenarios into the decrement rates that drive statutory reserves, capital, and pricing. This guide shows how to build one in Python so that every lapse rate applied to an in-force cohort is strongly typed, version-pinned, reproducible, and traceable to an approved assumption memorandum — the standard examiners now expect under principle-based reserving.

The Problem This Engine Solves #

Lapse and surrender behavior is the single most leveraged behavioral input on a life or annuity balance sheet, and it is also the least stable. A universal life block priced on a flat 4% lapse can move tens of basis points of reserve adequacy when policyholders respond to a rising-rate environment, and a variable annuity with rich living benefits behaves in the opposite direction — low utilization lapse that concentrates tail risk. Because the assumption is dynamic, it cannot be modeled as a static table; it must be modeled as a function of duration, surrender-charge position, and in-the-moneyness of the policyholder option.

The regulatory framing is explicit. Under NAIC VM-21, variable annuity reserves must reflect dynamic policyholder behavior including a prescribed relationship between lapse and the in-the-moneyness of guarantees, and the assumptions must sit inside prudent-estimate margins. VM-20 Section 9 requires that lapse and surrender assumptions for life PBR be anchored to a company experience study, graded to an industry basis where credibility is thin, and documented in the actuarial memorandum. The professional standards layered on top — ASOP 52 for principle-based reserves and ASOP 25 for credibility — require that the method used to blend company and industry data be reproducible and defensible. A hard-coded lapse table buried in a projection model satisfies none of this: it cannot show its source, its margin, or its version history.

This engine solves that by enforcing three properties. First, every assumption input is validated at the ingestion boundary before a single multiplier is applied. Second, the dynamic behavior is expressed as an explicit, unit-tested formula rather than an opaque lookup. Third, every projection run emits an immutable fingerprint that ties the output back to the exact assumption snapshot, so the number in the reserve model can be reconciled to the number in the memorandum. It is the behavioral counterpart to the Assumption Validation & Rule Engine Design control plane that governs all decrement inputs.

Architecture at a Glance #

The engine is a linear pipeline: typed input, a hierarchical rule matrix that resolves the base rate per cohort, a dynamic overlay that adjusts for the economic scenario, a drift check against emerging experience, and a filing-synchronization step that maps the validated output to the reporting schema. The base-rate lookup is decoupled from the dynamic overlay so that a change to the surrender-charge grid does not require touching the interest-sensitivity logic.

Prerequisites #

Before implementing the engine, the reader should understand the surrounding pipeline and have the following in place:

• Python packages. pydantic>=2.0 for schema enforcement, numpy>=1.24 for vectorized decrement math, pandas>=2.0 or polars for cohort framing, and pytest plus hypothesis for property-based validation.
• Data contracts. An in-force extract keyed by policy_id with issue_date, valuation_date, product_code, policy_duration, surrender_charge_year, account_value, and credited_rate. A separate assumption table keyed by product_code and policy_duration supplying base_lapse_rate.
• Regulatory context. Familiarity with the prudent-estimate/margin structure of VM-20 Section 9 and the dynamic-behavior requirement of VM-21.
• Upstream and sibling systems. Base rates must be harmonized with the Mortality & Morbidity Rate Validation engine so combined decrements never exceed unity; the dynamic overlay consumes scenarios from Economic Scenario Mapping & Yield Curve Alignment; and the recalibration loop is governed by Dynamic Threshold Tuning for Assumption Drift. Typed inputs arrive pre-cleaned from the Actuarial Model Ingestion & Testing Workflows pipeline.

Core Implementation #

The engine starts with a schema that rejects malformed assumptions at the boundary. Using Pydantic schema enforcement means an out-of-range duration, a future-dated effective date, or a lapse rate outside [0, 1] fails loudly at ingestion rather than silently corrupting a quarter of reserves.

from datetime import date
from pydantic import BaseModel, Field, field_validator


class LapseAssumption(BaseModel):
    """One row of a version-pinned lapse basis, validated at ingestion."""

    product_code: str = Field(pattern=r"^[A-Z]{2,4}\d{0,2}$")
    policy_duration: int = Field(ge=1, le=121)
    base_lapse_rate: float = Field(ge=0.0, le=1.0)
    surrender_charge_year: int = Field(ge=0, le=20)
    effective_date: date
    source_authority: str          # e.g. "2024 company study, VM-20 S9"
    version_hash: str              # SHA-256 of the source basis file

    @field_validator("effective_date")
    @classmethod
    def not_future_dated(cls, v: date) -> date:
        if v > date.today():
            raise ValueError("assumption effective_date cannot be in the future")
        return v

The base rate alone is not the assumption. The behavioral core is a dynamic overlay that scales the base lapse by the policyholder’s incentive to surrender. That incentive is the in-the-moneyness of the option — here proxied by the spread between the prevailing market rate and the rate credited to the policy. Above a tolerance band tau, disintermediation lapse rises; the response is scaled by an elasticity beta and capped so a single scenario cannot drive lapse to implausible levels.

$$q^{\text{lapse}}_{\text{dyn}}(t) = q^{\text{lapse}}_{\text{base}}(t) \cdot \left[\,1 + \beta \cdot \max\!\left(0,\; \frac{r_{\text{mkt}}(t) - r_{\text{cr}}(t)}{r_{\text{cr}}(t)} - \tau \right)\right]$$

The projection is fully vectorized so that a portfolio of tens of millions of policies evaluates in a single pass rather than a Python loop. Working in float32 keeps the in-force array memory-resident; the final clip enforces the regulatory ceiling.

import numpy as np


def project_dynamic_lapse(
    base_lapse_rate: np.ndarray,   # float32, per policy
    credited_rate: np.ndarray,     # float32, per policy
    market_rate: float,            # scenario forward rate at duration t
    beta: float = 1.35,            # disintermediation elasticity
    tau: float = 0.005,            # 50 bps indifference band
    lapse_cap: float = 0.60,       # regulatory ceiling on dynamic lapse
) -> np.ndarray:
    """Vectorized dynamic lapse per VM-21-style in-the-moneyness logic."""
    itm = (market_rate - credited_rate) / np.maximum(credited_rate, 1e-4)
    incentive = np.maximum(0.0, itm - tau)
    dynamic = base_lapse_rate * (1.0 + beta * incentive)
    return np.clip(dynamic, 0.0, lapse_cap).astype(np.float32)

At the end of the surrender-charge period, lapse behavior is discontinuous — the well-documented shock lapse as policyholders who were held by the charge exit. This is modeled as a multiplicative bump keyed on surrender_charge_year == 0, applied after the dynamic overlay so the two effects compound correctly rather than being averaged away.

def apply_shock_lapse(
    dynamic_lapse: np.ndarray,
    surrender_charge_year: np.ndarray,   # int, 0 = charge expired this year
    shock_multiple: float = 2.75,
    lapse_cap: float = 0.60,
) -> np.ndarray:
    shocked = np.where(
        surrender_charge_year == 0,
        dynamic_lapse * shock_multiple,
        dynamic_lapse,
    )
    return np.clip(shocked, 0.0, lapse_cap).astype(np.float32)

The two overlays compose into a lapse curve whose shape no static table can capture: a base decrement that the dynamic in-the-moneyness multiplier lifts as market rates outrun the credited rate, and a discontinuous spike at the duration the surrender charge expires. The diagram below traces both effects against policy duration for a seven-year surrender-charge product.

Configuration and Tuning #

The elasticity beta, indifference band tau, shock multiple, and cap are not code constants — they are governed assumptions in their own right, each sourced and margined. Externalizing them into a version-controlled configuration file keeps the actuarial basis auditable and lets compliance adjust behavior without a code deploy.

# lapse_config.yaml (loaded and hashed at engine init)
# beta / tau / shock_multiple carry a prudent-estimate margin over
# best-estimate per VM-20 Section 9; the cap is a hard regulatory guardrail.
UL01:
  beta: 1.35            # best-estimate 1.10 + 0.25 margin
  tau: 0.005
  shock_multiple: 2.75  # best-estimate 2.40 + margin
  lapse_cap: 0.60
FA05:                   # fixed annuity, higher rate sensitivity
  beta: 2.10
  tau: 0.0025
  shock_multiple: 3.20
  lapse_cap: 0.75

Two tuning rules matter in production. First, the margin direction must follow the reserve’s sensitivity: for products where higher lapse increases reserves (guarantee-rich VAs), the prudent estimate margins lapse down; for products where higher lapse decreases reserves, it margins up. Encoding the direction per product_code prevents a margin from accidentally reducing conservatism. Second, beta and shock_multiple should be recalibrated only through the drift-governed loop below, never by hand-editing production config, so that every change carries a triggering metric and an approver.

Step-by-Step Walkthrough #

1. Load and hash the basis. Read lapse_config.yaml and the assumption table, compute a SHA-256 version_hash over the raw bytes, and stamp it onto every LapseAssumption.
2. Validate at the boundary. Parse each row through the LapseAssumption model; route any ValidationError to a quarantine log rather than dropping it silently.
3. Resolve base rates. Join the in-force extract to the assumption table on product_code and policy_duration to produce the per-policy base_lapse_rate array.
4. Harmonize decrements. Confirm that base_lapse_rate + mortality_rate + morbidity_rate <= 1.0 for every policy, using rates from the mortality engine.
5. Apply the dynamic overlay. Call project_dynamic_lapse with the scenario’s market_rate and the per-policy credited_rate.
6. Apply shock lapse. Bump policies whose surrender charge expires this duration via apply_shock_lapse.
7. Check drift. Score projected versus actual lapse with the Population Stability Index; if it breaches tolerance, flag for recalibration instead of publishing.
8. Fingerprint and file. Hash the output array with the input version_hash, write the immutable audit record, and map the validated rates to the filing schema.

Validation and Testing #

Correctness here is not “the code runs” — it is “the number is reproducible and inside its margin.” Three layers of testing enforce that.

Property-based unit tests assert the invariants the formula must never violate: dynamic lapse is monotonic non-decreasing in the market rate, always at least the base rate when in-the-moneyness is positive, and never exceeds the cap. hypothesis generates thousands of rate combinations to probe the boundaries a hand-written case would miss.

import numpy as np
from hypothesis import given, strategies as st


@given(
    base=st.floats(0.0, 0.5),
    credited=st.floats(0.01, 0.08),
    market=st.floats(0.0, 0.15),
)
def test_dynamic_lapse_invariants(base, credited, market):
    out = project_dynamic_lapse(
        np.array([base], np.float32),
        np.array([credited], np.float32),
        market,
    )
    assert 0.0 <= out[0] <= 0.60          # respects the regulatory cap
    if market > credited:
        assert out[0] >= base - 1e-6      # in-the-money never lapses below base

Drift assertions guard the assumption over time. The Population Stability Index compares the projected lapse distribution against emerging actual experience across duration buckets:

$$\mathrm{PSI} = \sum_{i=1}^{B} \left(a_i - e_i\right)\,\ln\!\frac{a_i}{e_i}$$

A conventional reading is that PSI < 0.10 is stable, 0.10–0.25 warrants review, and > 0.25 forces recalibration through the governed loop. Wiring this threshold into the pipeline turns assumption maintenance from a periodic manual study into a continuous control, as detailed in Dynamic Threshold Tuning for Assumption Drift.

Audit-log assertions verify governance, not math: every published run must have a version_hash, a non-empty source_authority, and a recorded approver. A CI check that fails the build when any of these is missing is what makes the pipeline examiner-ready rather than merely correct.

Failure Modes and Gotchas #

• Rule-ordering conflicts. Applying the shock bump before the dynamic overlay averages two effects that should compound, understating end-of-charge lapse. The order in the walkthrough — dynamic first, shock second, clip last — is load-bearing.
• float32 overflow at the cap. Multiplying a high base rate by a large beta incentive before clipping can produce values well above 1.0 mid-computation; always clip in the same dtype and never feed an unclipped rate into a downstream survival product.
• Divide-by-zero on credited rate. A zero or missing credited_rate makes in-the-moneyness explode. The np.maximum(credited_rate, 1e-4) floor is mandatory, and a missing rate should trigger the fallback chain, not a silent zero.
• Margin sign errors. A prudent-estimate margin applied in the wrong direction for a guarantee-rich product reduces conservatism and can understate the reserve — exactly the finding that turns a routine exam into a remediation order. Encode the margin direction per product and test it.
• Silent fallback on sparse cohorts. New product lines with no experience must degrade explicitly — cohort interpolation, then product-line average, then a prescribed conservative default — with each tier flagged in the output metadata, never blended invisibly into a company-wide mean.

For a complete, memory-optimized reference implementation with cryptographic audit trails and multi-dimensional rule evaluation, see Building a Python Rule Engine for Policy Lapse Assumptions.

Related Guides #

• Building a Python Rule Engine for Policy Lapse Assumptions — the full engine implementation.
• Mortality & Morbidity Rate Validation — harmonizing lapse with the other decrement tables.
• Economic Scenario Mapping & Yield Curve Alignment — sourcing the market rates the dynamic overlay consumes.
• Dynamic Threshold Tuning for Assumption Drift — the PSI-governed recalibration loop.

Up a level: Assumption Validation & Rule Engine Design

---

Regulatory references: NAIC Valuation Manual VM-20 (life PBR) and VM-21 (variable annuities); Actuarial Standards Board ASOP No. 52 (principle-based reserves) and ASOP No. 25 (credibility).