AI for Insurance Professionals
Strategic · M2 · lesson 2 of 24 · queued
Preview — browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll →
Stand Up an AI Risk Inventory and Algorithm Registry - Required by Colorado Reg 10-1-1
📖
now learning

Stand Up an AI Risk Inventory and Algorithm Registry - Required by Colorado Reg 10-1-1

15 min

The algorithm registry - the system-of-record for every AI System a carrier or MGA operates - is the artifact that distinguishes a §4-compliant governance program from a memo. Colorado Reg 10-1-1 expanded effective October 15, 2025 to private passenger auto and health benefit plans in addition to life insurance, and the first annual compliance report is due July 1, 2026; the registry is what the report references. NAIC AISET Exhibit A (AI quantification) and Exhibit C (high-risk system detail) are extracted directly from the registry. NY DFS Circular Letter 2024-7, Connecticut MC-25-8, Nevada Bulletin 24-006, and emerging California / Texas / Florida guidance all converge on the same artifact: a structured, system-of-record inventory of every AI System with named owner, risk tier, last-tested date, bias-test status, drift-monitoring status, and version-control history. Carriers operating a registry as a spreadsheet survive the first DOI inquiry and fail the second; carriers operating a system-based registry with enforced fields, attestation cadence, and integration with vendor management absorb every regulatory cycle through operational refresh. This lesson is the registry build: the schema, the entry standards for three named models (Akur8 personal auto pricing, Shift fraud detection, Tractable photo estimate), the risk-tier discipline, the drift-and-fairness monitoring integration, and the version-control history that satisfies Colorado Reg 10-1-1 and NAIC AISET Exhibit A/C in a single artifact.

Why the Registry Is the Artifact - Not the Spreadsheet

Carriers approaching algorithm inventory for the first time typically build a spreadsheet: tab one is models in production, tab two is vendors, tab three is testing schedule. The spreadsheet survives the first internal review and fails the first external scrutiny because spreadsheets cannot enforce attestation cadence, version-control history, drift-threshold alerts, or integration with vendor scorecard refresh. A Colorado examiner asking "show me the last-updated registry entry for the personal auto pricing model" expects a system-generated timestamp from the registry; a spreadsheet timestamp can be edited by any user with file access. The evidentiary integrity of the registry depends on system-based controls.

The 2026 market for AI governance systems converges on three patterns: commercial platforms (Credo AI, Holistic AI, IBM watsonx.governance, Dataiku Govern, ModelOp Center) that specialize in algorithm registry and model lifecycle; ITSM-customized deployments (ServiceNow IRM, Jira with custom fields and workflows) that leverage existing enterprise infrastructure; and carrier-built systems (typical at larger carriers using internal MLOps platforms - Domino Data Lab, Databricks Unity Catalog, AWS SageMaker Model Registry - extended with governance fields). The choice depends on carrier scale and existing infrastructure; the discipline is consistent - required fields, enforced cadence, integrated workflow.

The registry feeds multiple regulatory artifacts simultaneously. Colorado Reg 10-1-1 compliance report (due July 1 annually) extracts the life and auto and health AI Systems and their bias-testing status. NAIC AISET Exhibit A (AI quantification - counts of AI Systems, decisioning categories, high-risk classifications) is a registry summary. Exhibit C (high-risk system detail - Tier 1 model documentation, testing results, third-party AI documentation) is registry depth. NY DFS Circular Letter 2024-7 proxy-test memos reference registry entries. State market-conduct exam information requests cite the registry by entry. One artifact, multiple uses; the carrier that built a clean registry produces every regulatory response through extraction.

The Registry Schema - Required Fields

Each registry entry contains a minimum field set the §4 program defines. The fields enforce the data the carrier needs to answer every regulatory question and the operational data the AI committee needs to manage the portfolio. Required fields at most mature carriers:

Identification: registry ID (e.g., AK-PL-AUTO-001), system name, vendor or internal-build flag, AI category (predictive model, generative model, agentic, deterministic-rules-AI-hybrid), business function (UW, claims, pricing, distribution, L&H, reserving, capital, fraud), line of business (personal auto, homeowners, commercial property, GL, WC, etc.), in-production date, retirement date if applicable.

Accountability: named accountable executive (CRO, CCO, Chief Actuary, etc. - by name not title), named algorithm owner (operational responsible party), named model validator (separate from owner for Tier 1), AI committee approval date and committee meeting reference.

Risk classification: tier assignment (1/2/3), classification rationale, consumer-adverse decision flag, FCRA adverse-action workflow integration flag, MHPAEA NQTL exposure flag (L&H), GLBA NPI handling flag, HIPAA PHI handling flag (L&H), affected protected classes per analysis, jurisdiction exposure.

Vendor lifecycle (third-party AI only): vendor name, contract effective date, BAA / DPA / sub-processor disclosure status, §4 vendor attestation date, SOC 2 Type II date and gap status, model-update notification clause, audit rights, exit clause, last vendor scorecard refresh date.

Testing and validation: last fairness test date and result, fairness test method (disparate impact ratio, equality of opportunity, calibration parity), last drift test date and result, drift metrics (PSI, KS, AUC trajectory), calibration metrics, last bias test for Tier 1 (Colorado SB 21-169 / Reg 10-1-1 quantitative methods, NY DFS proxy test method), ASOP No. 56 peer-review date (Tier 1), peer reviewer name.

Documentation: model card URL, training data lineage reference, feature list with protected-class proxy analysis, SHAP / PDP / ALE explainability artifacts, fairness test report URL, drift dashboard URL, latest attestation date and attestor name, prompt log location (generative AI), system prompt version (generative AI).

Version control: current version, prior versions with effective dates and retirement dates, change history with change type (training data refresh, hyperparameter change, feature addition or removal, threshold adjustment, vendor model update), associated AI committee approval if material.

Incident history: incident count by severity, last incident date, open remediation items, post-incident review status.

Registry Entry - Akur8 Personal Auto Pricing Model (Tier 1)

Sample registry entry for a personal-auto pricing model built in Akur8 and deployed in production for a $1B regional carrier's voluntary auto book in 14 states including Colorado:

Identification: AK-PL-AUTO-001 / Akur8 Personal Auto Pricing GLM v3.2 / vendor (Akur8) + internal-build hybrid (Akur8 platform with carrier-owned training data and feature engineering) / predictive model - GLM with elastic-net regularization / pricing / personal auto / in-production March 4, 2024 / current.

Accountability: Accountable executive - Jane Smith, Chief Actuary (FCAS, MAAA), attestation date March 15, 2026. Algorithm owner - Robert Chen, AVP Personal Lines Pricing (ACAS). Model validator - Emily Wong, Director Model Governance (FSA, dual-role with reserving validation). AI committee approval - January 12, 2024 (initial); March 10, 2026 (v3.2 refresh approval).

Risk classification: Tier 1. Rationale - consumer-adverse rating decision affecting policy premium, FCRA adverse-action workflow integration required, Colorado Reg 10-1-1 applicable (personal auto in scope post-October 2025), NY DFS Circular Letter 2024-7 applicable, multi-jurisdiction (14 states), protected classes per fairness analysis (race via BISG, ethnicity, gender, age, ZIP code as geographic proxy). Consumer-adverse flag - YES. FCRA workflow integration - YES (pre-notice and adverse-action letter triggered on premium impact above $X threshold). MHPAEA flag - N/A (P&C). GLBA NPI - YES (credit-based insurance score input where filed). HIPAA - N/A.

Vendor lifecycle: Akur8 SAS / contract effective January 8, 2024 / DPA + sub-processor disclosure (AWS US-East-1, OpenAI for in-platform LLM features not used in this model) / §4 vendor attestation March 8, 2026 / SOC 2 Type II March 2026 (no material gaps) / model-update notification 30 days / audit rights annual / exit clause 90-day transition with data return. Last scorecard refresh - March 25, 2026.

Testing and validation: Last fairness test - March 12, 2026 (quarterly cadence). Method - disparate impact ratio on rating relativities by race (BISG), gender, age band, urban/rural; equality-of-opportunity test on top-decile rating impact; calibration parity by protected class. Result - disparate impact ratio 0.94 (acceptable threshold 0.80-1.25); no material deviation by class. Drift test - March 28, 2026 (monthly). Drift metrics - PSI 0.07 (acceptable below 0.10), no calibration drift, AUC stable at 0.732. ASOP No. 56 peer review - completed January 2026 by external actuarial firm; review report filed in registry. NY DFS proxy test memo - refreshed February 2026.

Documentation: Model card URL (carrier intranet). Training data lineage - carrier policy database trailing 7 years, ISO Personal Auto rating data, MVR via LexisNexis, vehicle data via Verisk Vehicle. Feature list with proxy analysis - 47 features documented, BISG flag carried for monitoring not modeling. SHAP analysis - top 10 features ranked, no protected-class proxy in top 20. Latest attestation - March 15, 2026 by Chief Actuary.

Version control: v3.2 current (deployed March 4, 2026). v3.1 (June 2025 - March 2026; retired with current refresh; archived). v3.0 (January 2025 - June 2025). v2.x versions archived. Change from v3.1 to v3.2 - training data refresh through Q4 2025, one feature removed (state-specific vehicle-make indicator with proxy concern), AI committee approval March 10, 2026.

Incident history: Two incidents (one drift exceedance October 2025 with re-calibration response; one customer-disputed adverse-action notice March 2026 with FCRA workflow review - no §4 finding).

Registry Entry - Shift Fraud Detection Model (Tier 1)

Sample registry entry for a fraud-detection model operated by Shift Technology on the carrier's auto claims book:

Identification: SH-AUTO-FRD-001 / Shift Claims Fraud Detection v4.1 / vendor (Shift Technology) / predictive model - ensemble (gradient boosting + graph neural network for network analysis) / claims SIU / auto bodily injury and property damage / in-production November 2023 / current.

Accountability: Accountable executive - Michael Torres, Chief Claims Officer (AIC, CPCU), attestation date March 18, 2026. Algorithm owner - Sarah Patel, Director SIU Operations (FCLS, CFE). Model validator - Robert Chen, dual-role with pricing (cross-functional validator pool). AI committee approval - October 2023 (initial); ongoing annual reviews.

Risk classification: Tier 1. Rationale - consumer-adverse referral decision (SIU investigation impact on claim handling), §4 reason-code documentation required, NY DFS proxy-test applicable, multi-jurisdiction. Consumer-adverse flag - YES (referral). FCRA workflow integration - partial (referral is not adverse action by itself; adverse action triggered by subsequent denial decision). GLBA NPI - YES.

Vendor lifecycle: Shift Technology / contract effective October 2023 / DPA + sub-processor (AWS US-East-1, Google BigQuery for analytics) / §4 vendor attestation February 2026 / SOC 2 Type II February 2026 / model-update notification 30 days / audit rights annual / exit clause 60-day with data return. Last scorecard refresh - March 22, 2026 (concentration analysis - Shift at 19% of fraud-detection critical decision flow; below 25% limit).

Testing and validation: Last fairness test - March 14, 2026 (quarterly). Method - referral rate by demographic proxies (BISG for race, surname-based language proxy for ethnicity, urban/rural geocode), false-positive rate by class, network-analysis aggregation analysis. Result - referral rate ratio 1.08 across classes (acceptable); false-positive rate by Hispanic-surname cluster flagged at 1.22 in October 2025, remediated in v4.0 deployment November 2025, re-tested March 2026 at 1.04. Drift test - March 29, 2026 (monthly). PSI 0.09, AUC stable at 0.81. Peer review - completed by SIU governance committee December 2025.

Documentation: Model card. Training data - carrier claims history 5 years, ISO ClaimSearch hits, public records via LexisNexis. Feature list - 134 features including network-analysis derived features; protected-class proxy analysis documented. SHAP - top 20 features, network-centrality measures dominant, no surname-derived feature in feature list as of v4.0.

Version control: v4.1 current (March 2026 - training data refresh, threshold adjustment for false-positive control). v4.0 (November 2025 - March 2026; deployed after Hispanic-surname cluster remediation). v3.x versions archived with documented retirement rationale.

Incident history: One material incident October 2025 (Hispanic-surname false-positive cluster) with full §4 incident response activation, customer remediation, model rollback to v3.4 pending v4.0 deployment, post-incident review filed with AI committee December 2025. State DOI notification - Colorado DOI notified November 2025 per Reg 10-1-1 incident-reporting expectation; no enforcement action; documented in incident file.

Registry Entry - Tractable Photo Estimate Model (Tier 1)

Sample registry entry for Tractable's photo-damage assessment model used on auto first-party physical damage claims:

Identification: TR-AUTO-PD-001 / Tractable AI Estimating v5.3 / vendor (Tractable) / predictive model - computer vision (CNN-based damage classification + cost estimation) / claims estimating / auto first-party physical damage / in-production September 2024 / current.

Accountability: Accountable executive - Michael Torres, Chief Claims Officer, attestation March 18, 2026. Algorithm owner - David Kim, Director Auto Claims Estimating (AIC). Model validator - third-party (Verisk consulting engagement Q1 2026). AI committee approval - August 2024 (initial); March 2026 (v5.3 approval).

Risk classification: Tier 1. Rationale - consumer-adverse impact on total-loss decisions and ACV determination, §4 reason-code documentation required, jurisdiction-sensitive (total-loss thresholds vary by state), multi-jurisdiction. Consumer-adverse flag - YES (estimate drives indemnity payment). FCRA - N/A (no consumer-report data inputs).

Vendor lifecycle: Tractable / contract effective August 2024 / DPA + sub-processor (AWS US-East-1, Google Cloud Vision for OCR fallback) / §4 vendor attestation January 2026 / SOC 2 Type II December 2025 / model-update notification 30 days / audit rights annual / exit clause 90-day. Last scorecard refresh - March 28, 2026 (concentration - Tractable at 31% of photo-estimating critical decision flow; above 25% limit, AI committee documented acceptance with mitigation plan to onboard CCC as parallel-capability vendor in 2026 H2).

Testing and validation: Last fairness test - March 15, 2026 (quarterly). Method - estimate variance vs. field-adjuster ground-truth by demographic proxies, total-loss-threshold incidence by demographic. Result - estimate variance within 4.2% across classes (acceptable below 5%), total-loss-threshold incidence ratio 0.97. Drift test - March 30, 2026 (monthly). PSI 0.06, ground-truth variance trajectory stable. ASOP No. 56 peer review - Q1 2026 external validation completed.

Documentation: Model card. Training data - Tractable proprietary auto-damage corpus + carrier-specific photo-and-estimate pairs from prior 18 months. Feature analysis - CV model with feature attribution via Grad-CAM; no demographic features ingested. Explainability - Grad-CAM heatmap per estimate accessible to adjuster.

Version control: v5.3 current (March 2026 - model architecture refresh, training data refresh through January 2026). v5.2 (October 2025 - March 2026). v5.1 archived.

Incident history: Three incidents (asphalt-vs-architectural-shingle confusion December 2024 - predicate to Tractable v5.2 architecture refresh; two single-claim disputes resolved via field-adjuster override in 2025; no §4 findings).

Risk-Tier Discipline and the Drift-and-Fairness Integration

The registry's risk-tier classifications drive testing cadence. Without enforced tier discipline, every model defaults to whatever cadence the algorithm owner prefers - typically the minimum the resource allows - and the §4 program's testing differentiation collapses. Enforced tier discipline means: Tier 1 systems are flagged in the registry with mandatory quarterly fairness review, mandatory monthly drift review, mandatory ASOP No. 56 peer review if actuarial in scope, and mandatory dual attestation. The registry generates the testing calendar from the entry; the testing calendar flows to the testing team; the testing results flow back to the registry entry. The loop is enforced by the system, not by the algorithm owner's discipline.

Drift monitoring integration: registry entry references the drift dashboard URL; dashboard generates threshold alerts at PSI 0.10 (yellow) and 0.25 (red); alerts route to algorithm owner and accountable executive; threshold breach triggers escalation per §4 incident response runbook. Without registry-to-dashboard integration, drift detection depends on algorithm owner's monitoring discipline; with integration, the system enforces the loop.

Fairness testing integration: registry entry references the fairness test schedule; quarterly tests for Tier 1 are calendared from the entry; test execution by the testing team flows back to the entry with method, result, and report URL; failures (disparate impact ratio outside 0.80-1.25 band or NY DFS proxy-test material gap) trigger §4 incident response. Colorado Reg 10-1-1 quantitative methods, Bayesian Improved Surname Geocoding application notes, and NY DFS proxy-test methodology references are documented in the entry's testing methodology section so the carrier can produce method documentation alongside results.

Version Control History - The Audit Trail That Survives Discovery

Version-control history in the registry serves two purposes: operational (which model version produced a specific decision) and evidentiary (which version was in effect when a consumer-impact event happened). Operational use cases include claim disputes ("what version of Tractable produced the estimate on the policyholder's claim of October 12, 2025"), rate-filing inquiries ("what version of Akur8 produced the rating relativity referenced in the Q3 2025 SERFF filing"), and incident response ("which Shift fraud-detection version produced the Hispanic-surname false-positive cluster in October 2025"). Each requires registry-to-decision-log linkage with version stamping at decision time.

Evidentiary use cases include discovery in bad-faith litigation, DOI exam information requests, and AISET Exhibit C high-risk system detail. The registry's version-control history serves all three when the carrier maintains the change log with effective dates, change type, and AI committee approval reference. Change types to log: training data refresh, hyperparameter change, feature addition or removal, threshold adjustment, vendor model update accepted, vendor model update declined with rationale, retirement and replacement. Material changes - typically anything affecting consumer-facing decisions - trigger AI committee approval before deployment; committee approval references in registry version log.

Colorado Reg 10-1-1 Compliance Report Extracts From the Registry

Colorado Reg 10-1-1 compliance report due July 1, 2026 requires algorithm inventory documentation for life insurance (in scope since 2021), private passenger auto (in scope from October 15, 2025), and health benefit plans (in scope from October 15, 2025). The compliance report extracts registry entries for each Colorado-applicable line: entry-by-entry summary with model name, accountable executive, risk tier, last bias-test date and result, drift-monitoring status, External Consumer Data and Information Source (ECDIS) inventory for L&H, and consumer-facing disclosure status.

The carrier that built the registry to §4 standards extracts the Reg 10-1-1 report through query and formatting; the carrier that built the registry as a spreadsheet writes the report from scratch and faces gaps because the spreadsheet doesn't enforce the field set the report requires. ECDIS inventory for life insurance is a specific Reg 10-1-1 obligation - the carrier must inventory which external consumer data sources feed which models, with bias-testing artifacts for each. The registry's training data lineage field carries this; the spreadsheet typically doesn't.

Key Takeaways

  • The registry is a system, not a spreadsheet. System enforces attestation cadence, version-control history, drift-threshold alerts, fairness-test calendaring, vendor scorecard integration. Spreadsheet survives the first internal review and fails the first external scrutiny.
  • Required field set covers identification, accountability, risk classification, vendor lifecycle, testing and validation, documentation, version control, incident history. Missing fields create regulatory-response gaps that surface during AISET inquiry or Colorado Reg 10-1-1 report.
  • Akur8 personal auto pricing entry (AK-PL-AUTO-001) is Tier 1: quarterly disparate-impact testing, monthly drift, ASOP No. 56 peer review, dual attestation by Chief Actuary and Director Personal Lines Pricing. 14-state deployment with Colorado in scope post-October 2025.
  • Shift fraud detection entry (SH-AUTO-FRD-001) is Tier 1 with documented incident remediation. October 2025 Hispanic-surname false-positive cluster triggered §4 incident response, model rollback, v4.0 redeployment, Colorado DOI notification. v4.1 current with quarterly fairness re-test at acceptable thresholds.
  • Tractable photo estimate entry (TR-AUTO-PD-001) is Tier 1 with vendor concentration risk documented. 31% concentration above 25% limit; committee approval with CCC parallel-vendor mitigation plan for 2026 H2.
  • Risk-tier discipline drives testing cadence by registry enforcement. Tier 1 quarterly fairness + monthly drift + ASOP 56 peer review + dual attestation. Registry generates testing calendar; results flow back; loop enforced by system not algorithm owner discipline.
  • Version control history serves operational (which version produced a specific decision) and evidentiary (discovery, exam, AISET) use cases. Material changes trigger AI committee approval; change log carries change type, effective date, committee reference.
  • Colorado Reg 10-1-1 compliance report extracts from the registry. Life, private passenger auto, health benefit plans all in scope by October 15, 2025; first report due July 1, 2026. Carriers with §4-grade registry extract through query; spreadsheet carriers write from scratch and face gaps.
  • One registry, multiple regulatory artifacts. NAIC AISET Exhibit A (counts and quantification) is a registry summary; Exhibit C (high-risk system detail) is registry depth; NY DFS Circular Letter 2024-7 proxy-test memos reference entries; market-conduct exam information requests cite registry by entry. Build it once; use it everywhere.