All Case Studies
TechnologyLed by Endrit Basha · Fortune 500 insurer · 2024–present

Moving 30M+ Messages a Day Without Losing a Record

Insurance-grade reliability for data that has to survive an audit.

Sound familiar? A member's dues history disappears — and you find out from them.

30M+
messages processed daily
33
enterprise data sources
40%
projected storage costs cut
$150K/yr
vendor cost eliminated
The Challenge

A Fortune 500 insurer's data pipelines feed everything downstream — analytics, reporting, regulatory filings. The data arrives from dozens of enterprise systems at wildly variable volume, and the failure mode is silent: a message drops, a source stalls, and nobody notices until the numbers downstream are wrong. Layer on New York's financial regulator, NYDFS, which requires provable policy-based data retention — keeping data too long is a compliance failure, and deleting the wrong data is worse. The engineering standard is simple to state and hard to meet: ingest everything, lose nothing, prove both.

The Approach
  1. High-throughput distributed ingestion microservice architected and scaled to stream from enterprise message queues into CosmosDB and Snowflake
  2. 33 enterprise data sources onboarded, reliably processing 30M+ messages per day
  3. Adaptive batching, concurrency controls, and backpressure handling engineered to absorb variable load at low latency
  4. Data observability platform and health engine built with lineage-aware checks — real-time visibility into failures before they distort downstream analytics
  5. Proactive failure detection and impact analysis that reduced incident resolution time across critical pipelines
  6. Fault-tolerant data retention platform (Go, Docker, workflow orchestration) automating large-scale policy-based deletion to NYDFS compliance standards
  7. Legacy translation system replaced with a custom-built Go service, eliminating a $150K annual vendor dependency
The Outcome

The ingestion platform onboarded 33 enterprise data sources and reliably processes 30M+ messages per day into CosmosDB and Snowflake. The retention platform automated policy-based deletion to NYDFS standards, reducing audit risk and cutting projected storage costs by 40%. Replacing a legacy translation system with a custom Go service eliminated a $150K-per-year vendor contract. Endrit was selected as a peer mentor for new engineers.

AWS Certified Cloud PractitionerTestOut CyberDefense ProPeer Mentor for New Engineers
Why this matters for your association

Your member database, dues history, and financial records deserve the same reliability and security discipline an insurer applies to policy data. When associations migrate between AMS platforms, records get lost and dues histories get corrupted — and nobody notices until a member complains. The observability and reliability practices described here are exactly what prevents that. And when a vendor is charging you for something a purpose-built tool does better, we know when building beats renting — because Endrit has done it at enterprise scale in his role at a Fortune 500 insurer.

Talk to us about protecting member records

Want this for your organization?

We're accepting Founding Partners now. Complimentary services during our launch period — so you can see the work before you commit a dollar.