The 3 a.m. Test<\/strong><\/p>\n The test is simple: When this system breaks at 3 a.m., will the on-call engineer<\/a> be able to diagnose and fix it quickly?<\/p>\n This single question has eliminated a surprising number of \u201cclever\u201d design choices from my architectures:<\/p>\n After that incident, I rebuilt our API infrastructure from the ground up. Over the next three years, handling 50 million daily requests, I developed five principles that transformed our reliability from 99.2% to 99.95% and let me sleep through the night.<\/p>\n Principle 1: Design for Partial Failure<\/strong><\/p>\n Six months after the initial incident, we had another outage. This time, a downstream payment processor went unresponsive. Our API dutifully waited for responses that never came, and request threads piled up until we crashed.<\/p>\n I realized we\u2019d solved one problem but created another. We needed systems that degraded gracefully instead of failing catastrophically.<\/p>\n Here\u2019s what we built:<\/p>\n \nclass ResilientServiceClient:
\t\t\t\t
<\/p>\n 1<\/p>\n 2<\/p>\n 3<\/p>\n 4<\/p>\n 5<\/p>\n 6<\/p>\n 7<\/p>\n 8<\/p>\n 9<\/p>\n 10<\/p>\n 11<\/p>\n 12<\/p>\n 13<\/p>\n 14<\/p>\n 15<\/p>\n 16<\/p>\n 17<\/p>\n 18<\/p>\n 19<\/p>\n 20<\/p>\n 21<\/p>\n 22<\/p>\n 23<\/p>\n 24<\/p>\n 25<\/p>\n 26<\/p>\n 27<\/p>\n 28<\/p>\n 29<\/p>\n
\n\t\t\t\t<\/td>\n
<\/p>\n class ResilientServiceClient:<\/p>\n \u00a0\u00a0\u00a0\u00a0def __init__(self, primary_url, fallback_url):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.primary = primary_url<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.fallback = fallback_url<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.circuit_breaker = CircuitBreaker(<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0failure_threshold=5,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0recovery_timeout=30<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/p>\n \u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0async def fetch(self, request):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Try primary with circuit breaker protection<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if self.circuit_breaker.is_closed():<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0response = await self.call_with_timeout(<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.primary, request, timeout_ms=500<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.circuit_breaker.record_success()<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return response<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0except (TimeoutError, ConnectionError):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.circuit_breaker.record_failure()<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Fall back to secondary<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return await self.call_with_timeout(<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.fallback, request, timeout_ms=1000<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0except Exception:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Return degraded response rather than error<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return self.degraded_response(request)<\/p>\n<\/td>\n
<\/p>\n The key insight: A degraded response is almost always better than an error. Users can work with stale data or reduced functionality. They can\u2019t work with a 500 error.<\/p>\n After implementing this pattern<\/a> across our services, we stopped having cascading failures. When the payment processor went down again (it did, three more times that year), our API returned cached pricing and queued transactions for later processing. Customers barely noticed.<\/p>\n Principle 2: Make Idempotency Non-Negotiable<\/strong><\/p>\n This lesson came from a $27,000 mistake.<\/p>\n A mobile client had a bug that caused it to retry failed requests aggressively. One of those requests was a payment. The retry logic didn\u2019t include idempotency keys. You can guess what happened next.<\/p>\n A single customer got charged 23 times for the same order. By the time we noticed, we\u2019d processed duplicate charges across hundreds of accounts. The refunds, the customer service hours, the engineering time to fix<\/a> it cost $27,000.<\/p>\n Now, every mutating endpoint requires an idempotency key. No exceptions.<\/p>\n \nclass IdempotentEndpoint:
\t\t\t\t
<\/p>\n 1<\/p>\n 2<\/p>\n 3<\/p>\n 4<\/p>\n 5<\/p>\n 6<\/p>\n 7<\/p>\n 8<\/p>\n 9<\/p>\n 10<\/p>\n 11<\/p>\n 12<\/p>\n 13<\/p>\n 14<\/p>\n 15<\/p>\n 16<\/p>\n 17<\/p>\n 18<\/p>\n 19<\/p>\n 20<\/p>\n 21<\/p>\n 22<\/p>\n 23<\/p>\n 24<\/p>\n 25<\/p>\n
\n\t\t\t\t<\/td>\n
<\/p>\n class IdempotentEndpoint:<\/p>\n \u00a0\u00a0\u00a0\u00a0def __init__(self):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.idempotency_store = RedisStore(ttl_hours=24)<\/p>\n \u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0async def handle_request(self, request, idempotency_key):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Check if we’ve already processed this request<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0existing = await self.idempotency_store.get(idempotency_key)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if existing:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Return cached response \u2014 don’t re-execute<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return Response(<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0data=existing[‘response’],<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0headers={‘X-Idempotent-Replay’: ‘true’}<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Process the request<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0result = await self.execute_operation(request)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Cache for future retries<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0await self.idempotency_store.set(<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0idempotency_key,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{‘response’: result, ‘timestamp’: now()}<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return Response(data=result)<\/p>\n<\/td>\n
<\/p>\n We also started rejecting requests without idempotency keys for any POST, PUT or DELETE operation. Some client developers complained initially. Then they thanked us when their retry bugs didn\u2019t cause data corruption.<\/p>\n Principle 3: Version in the URL, Not the Header<\/strong><\/p>\n I learned this one by watching a junior engineer debug an issue for six hours.<\/p>\n We\u2019d been versioning our API through a custom header: X-API-Version: 2. It seemed clean. Kept the URLs tidy.<\/p>\n But when something went wrong, our logs showed the URL and response code \u2014 not the headers. The engineer was looking at logs for \/users\/123 and couldn\u2019t figure out why the behavior was different between two clients. Six hours later, he finally thought to check the version header.<\/p>\n We moved versioning to the URL path that week:<\/p>\n \n\/v1\/users\/123
\t\t\t\t
<\/p>\n
<\/p>\n \/v1\/users\/123<\/p>\n \/v2\/users\/123<\/p>\n<\/td>\n
<\/p>\n Now version information shows up in:<\/p>\n The debugging time savings alone justified the migration. But we also established versioning rules that prevented future pain:<\/p>\n When we do deprecate a version, clients get a Deprecation header warning them for months before we actually turn it off.<\/p>\n Principle 4: Rate Limit Before You Need To<\/strong><\/p>\n We almost learned this lesson the hard way.<\/p>\n A partner company integrated with our API. Their team\u2019s implementation had a bug: When they got a timeout, they\u2019d retry immediately. Infinitely. With exponential parallelism.<\/p>\n At 2 p.m. on a Thursday, their system started sending 50,000 requests per second. We didn\u2019t have rate limiting. We\u2019d always planned to add it \u201cwhen we needed it.\u201d<\/p>\n We needed it.<\/p>\n Fortunately, our load balancer had basic protection that kicked in and started dropping requests. But legitimate traffic got dropped too. For 47 minutes, our API was essentially a lottery \u2014 maybe your request would get through, maybe it wouldn\u2019t.<\/p>\n The next week, we implemented tiered rate limiting:<\/p>\n \nclass TieredRateLimiter:
\t\t\t\t
<\/p>\n 1<\/p>\n 2<\/p>\n 3<\/p>\n 4<\/p>\n 5<\/p>\n 6<\/p>\n 7<\/p>\n 8<\/p>\n 9<\/p>\n 10<\/p>\n 11<\/p>\n 12<\/p>\n 13<\/p>\n 14<\/p>\n 15<\/p>\n 16<\/p>\n 17<\/p>\n 18<\/p>\n 19<\/p>\n 20<\/p>\n 21<\/p>\n 22<\/p>\n
\n\t\t\t\t<\/td>\n
<\/p>\n class TieredRateLimiter:<\/p>\n \u00a0\u00a0\u00a0\u00a0def __init__(self):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.limiters = {<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘per_client’: TokenBucket(rate=100, burst=200),<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘per_endpoint’: TokenBucket(rate=1000, burst=2000),<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘global’: TokenBucket(rate=10000, burst=15000)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}<\/p>\n \u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0async def check_limit(self, client_id, endpoint):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Check all tiers, return first failure<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0for tier_name, limiter in self.limiters.items():<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0key = client_id if tier_name == ‘per_client’ else endpoint<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0result = await limiter.check(key)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if not result.allowed:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return RateLimitResponse(<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0allowed=False,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0retry_after=result.retry_after,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0limit_type=tier_name<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return RateLimitResponse(allowed=True)<\/p>\n<\/td>\n
<\/p>\n The key details that made this actually useful:<\/p>\n That partner\u2019s bug happened again six months later. This time, their requests got rate-limited, our other clients were unaffected, and I didn\u2019t even find out until I checked the metrics the next morning.<\/p>\n Principle 5: If You Can\u2019t See It, You Can\u2019t Fix It<\/strong><\/p>\n The scariest outages aren\u2019t the ones where everything breaks. They\u2019re the ones where something is subtly wrong and you don\u2019t notice for days.<\/p>\n We had an issue where 3% of requests were failing with a specific error code. Not enough to trigger our availability alerts (we\u2019d set those at 5%). Not enough for customers to flood support. But enough that hundreds of users per day were having a bad experience.<\/p>\n It took us two weeks to notice. Two weeks of a broken experience for real users.<\/p>\n After that, we built observability into every endpoint:<\/a><\/p>\n \nclass ObservableEndpoint:
\t\t\t\t
<\/p>\n 1<\/p>\n 2<\/p>\n 3<\/p>\n 4<\/p>\n 5<\/p>\n 6<\/p>\n 7<\/p>\n 8<\/p>\n 9<\/p>\n 10<\/p>\n 11<\/p>\n 12<\/p>\n 13<\/p>\n 14<\/p>\n 15<\/p>\n 16<\/p>\n 17<\/p>\n 18<\/p>\n 19<\/p>\n 20<\/p>\n 21<\/p>\n 22<\/p>\n 23<\/p>\n 24<\/p>\n 25<\/p>\n 26<\/p>\n 27<\/p>\n 28<\/p>\n 29<\/p>\n 30<\/p>\n 31<\/p>\n 32<\/p>\n
\n\t\t\t\t<\/td>\n
<\/p>\n class ObservableEndpoint:<\/p>\n \u00a0\u00a0\u00a0\u00a0async def handle(self, request):<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0trace_id = self.tracer.start_trace()<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0start_time = time.time()<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0response = await self.process(request)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Record success metrics<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0duration_ms = (time.time() – start_time) * 1000<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.metrics.histogram(‘request_duration_ms’, duration_ms, {<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘endpoint’: request.path,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘status’: response.status<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0})<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.metrics.increment(‘requests_total’, {<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘endpoint’: request.path,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘status’: response.status<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0})<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return response<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0except Exception as e:<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Record failure with context<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.metrics.increment(‘requests_errors’, {<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘endpoint’: request.path,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘error_type’: type(e).__name__<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0})<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.logger.error(‘request_failed’, {<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘trace_id’: trace_id,<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0‘error’: str(e)<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0})<\/p>\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0raise<\/p>\n<\/td>\n
<\/p>\n Our minimum observability requirements now:<\/p>\n The 3% failure issue? With our new observability, we would have caught it in minutes, not weeks.<\/p>\n The Results<\/strong><\/p>\n After three years of applying these principles across our API infrastructure:<\/p>\n That last metric is the one I care about most. I went from being woken up twice a week to once every two months.<\/p>\n What I\u2019d Tell My Past Self<\/strong><\/p>\n If I could go back to before that first 3 a.m. call, I\u2019d tell myself:<\/p>\n APIs don\u2019t survive by accident. They survive by design, specifically, by designing for the moment when everything goes wrong.<\/p>\n Now, when my phone buzzes at 3 a.m., it\u2019s usually just spam. And that\u2019s exactly how I like it.<\/p>\n\n
\n def __init__(self, primary_url, fallback_url):
\n self.primary = primary_url
\n self.fallback = fallback_url
\n self.circuit_breaker = CircuitBreaker(
\n failure_threshold=5,
\n recovery_timeout=30
\n )
\n
\n async def fetch(self, request):
\n # Try primary with circuit breaker protection
\n if self.circuit_breaker.is_closed():
\n try:
\n response = await self.call_with_timeout(
\n self.primary, request, timeout_ms=500
\n )
\n self.circuit_breaker.record_success()
\n return response
\n except (TimeoutError, ConnectionError):
\n self.circuit_breaker.record_failure()
\n
\n # Fall back to secondary
\n try:
\n return await self.call_with_timeout(
\n self.fallback, request, timeout_ms=1000
\n )
\n except Exception:
\n # Return degraded response rather than error
\n return self.degraded_response(request)<\/p>\n
<\/p>\n
<\/p>\n \n
\n\t\t\t\t\t<\/tr>\n
\n def __init__(self):
\n self.idempotency_store = RedisStore(ttl_hours=24)
\n
\n async def handle_request(self, request, idempotency_key):
\n # Check if we’ve already processed this request
\n existing = await self.idempotency_store.get(idempotency_key)
\n
\n if existing:
\n # Return cached response \u2014 don’t re-execute
\n return Response(
\n data=existing[‘response’],
\n headers={‘X-Idempotent-Replay’: ‘true’}
\n )
\n
\n # Process the request
\n result = await self.execute_operation(request)
\n
\n # Cache for future retries
\n await self.idempotency_store.set(
\n idempotency_key,
\n {‘response’: result, ‘timestamp’: now()}
\n )
\n
\n return Response(data=result)<\/p>\n
<\/p>\n
<\/p>\n \n
\n\t\t\t\t\t<\/tr>\n
\n\/v2\/users\/123<\/p>\n
<\/p>\n
\n\t\t\t\t\t
\n\t\t\t\t<\/td>\n\n
\n\t\t\t\t\t<\/tr>\n\n
\n
\n def __init__(self):
\n self.limiters = {
\n ‘per_client’: TokenBucket(rate=100, burst=200),
\n ‘per_endpoint’: TokenBucket(rate=1000, burst=2000),
\n ‘global’: TokenBucket(rate=10000, burst=15000)
\n }
\n
\n async def check_limit(self, client_id, endpoint):
\n # Check all tiers, return first failure
\n for tier_name, limiter in self.limiters.items():
\n key = client_id if tier_name == ‘per_client’ else endpoint
\n result = await limiter.check(key)
\n
\n if not result.allowed:
\n return RateLimitResponse(
\n allowed=False,
\n retry_after=result.retry_after,
\n limit_type=tier_name
\n )
\n
\n return RateLimitResponse(allowed=True)<\/p>\n
<\/p>\n
<\/p>\n \n
\n\t\t\t\t\t<\/tr>\n\n
\n async def handle(self, request):
\n trace_id = self.tracer.start_trace()
\n start_time = time.time()
\n
\n try:
\n response = await self.process(request)
\n
\n # Record success metrics
\n duration_ms = (time.time() – start_time) * 1000
\n self.metrics.histogram(‘request_duration_ms’, duration_ms, {
\n ‘endpoint’: request.path,
\n ‘status’: response.status
\n })
\n self.metrics.increment(‘requests_total’, {
\n ‘endpoint’: request.path,
\n ‘status’: response.status
\n })
\n
\n return response
\n
\n except Exception as e:
\n # Record failure with context
\n self.metrics.increment(‘requests_errors’, {
\n ‘endpoint’: request.path,
\n ‘error_type’: type(e).__name__
\n })
\n self.logger.error(‘request_failed’, {
\n ‘trace_id’: trace_id,
\n ‘error’: str(e)
\n })
\n raise<\/p>\n
<\/p>\n
<\/p>\n \n
\n\t\t\t\t\t<\/tr>\n\n
\n
![](\"https:\/\/www.europesays.com\/ie\/wp-content\/uploads\/2025\/12\/1507ea85-cropped-e67f5a04-sreenivasa-reddy-hulebeedu-reddy.jpg\"\/)
<\/p>\n