Building resilient webhook listeners for high-concurrency UPI and payment rails
When integrating Indian payment gateways (such as Razorpay, Cashfree, or PayU) handling UPI intent flows, gateway servers expect an immediate 200 OK response within a tight timeout (often under 2–3 seconds). Synchronous operations—like complex database writes or notification dispatches—cause timeouts, triggering aggressive retries, race conditions, and duplicated fulfillment.
Follow this battle-tested pattern to decouple ingestion from processing:
Step 1: Validate payload signatures in memory
Verify the cryptographic signature (HMAC-SHA256) using raw request buffers before running parsing middleware. Reject invalid signatures immediately with a 400 status to block unauthorized traffic at the perimeter.
Step 2: Enqueue to a durable broker and return 200 immediately
Push the raw webhook event into a lightweight message queue (e.g., Redis Streams, RabbitMQ, or AWS SQS). Return an immediate 200 OK response with an acknowledgment ID. Your HTTP layer should take less than 40ms end-to-end.
Step 3: Implement an idempotent worker layer
Consume events downstream using worker pools. Ensure idempotency by tracking the gateway's unique transaction/event ID in a dedicated key-value store with an atomic lock (SETNX in Redis):
Check if the event ID is already processed.
If locked/processed, acknowledge and skip.
If new, acquire the lock, update order state in your primary database, release the lock, and acknowledge the message.
Step 4: Configure dead-letter queues (DLQ) and backoff retries
Route unprocessable payloads to a DLQ after three exponential backoff retries. This isolates poisoned payloads without stalling the processing of incoming real-time transactions.
Key Takeaways
Acknowledge first, process second: Never execute downstream business logic or external API calls inside the HTTP webhook handler.
Idempotency is mandatory: Payment aggregators guarantee at-least-once delivery; your worker must natively handle duplicate webhook deliveries safely.
Isolate failures with DLQs: Prevent poisoned payloads from blocking queue throughput during high-traffic surges.
CTA (Join Techawks India)
Scaling systems for high-throughput Indian fintech rails? Join Techawks India to collaborate with backend engineers, access production-ready architectural patterns, and level up your distributed systems design.
When integrating Indian payment gateways (such as Razorpay, Cashfree, or PayU) handling UPI intent flows, gateway servers expect an immediate 200 OK response within a tight timeout (often under 2–3 seconds). Synchronous operations—like complex database writes or notification dispatches—cause timeouts, triggering aggressive retries, race conditions, and duplicated fulfillment.
Follow this battle-tested pattern to decouple ingestion from processing:
Step 1: Validate payload signatures in memory
Verify the cryptographic signature (HMAC-SHA256) using raw request buffers before running parsing middleware. Reject invalid signatures immediately with a 400 status to block unauthorized traffic at the perimeter.
Step 2: Enqueue to a durable broker and return 200 immediately
Push the raw webhook event into a lightweight message queue (e.g., Redis Streams, RabbitMQ, or AWS SQS). Return an immediate 200 OK response with an acknowledgment ID. Your HTTP layer should take less than 40ms end-to-end.
Step 3: Implement an idempotent worker layer
Consume events downstream using worker pools. Ensure idempotency by tracking the gateway's unique transaction/event ID in a dedicated key-value store with an atomic lock (SETNX in Redis):
Check if the event ID is already processed.
If locked/processed, acknowledge and skip.
If new, acquire the lock, update order state in your primary database, release the lock, and acknowledge the message.
Step 4: Configure dead-letter queues (DLQ) and backoff retries
Route unprocessable payloads to a DLQ after three exponential backoff retries. This isolates poisoned payloads without stalling the processing of incoming real-time transactions.
Key Takeaways
Acknowledge first, process second: Never execute downstream business logic or external API calls inside the HTTP webhook handler.
Idempotency is mandatory: Payment aggregators guarantee at-least-once delivery; your worker must natively handle duplicate webhook deliveries safely.
Isolate failures with DLQs: Prevent poisoned payloads from blocking queue throughput during high-traffic surges.
CTA (Join Techawks India)
Scaling systems for high-throughput Indian fintech rails? Join Techawks India to collaborate with backend engineers, access production-ready architectural patterns, and level up your distributed systems design.
Building resilient webhook listeners for high-concurrency UPI and payment rails
When integrating Indian payment gateways (such as Razorpay, Cashfree, or PayU) handling UPI intent flows, gateway servers expect an immediate 200 OK response within a tight timeout (often under 2–3 seconds). Synchronous operations—like complex database writes or notification dispatches—cause timeouts, triggering aggressive retries, race conditions, and duplicated fulfillment.
Follow this battle-tested pattern to decouple ingestion from processing:
Step 1: Validate payload signatures in memory
Verify the cryptographic signature (HMAC-SHA256) using raw request buffers before running parsing middleware. Reject invalid signatures immediately with a 400 status to block unauthorized traffic at the perimeter.
Step 2: Enqueue to a durable broker and return 200 immediately
Push the raw webhook event into a lightweight message queue (e.g., Redis Streams, RabbitMQ, or AWS SQS). Return an immediate 200 OK response with an acknowledgment ID. Your HTTP layer should take less than 40ms end-to-end.
Step 3: Implement an idempotent worker layer
Consume events downstream using worker pools. Ensure idempotency by tracking the gateway's unique transaction/event ID in a dedicated key-value store with an atomic lock (SETNX in Redis):
Check if the event ID is already processed.
If locked/processed, acknowledge and skip.
If new, acquire the lock, update order state in your primary database, release the lock, and acknowledge the message.
Step 4: Configure dead-letter queues (DLQ) and backoff retries
Route unprocessable payloads to a DLQ after three exponential backoff retries. This isolates poisoned payloads without stalling the processing of incoming real-time transactions.
Key Takeaways
Acknowledge first, process second: Never execute downstream business logic or external API calls inside the HTTP webhook handler.
Idempotency is mandatory: Payment aggregators guarantee at-least-once delivery; your worker must natively handle duplicate webhook deliveries safely.
Isolate failures with DLQs: Prevent poisoned payloads from blocking queue throughput during high-traffic surges.
CTA (Join Techawks India)
Scaling systems for high-throughput Indian fintech rails? Join Techawks India to collaborate with backend engineers, access production-ready architectural patterns, and level up your distributed systems design.
English
Arabic
French
Spanish
Portuguese
Deutsch
Turkish
Dutch
Italiano
Romaian
Portuguese (Brazil)
Greek