Payment Incident Response Runbook for UPI Merchants

A payment incident is any failure where payment evidence, order state, or fulfilment may be wrong at meaningful scale. Examples include a secret exposure, broad signature rejection after deployment, duplicate credits, delayed event processing, cross-merchant data access, or a checkout that shows success without backend verification.

The first objective is financial correctness. Traffic, conversion, and deployment speed are secondary while the system may release goods or access incorrectly.

Declare and Scope

Create one incident record and name an incident commander, engineering lead, operations lead, and customer-communication owner. Record the start time, detection source, affected environment, known merchant tenants, application version, and first safe event references.

Use a severity model:

Severity	Example	Response
Critical	False paid states, cross-tenant exposure, active secret compromise	Immediate containment and executive/security ownership
High	Duplicate fulfilment or widespread valid-event rejection	Stop affected effects and reconcile urgently
Medium	Elevated pending orders with a working recovery path	Assign owners and increase monitoring
Low	Isolated delivery failure with no incorrect state	Handle through normal operations

Do not wait for perfect certainty before declaring. An incident can be downgraded after evidence improves.

Contain Financial Effects

Choose controls that stop new incorrect decisions:

• Pause the affected fulfilment worker.
• Disable manual paid overrides.
• Keep uncertain orders pending instead of failed or paid.
• Disable one merchant/provider route rather than unrelated checkout paths.
• Rotate exposed API or webhook secrets.
• Roll back the application version that changed verification behavior.
• Preserve the original order reference during customer retry guidance.

Never “fix” broad webhook rejection by accepting unsigned events. If verification is unhealthy, use authenticated backend status checks for known orders and a bounded reconciliation queue.

Preserve Safe Evidence

Capture deployment IDs, configuration versions, event IDs, order references, response-code distribution, signature rejection reason, worker queue depth, and state-transition audit records. Protect logs from editing and restrict access by tenant and role.

Do not copy raw secrets or customer banking details into the incident channel. If a sensitive value must be examined, use the approved secret or evidence system and record access separately.

Build an affected-order query that is reproducible. Define start/end timestamps, environment, merchant scope, application version, and the exact state predicate. Save the query or report version so later totals can be explained.

Recover and Reconcile

Restore authentication and persistence first, then process affected orders in a controlled order:

1. Prove a new test event returns actual 2xx.
2. Confirm event identity is deduplicated.
3. Reconcile known unresolved orders against authenticated evidence.
4. Apply authorized state transitions once.
5. Resume fulfilment with a low concurrency limit.
6. Check for duplicate business effects.
7. Increase throughput while error metrics remain healthy.

Use the daily reconciliation checklist and webhook delivery monitoring guide to separate event delivery, database processing, and fulfilment.

Customer updates should state what is affected, what users should avoid, and when the next update will arrive. Do not claim payments failed when they are still being verified, and do not instruct customers to repay an uncertain order.

Close and Prevent Recurrence

Close only after every affected population has an owner or verified resolution, monitoring is stable, temporary access is removed, and secrets or feature flags are in their intended state. Record customer impact, financial impact, timeline, root cause, contributing controls, and evidence-backed corrective actions.

Corrective actions need an owner and deadline. Prefer controls that make recurrence difficult: database uniqueness, raw-body integration tests, deployment canaries, secret-version metrics, checkout kill switches, and reconciliation alerts. A post-incident document without enforced changes is only a history lesson.

Measure incident response itself: time to detection, time to containment, number of affected orders, number reconciled, incorrect fulfilment count, customer contacts, and time until the last case had an owner. Do not optimize only for the moment the service returned; payment recovery continues until order and evidence populations agree.

Run a tabletop exercise before a real incident. Give the team a simulated signature failure after deployment, a queue backlog, and several paid-but-pending orders. Verify that staff can find the correct dashboard, pause the narrowest affected path, locate safe event references, issue accurate customer wording, and start reconciliation without accessing secrets or editing database rows directly.