1. Requirements Clarification
Functional Requirements
- Accept payments from users (credit/debit card, digital wallets)
- Process payouts to sellers/merchants
- Support refunds (full and partial)
- Maintain a ledger of all financial movements
- Provide payment history to users
- Handle currency conversion for international payments
Non-Functional Requirements
- Correctness: Zero tolerance for double charges or lost money — this is the top priority above performance
- Scale: 1 million transactions per day; peak 100 transactions per second
- Availability: 99.99% (52 min downtime/year)
- Compliance: PCI DSS for card data; GDPR for personal data
- Auditability: Every state change in a transaction must be logged immutably
2. High-Level Architecture
User / Mobile App
│
▼
┌─────────────────┐
│ Payment API │ ← validates, deduplicates (idempotency key check)
│ Service │
└────────┬────────┘
│
┌────▼─────────────────────────────────────────┐
│ MySQL (ACID transactions) │
│ accounts │ transactions │ ledger_entries │
└────┬──────────────────────────────────────────┘
│
┌────▼────────┐ ┌─────────────────────┐
│ PSP Layer │──────▶│ Stripe / PayPal │
│ (adapter) │ │ (external provider) │
└────┬────────┘ └──────────┬────────────┘
│ │ webhook
┌────▼────────┐ ┌──────────▼────────────┐
│ Webhook │◀──────│ Webhook Endpoint │
│ Processor │ │ /api/webhooks/stripe │
└────┬────────┘ └───────────────────────┘
│
┌────▼──────────┐
│ Downstream │ (order fulfillment, email receipt, fraud scoring)
│ Services (MQ) │
└───────────────┘
3. Idempotency — Preventing Double Charges
The most important correctness mechanism in a payment system. Every payment API call must be idempotent.
Interview Tip — Idempotency Key TTL
Store idempotency keys in a separate table with a 24-hour retention policy (or 7 days for payment reconciliation). After expiry, the same key can be reused — but that should be rare since UUIDs are unique by design. The client should generate a fresh UUID for each new payment intent, reusing the same UUID only for retries of the same payment.
4. ACID Transactions for Money Movement
Moving money requires ACID properties — anything less risks creating or destroying money.
5. Double-Entry Bookkeeping
Double-entry bookkeeping is the accounting standard that underpins every financial system. The invariant is simple: every debit must have a corresponding equal credit.
| Transaction | Debit Account | Debit Amount | Credit Account | Credit Amount |
|---|---|---|---|---|
| User tops up wallet | Payment Gateway (external) | $100 | User Wallet | $100 |
| User buys product | User Wallet | $50 | Merchant Wallet | $50 |
| Platform takes fee | Merchant Wallet | $2.50 | Platform Revenue | $2.50 |
| Merchant payout | Merchant Wallet | $47.50 | Bank Account (external) | $47.50 |
| Refund | Merchant Wallet | $50 | User Wallet | $50 |
To verify ledger integrity at any time: SUM(debit entries) = SUM(credit entries). A discrepancy means money was either created or destroyed — a critical bug.
6. PSP Integration and Async Webhooks
A Payment Service Provider (PSP) like Stripe handles the complex parts: card network communication, 3D Secure authentication, bank authorization, and settlement. Your system integrates with the PSP via API and webhooks.
Payment Flow with Stripe:
1. Client: create PaymentIntent (your backend → Stripe API)
Returns: client_secret for frontend
2. Client: collect card details → Stripe.js tokenizes in browser
→ Card data never touches your servers (PCI scope reduction)
3. Client: confirmPayment(client_secret) → Stripe processes
→ Stripe communicates with card network (Visa/MC)
4. Stripe: async result via webhook to your server
POST /webhook/stripe
{ "type": "payment_intent.succeeded", "data": { "id": "pi_xxx" } }
5. Your server: update transaction status → release order → send receipt
Timeline: steps 1–3 take 0.5–3 seconds. Step 4 may take 1–60 seconds.
Never block the user's request waiting for step 4 — return "payment processing"
and update when the webhook arrives.
7. Database Schema
8. Reconciliation
Reconciliation catches cases where your internal records diverge from what the PSP reports. This happens more often than expected — network timeouts, race conditions, and webhook failures all create discrepancies.
- Run a daily reconciliation job at 2am UTC (after PSP settlement window closes)
- Download PSP settlement report for the previous day
- Match each PSP transaction ID to your internal
transactions.psp_ref - Flag: PSP shows success but your DB shows PENDING → likely missed webhook → mark complete
- Flag: your DB shows success but PSP shows failure → refund the user immediately
- Flag: amount mismatch → investigate potential fee calculation bug
Never Trust Client-Side Payment Confirmation
The frontend will call your API saying "payment succeeded" based on the Stripe.js response. Always verify payment status server-side via the PSP API or webhook before releasing goods/services. Fraudsters regularly modify client-side responses to claim payment success without actually paying.
How We Research and Update This Guide
We test the underlying formula or workflow, compare outputs with reliable references, and revise examples whenever the page content changes.
- The workflow or formula is tested directly in the tool and compared against independent reference examples.
- Examples are kept practical so readers can verify the result without hidden assumptions.
- Pages are revised whenever the interface, calculation flow, or surrounding guidance materially changes.
Frequently Asked Questions — Payment System Design
An idempotency key is a unique client-generated ID (UUID) sent with every payment request. If the request fails or times out and the client retries, it sends the same idempotency key. The server looks up the key — if it already exists, it returns the result of the first request without processing a second payment. Without idempotency keys, a network timeout could result in a charge being applied twice: the server processed it, but the client never got the response and retried. Stripe, PayPal, and every serious payment API require idempotency keys for all write operations.
Double-entry bookkeeping records every financial transaction as two entries: a debit in one account and an equal credit in another. For a $100 payment from Alice to Bob: DEBIT Alice's account $100 (her balance decreases) and CREDIT Bob's account $100 (his balance increases). The fundamental invariant: sum of all debits = sum of all credits. This means ledger entries can never disappear — money is never created or destroyed, only moved. Every balance discrepancy is detectable by checking if debits equal credits.
Payment processing is asynchronous — the PSP (Stripe/PayPal) may take seconds or minutes to confirm a charge. Rather than polling the PSP, subscribe to webhooks: the PSP sends an HTTP POST to your endpoint when the payment status changes (payment_intent.succeeded, payment_intent.payment_failed). Your webhook handler: (1) validates the signature (HMAC-SHA256 with webhook secret); (2) checks for duplicate delivery (store processed event_ids in Redis for 24h); (3) updates the transaction status in your DB; (4) triggers downstream actions (send receipt, release order).
Reconciliation compares your internal ledger against the PSP's transaction records to find discrepancies. Run daily: (1) Download the PSP's settlement report (CSV or API) for the previous day; (2) Match each PSP transaction to an internal transaction by PSP reference ID; (3) Flag unmatched transactions — money moved at the PSP that you don't have in your ledger (or vice versa); (4) Flag amount mismatches; (5) Investigate and correct discrepancies within SLA. Automate steps 1–4 with a reconciliation job; human review for step 5.
PCI DSS (Payment Card Industry Data Security Standard) has 12 requirements. The key system design impacts: never store raw card numbers (PAN), CVV, or magnetic stripe data in your systems — this is the #1 rule. Use a PCI-compliant PSP (Stripe, Braintree) which tokenizes the card on their servers; you only store a token. If you must store card data, it must be encrypted with AES-256, with encryption keys in a separate Hardware Security Module (HSM). Network: isolate payment services in a separate VPC with strict inbound/outbound rules. Logging: mask card numbers in logs. Auditing: all admin access to payment systems must be logged.
Fraud detection is a multi-layered system. Rule-based checks (fast, deterministic): block transactions over $10,000 from new accounts, flag if shipping address doesn't match billing country, block velocity — more than 3 failed card attempts in 1 hour. ML-based scoring (async): score each transaction using a fraud model trained on historical fraud patterns; high-score transactions trigger 3DS authentication or manual review. Third-party services: Stripe Radar, Kount, and Sift provide pre-built fraud scoring. The key architectural point: rule-based checks happen synchronously in the payment flow; ML scoring can happen asynchronously after capture.