From PoC to Production: Scalable Software Architecture, System Reliability, and Cost Engineering

An order demo can look convincing within hours: add a product to a cart, call a payment provider, and display “Order received.” Production asks a harder question: which facts will remain true when the customer clicks twice, the bank takes the money but its response is lost, or a worker stops during deployment?

This article follows one B2B ordering and payment system from PoC to live operation. PostgreSQL owns orders, inventory reservations, and payment state. Order acceptance is synchronous; charging, notifications, and optional AI assistance are asynchronous. Every architectural decision has consequences for correctness, latency, recovery, and cost.

The monthly average is only about 0.39 requests/s, so a 200 requests/s peak deliberately assumes concentrated campaigns or imports. Capacity follows peak load and latency budgets; unit cost follows actual monthly volume. Confusing the two produces either excessive idle capacity or collapsing queues.

1. Architectural Differences Between Proof of Concept, MVP, and Production-Ready Systems

A PoC answers “Can this integration work?” A payment sandbox, one application process, and sample data may be sufficient. An MVP tests “Does this workflow create value for real users?” It has fewer features, but handling real money still requires authentication, data integrity, and basic reconciliation. Production readiness means keeping an explicit promise under defined load and failure conditions at an operable cost.

The first production step is a modular monolith with explicit interfaces for Orders, Inventory, Payments, and Notifications. API and worker processes can run from the same codebase. PostgreSQL outbox records provide a durable boundary between them. An in-memory queue is unsuitable for payments because a process restart can lose accepted work.

Two API instances on the same VPS help with process failure; they do not provide high availability against host or disk loss. Our production target uses API nodes in separate failure domains, independent worker capacity, and PostgreSQL with tested failover and point-in-time recovery. A small business may start on one VPS, but must explicitly accept its recovery, downtime, and data-loss limits.

2. Distributed Systems: Concurrency, Race Conditions, and Data Consistency

When one item remains, two requests can both SELECT “1” and later UPDATE to “0,” accepting two orders. An application mutex does not coordinate independent API processes. Correctness belongs in the transaction that changes the data and in database constraints. Represent money as bigint minor units with a separate currency field, and derive prices and discounts on the server.

Atomic inventory reservation and outbox

The fragment below is intended for an application SQL driver with named bind parameters. The API first claims the idempotency record inside the same transaction. If the inventory UPDATE returns no row, the application rolls back and does not execute the remaining INSERTs. On success, the order, reservation, outbox event, and replayable response become durable in one commit.

BEGIN;
-- Prerequisites:
-- inventory PRIMARY KEY (tenant_id, sku), CHECK (available >= 0)
-- idempotency PRIMARY KEY (tenant_id, operation, request_key)
-- orders UNIQUE (tenant_id, request_key)
-- reservations UNIQUE (tenant_id, order_id, sku)
-- outbox PRIMARY KEY (event_id)
-- Tenant-scoped foreign keys connect reservations to orders/inventory.
-- The idempotency claim and payload-hash check happen here (see flow below).

UPDATE inventory
SET available = available - :qty
WHERE tenant_id = :tenant_id AND sku = :sku
  AND :qty > 0 AND available >= :qty
RETURNING available;
-- Application MUST verify exactly one row; otherwise ROLLBACK.

INSERT INTO orders
  (id, tenant_id, request_key, amount_minor, currency, status)
VALUES
  (:order_id, :tenant_id, :request_key, :server_total, :currency,
   'PAYMENT_PENDING');

INSERT INTO reservations
  (tenant_id, order_id, sku, qty, state)
VALUES (:tenant_id, :order_id, :sku, :qty, 'HELD');

INSERT INTO outbox (event_id, tenant_id, event_type, aggregate_id, payload)
VALUES (:event_id, :tenant_id, 'PaymentRequested', :order_id, :payload_json);

UPDATE idempotency
SET http_status = 202, response_body = :accepted_response_json
WHERE tenant_id = :tenant_id AND operation = 'POST /orders'
  AND request_key = :request_key;
COMMIT;
-- Call the payment provider only AFTER commit, from a worker.

At Read Committed, this conditional UPDATE serializes competing writers on the inventory row and re-evaluates its condition after waiting. It does not make every business rule serializable. For multi-SKU carts, lock rows in a stable SKU order and roll back the entire reservation if any item is unavailable. Cross-row invariants may require explicit locking or Serializable isolation; retry the entire transaction within a bounded budget after serialization failures or deadlocks. See PostgreSQL transaction isolation.

Idempotency: binding repeated requests to one business outcome

Storing a key only in Redis is insufficient: TTL expiry, eviction, or failover can reopen the path to another payment. PostgreSQL enforces durable uniqueness. Scope the key by tenant + operation + request_key and hash the canonical product, quantity, currency, and quote payload. The client keeps its key for one logical attempt and generates a new key for a different order.

authenticate_and_authorize_tenant()
validate_payload_and_server_side_quote()
with db.transaction():
    claimed = INSERT_idempotency_ON_CONFLICT_DO_NOTHING_RETURNING_key(
        tenant, "POST /orders", key, canonical_payload_hash
    )
    if not claimed:
        saved = SELECT_idempotency_FOR_UPDATE(tenant, operation, key)
        if saved.payload_hash != canonical_payload_hash:
            return HTTP_409_KEY_REUSED_WITH_DIFFERENT_PAYLOAD
        return saved.http_status, saved.response_body
    # Execute stock/order/reservation/outbox SQL above.
    # Persist the 202 response before COMMIT.
return HTTP_202(order_id, status_url)

# A concurrent INSERT waits for the conflicting transaction.
# If its owner rolls back, another caller can claim the key.
# If it commits, replay reads the committed response in a new statement.
# On lock timeout: roll back, return retryable error; keep the SAME key.

Order keys and provider keys have different lifetimes. A worker persists its payment_attempt before making the network call and sends a stable key derived from tenant/order/attempt identity. A timeout does not create a new attempt. Retention must account for the provider’s idempotency window; once that window expires, query and reconcile instead of blindly retrying. Stripe’s idempotency contract illustrates why behavior must be checked per provider and API version.

3. Fault Tolerance and Resilience Engineering: What Happens When Systems Fail?

A local PostgreSQL transaction cannot atomically commit a bank charge. Therefore, “HTTP timed out, payment failed” is an invalid inference. Model three outcomes: confirmed success, confirmed rejection, and an unknown outcome. Cancelling every order on an error can release inventory for a customer who has already paid.

Concrete failure: the provider completes a TRY 1,250 charge for order O-104, but the worker dies before recording the response. After the lease expires, another worker takes over the same attempt and safely replays the provider key or queries its transaction reference. A signed webhook may already have moved the order to PAID. Lock the order row and inspect its current state before transitioning, so a second OrderPaid event and another shipment are not created.

An unknown payment must not hold inventory silently forever: illustrative escalation targets are automatic reconciliation within 5 minutes and an operations review queue after 15 minutes, with a visible customer status. Set these deadlines through business policy. An expired TTL in UNKNOWN is not evidence that stock can be released and the order declared failed. If the provider cannot establish a final outcome, require controlled manual review and alert on unresolved value and reservation age.

For a confirmed rejection, compensation releases inventory. Conditionally transition the reservation from HELD to RELEASED; restore stock only when that transition changed a row, and cancel the order in the same transaction. A webhook racing a scheduled job cannot then restore stock twice. If charging succeeded but fulfillment is impossible, compensation is a refund: a distinct outbox event requests it, and REFUND_PENDING remains observable until reconciliation confirms the result. A saga compensates through another business operation; it does not erase history like a database rollback.

An outbox worker claims work in a short transaction using FOR UPDATE SKIP LOCKED, records a timed lease and ownership token, then closes the transaction before calling the provider. Only that token may acknowledge the claim; expired leases can be reclaimed. This produces at-least-once processing. Rather than claiming end-to-end exactly-once delivery, combine unique event IDs, consumer inbox records, idempotent state transitions, and provider keys. Quarantine poison messages after bounded attempts. PostgreSQL SKIP LOCKED supports queue-like consumption; it is not a general-purpose consistent reporting read.

Timeouts, retries, and circuit breakers solve different problems

Illustrative budgets: an 800 ms order-acceptance deadline, 100 ms database lock wait, and a payment worker with a 300 ms connect timeout and 3 second total call deadline. Tune these starting values from measurements. Retry only transient failure classes when the provider contract and remaining deadline permit it: for example, one initial call plus at most two retries with exponential backoff and full jitter. Business rejections are not retryable. Ambiguous outcomes follow reconciliation.

A circuit breaker stops adding load to an unhealthy provider; bulkheads isolate payment connections and concurrency from other workers. Limits on queue age and size provide backpressure. Three attempts at each of three layers can multiply into 27 downstream calls, so one layer owns retries. Design reference: AWS on timeouts, retries, and jitter.

4. Observability and SRE: Is a Running System Actually Healthy?

CPU can be at 20% and /health can return 200 while customers never receive payment outcomes. Liveness measures process survival, readiness measures the ability to accept traffic, and business SLIs measure promises kept to users. Restarting every process because a third party is unavailable amplifies the incident. Order acceptance and payment completion need separate SLIs.

Fix the eligible request population in the contract: checkout requests with valid identity and schema. Expected stock rejection or card decline is a documented business outcome and tracked separately; server 5xx, deadlines, and capacity-driven 429 responses are failures. Explicit abuse limits are reported separately. Do not improve an SLO by changing its denominator after an incident. At 100,000 eligible monthly order requests, 99.9% availability allows 100 bad requests, while the 95% latency target allows up to 5,000 requests above 300 ms. These are distinct budgets.

Metrics expose rates, errors, and duration histograms; logs record state transitions and failure classes; distributed traces connect API → database → outbox → worker → provider. Carry trace context across asynchronous boundaries or use span links. Keep order_id and trace_id in logs and traces; per-order metric labels create high cardinality and cost. Do not log card data, credentials, or personal information inside prompts. OpenTelemetry signals provide a shared vocabulary.

# 5-minute p95; compute over merged buckets, never average per-instance p95.
histogram_quantile(0.95,
  sum by (le) (
    rate(order_acceptance_duration_seconds_bucket{eligible="true"}[5m])
  )
)

# Failure ratio: counter includes 5xx, deadline and capacity rejection.
sum(rate(order_acceptance_bad_total[5m]))
/
clamp_min(sum(rate(order_acceptance_eligible_total[5m])), 0.000001)

# 99.9% availability => error budget fraction = 0.001
# burn_rate = observed_failure_ratio / 0.001

A dashboard is not an on-call policy. For example, page when both the 1-hour and 5-minute windows show 14.4× budget burn; add minimum event counts and synthetic checks at low traffic. Every alert needs an owner, runbook, provider-status check, safe traffic reduction, and incident record. Rapid budget consumption pauses risky deployments; track customer impact and prevention work. Google SRE’s SLO implementation guide explains the underlying approach.

5. Performance Engineering: Capacity Planning and Scalability

“We will add servers” is not a capacity plan. The API may wait on I/O, workers may hit provider quotas, and PostgreSQL may serialize on one hot inventory row. Little’s Law estimates average in-flight work as L = λ × W. At 200 requests/s and an assumed mean response time of 120 ms, about 24 requests are in flight. Use the mean, not p95. This does not imply 24 database connections; connection hold time must be measured separately.

Do not load-test real customer data or send artificial load to a live payment provider. Use separate staging with production-like indexes, data distribution, TLS, and logging. Illustrative setup: two API instances at 2 vCPU/4 GiB each, two worker processes, separate PostgreSQL at 4 vCPU/8 GiB, an independent load generator, and network latency scenarios from 5–50 ms. This hardware has not yet been proven to meet the targets.

The mix is 70% catalog/status reads, 20% new checkouts, and 10% repeated requests with existing keys. At a 200 requests/s peak, new payment arrivals are about 40 jobs/s. With an assumed 500 ms mean provider response, 32 worker slots have a theoretical ceiling of 64 jobs/s before quotas, retries, and tail latency. A 70% utilization target leaves about 44.8 jobs/s, a narrow margin above 40. Provider quotas and queue age must therefore be measured before declaring capacity sufficient.

Use a load generator that maintains the intended arrival rate. A closed model that also slows arrivals when the system slows can hide latency through coordinated omission. Report offered and completed throughput, dropped iterations, timeouts, and every HTTP outcome together. Uniform random SKUs are insufficient: separately test a hot row receiving 20% of requests and concurrent callers reusing one idempotency key.

The evidence package includes application commit, migration version, test seed, load script, environment settings, and raw histograms. Optimize from measurements: remove N+1 queries, verify the (tenant_id, created_at) index, paginate, and bound connection pools before adding cache or instances. Total connections equal API instances × pool size plus workers and maintenance headroom. Giving every process 100 connections does not increase database capacity.

6. Production-Grade AI: Why LLM Integration Is More Than an API Call

In the same system, let the LLM draft a customer explanation for a delayed order. It cannot approve payment, decrement inventory, or initiate a refund. Authoritative services and the database supply the actual order state; the model drafts from those facts. AI workers have their own queue and concurrency budget, so ordering and payment continue when the LLM is unavailable.

{
  "order_id": "O-104",
  "template": "payment_under_review",
  "draft": "Your payment status is being verified.",
  "evidence_refs": ["order:O-104:status:v7"],
  "requires_review": true
}

Valid JSON is not proof of correctness. After schema validation, check that order_id matches the request, evidence references are authorized, and the draft does not contradict payment state. A model returning “requires_review: false” gains no authority; an external policy decides review requirements. Filter retrieval by tenant and enforce access again on retrieved records. Instructions inside customer notes or documents are untrusted data. OWASP’s prompt injection guidance explains why a system prompt alone cannot enforce this boundary.

Illustrative limits: at most 2,000 input and 400 output tokens per job, an 8 second deadline, bounded global concurrency, and tenant quotas. On timeout or validation failure, use a reviewed deterministic template rather than an unbounded “repair JSON” loop. Record model, prompt, retrieval, and schema versions without logging raw personal data. Evaluation must cover wrong-tenant documents, empty retrieval, conflicting payment facts, and injected “refund first” instructions as well as normal cases.

Track schema validity, verified evidence use, human acceptance, fallback rate, p95 duration, and cost per accepted draft. A cross-tenant disclosure or unauthorized action attempt triggers a stop; it has no tolerated error budget. Evaluate a new model against a versioned regression set before a limited canary. A low token bill does not compensate for the operational cost of sending an explanation to the wrong customer.

7. Architectural Decision Records: Monolith, Microservices, and Unnecessary Complexity

An ADR is not a technology shopping list. Record context, alternatives, the decision, trade-offs, ownership, and a trigger for reconsideration. Assume four engineers, no dedicated platform team, and no demonstrated need for independently released business domains. These constraints are more consequential than technology popularity.

Why Redis? Only if measurements justify its latency or database-load benefit. A catalog might tolerate a 30 second TTL plus invalidation, while checkout still verifies price and inventory in PostgreSQL. TTL jitter and request coalescing prevent a wave of misses from overwhelming the database. Define fail-open or fail-closed behavior per endpoint when rate limiting is unavailable; payment and administration security cannot depend solely on cache availability.

Why not Kafka? The present need is to distribute work to a few workers, not yet to maintain a high-volume retained event log replayed by many independent consumers. Kafka is not the wrong technology; its partitioning, schema, lag, and operational ownership are simply not justified by this requirement. Why is Kubernetes unnecessary here? A scheduler does not design transaction correctness, provider idempotency, or backup recovery. The team still needs health checks, controlled rollouts, secret management, and capacity planning.

If a module becomes a microservice, separate containers writing the same tables do not establish independence. Separate data ownership and contracts; messaging, sagas, and reconciliation replace cross-service transactions. Record the development and incident cost in the ADR. Introduce a service boundary to enable independent change or failure isolation, not merely to divide code.

8. Software FinOps: How Architecture Affects TCO and Operating Cost

The cloud bill is only part of TCO. Compute, database, storage, egress, observability, and model calls are direct costs; maintenance, on-call, incidents, and change lead time consume engineering effort. Amortize initial implementation separately. The table below is a hypothetical accounting model for comparing designs at equal volume, not a quote or a provider price list.

At 1 million eligible monthly API requests, the first design costs 640 / 1,000,000 = USD 0.00064/request in direct service expense, or USD 0.00124/request including maintenance. Allocating that total across 100,000 accepted orders yields USD 0.0124/order. If only 90,000 orders are successfully paid, the same total becomes about USD 0.01378 per paid order. “Cost per request” is misleading without a named denominator and scope. Add payment fees, support, incident cost, and initial development amortization separately in the TCO report.

# Hypothetical token rates; NOT a vendor price quote.
input_rate_per_million  = 1.00  # USD
output_rate_per_million = 4.00  # USD
cost_per_generation = (2000 * 1.00 + 400 * 4.00) / 1_000_000
# = 0.0036 USD

generations = 10_000
retry_multiplier = 1.10
monthly_model_cost = generations * retry_multiplier * cost_per_generation
# = 39.60 USD (~40 budgeted)

accepted_drafts = 8_000
model_cost_per_accepted_draft = 39.60 / accepted_drafts
# = 0.00495 USD; human review and retrieval are additional.

# TCO(period) = implementation + migration + recurring services
#             + engineering operations + incident impact + exit cost

Calculate automation savings net of overhead: saving 3 minutes across 100,000 monthly operations is 5,000 theoretical hours, before adoption, exceptions, and human review. Logging entire responses expands storage, unbounded retries expand model spend, and unnecessary service boundaries expand on-call work. The illustrative table differs by USD 1,550 per month or USD 18,600 in annual recurring cost; this is not a universal microservice cost multiplier.

Budget alerts should follow tenants and business capabilities: accepted orders, successful charges, and accepted AI drafts. When a daily limit is reached, reduce optional AI first rather than disabling payment correctness or audit records. Record that halving traffic can roughly double fixed-cost unit economics. See FinOps Foundation’s Unit Economics capability for the underlying cost-accountability approach.

9. Production Readiness Checklist: When Can a System Go Live?

“Tests are green” is not a sufficient go-live decision. Every acceptance gate needs evidence, an owner, and an action if it fails. The gates below belong to this order/payment scenario; other business and data contexts can add requirements. Critical correctness and authorization violations are not deferred as post-launch cleanup.

Verify security in the workflow: derive tenant identity from the authenticated session, not the payload; knowing an order ID grants no access. Use least-privileged database roles, keep secrets out of source code, and test rotation. Prefer provider tokenization or hosted checkout so card data does not pass through the application server. Verify webhook signatures over the raw body and enforce the provider’s time window; persist provider_event_id with a uniqueness constraint and return 2xx only after durable receipt. Late events must not move PAID backwards. Consult the provider’s webhook documentation for exact behavior.

Illustrative recovery objectives are RPO ≤ 5 minutes for database data and RTO ≤ 30 minutes for service restoration; they remain targets until a drill verifies them. A provider may hold charges newer than the restored database. Reconcile ambiguous payments and reservations against provider records before reopening customer traffic. A nightly backup cannot support a five-minute RPO; continuous archiving/PITR and isolated restore practice are needed. A completed backup is not evidence of a successful restore.

Use expand–migrate–contract for deployments: add a schema readable by both versions, migrate data in the background, switch reads, and remove the old field in a later release. Rolling back a binary after a destructive migration may be unsafe. On SIGTERM, a worker stops claiming work and either finishes within its deadline or lets its lease expire safely. Watch SLOs, payment inconsistencies, and cost during the canary.

The product owner accepts explicit business risks, the technical owner signs off correctness evidence, and operations owns recovery and alert drills. Missing evidence delays traffic or reduces scope explicitly—for example, AI remains disabled. Production readiness is a demonstrated contract for a particular product, load, and failure model, not a permanent badge.

We design the conditions under which software remains reliable: its supported load, failure behavior, data guarantees, and engineering cost.

Start with one critical workflow: define its invariants, SLO, compensation behavior, and unit economics together. Discuss your engineering needs. Related technical reading: system integration and modern enterprise software architecture.