API integrations usually fail in ways that do not appear in the happy-path demo. Networks time out after the remote service has already completed an action. Webhooks arrive twice or out of order. Rate limits appear under real load. Credentials expire. Schemas change. A downstream service succeeds while the local database transaction fails.
Production integration engineering is therefore less about making one API call succeed and more about keeping two systems consistent when failures, delays, duplicates, and partial completion are normal.
This guide describes the failure modes that should be designed before launch. For broader engineering-vendor evaluation, see How to Choose a Custom Software Development Company.
1. A timeout is not proof of failure
If a request times out, the remote service might have:
- never received it;
- received and rejected it;
- completed it before the response was lost;
- still be processing it.
This uncertainty matters for side effects such as charging a card, creating an order, provisioning a resource, or sending a message.
Before retrying, ask whether the operation is safe to repeat and whether the destination exposes a request/operation identifier that can confirm the previous result.
2. Make write operations idempotent where retries are possible
AWS defines an idempotent operation as one where repeated calls have the same system-state effect as a single call. Its Builders' Library recommends caller-provided unique request identifiers so retries can be recognized as the same logical operation.
A common pattern is:
- generate an idempotency/request key for one logical operation;
- persist it locally before the external side effect;
- send it to the destination if the API supports it;
- reuse the same key for retries;
- record the destination result/identifier;
- return the stored result for duplicate requests.
Do not generate a new key for every retry; that defeats duplicate detection.
3. Retry only transient failures
Retries can improve resilience, but uncontrolled retries can make an outage worse.
AWS Well-Architected recommends bounded retries with exponential backoff and jitter, and specifically notes that retries should be used with idempotent operations when repeated execution can create side effects.
Classify errors:
- transient: network failure, temporary 5xx, some throttling conditions;
- permanent: invalid input, forbidden operation, unknown resource, unsupported version;
- unknown outcome: timeout or connection loss after the request may have been accepted.
Fail fast on permanent errors. Reconcile unknown outcomes before blindly repeating a side effect.
4. Respect rate limits as a capacity signal
429 responses are not just errors to suppress.
Design for:
- provider retry-after guidance;
- exponential backoff;
- jitter;
- maximum retries/time budget;
- local concurrency limits;
- queues for burst smoothing;
- tenant/customer quotas when appropriate.
Retry storms can turn a partial dependency problem into a wider outage.
5. Separate synchronous requests from long-running work
Some integrations should not keep a browser request open while work completes.
Use asynchronous processing for:
- large imports/exports;
- media processing;
- bulk synchronization;
- external provisioning;
- slow AI or document workflows;
- high-volume webhook processing.
AWS guidance for asynchronous communication recommends durable persistence before acknowledgement, idempotent message processing, dead-letter handling, and monitoring processing success.
6. Treat webhooks as at-least-once, delayed, and potentially out of order
Even if a provider usually sends one webhook quickly, design as if:
- the same event can arrive more than once;
- events can be delayed;
- events can arrive out of order;
- delivery can fail temporarily;
- the receiver can be down during delivery.
Store provider event IDs and deduplicate them. Verify signatures. Process asynchronously where possible. If ordering matters, compare event versions/timestamps or query the provider's authoritative state instead of trusting arrival order.
7. Plan for partial failure across systems
A distributed workflow does not share one database transaction.
Example:
- your system creates an order;
- payment succeeds;
- inventory reservation fails;
- the response process crashes.
You need an explicit recovery policy: retry inventory, compensate/refund payment, move the order to manual review, or another business-approved state.
Model integration workflows as state machines with durable statuses rather than a chain of assumptions inside one request handler.
8. Add reconciliation jobs
Webhooks and API responses are not enough for high-value integrations.
Periodically compare local state with the authoritative external system for objects where inconsistency matters: payments, orders, subscriptions, shipments, provisioning, or financial postings.
Reconciliation detects missed webhook deliveries, partial failures, manual external changes, and bugs that normal request logs may not reveal.
9. Version API contracts
External providers change schemas and behavior. Internal teams change APIs too.
Track:
- provider/API version;
- deprecated fields/endpoints;
- required migration deadlines;
- schema validation;
- backward compatibility;
- consumer contract tests;
- feature-flagged rollout where needed.
Do not let a third-party deprecation become an emergency because nobody owns the integration lifecycle.
10. Secure outbound and inbound integrations
OWASP's API Security Top 10 highlights risks such as broken object-level authorization, broken authentication, unrestricted resource consumption, SSRF, inventory problems, and unsafe consumption of APIs.
For integration services:
- use scoped credentials;
- store secrets outside code;
- verify webhook signatures;
- validate remote payloads;
- validate URLs/hosts for server-side fetches;
- apply authorization at object/function level;
- maintain an inventory of active APIs and versions;
- rotate credentials and revoke unused ones.
An authenticated API is not automatically a safe API.
11. Record correlation identifiers
Every integration flow should be traceable across systems.
Record identifiers such as:
- internal request/job ID;
- idempotency key;
- provider request ID;
- provider object/transaction ID;
- webhook event ID;
- tenant/customer ID;
- retry attempt;
- final status.
This turns "something duplicated" into an investigable chain of events.
12. Define timeout budgets at each layer
One giant timeout hides which dependency is slow.
Set explicit budgets for:
- connection timeout;
- request/read timeout;
- overall workflow deadline;
- queue/job timeout;
- maximum retry elapsed time.
The caller's deadline should be longer than downstream budgets, leaving time to handle an error cleanly rather than being terminated mid-recovery.
13. Integration red flags
| Red flag | Failure it creates |
|---|---|
| Retry every error immediately | Retry storms and duplicated side effects |
| No idempotency strategy for writes | Duplicate payments/orders/resources |
| Webhook handled directly in a long request | Timeouts, missed acknowledgements, repeated processing |
| Webhook arrival order treated as truth | Older events overwrite newer state |
| No reconciliation | Silent divergence persists indefinitely |
| No API-version/deprecation owner | External changes become emergency outages |
| Remote payload trusted without validation | Unsafe consumption and security defects |
| No correlation IDs | Incidents cannot be reconstructed |
14. Production readiness checklist
- Which operations are idempotent?
- Which errors are retried?
- What is the backoff/jitter policy?
- What is the maximum retry/time budget?
- How are unknown outcomes reconciled?
- How are webhook duplicates/out-of-order events handled?
- Where are failed asynchronous messages sent?
- How are credentials rotated?
- How are provider version/deprecation changes tracked?
- How are local and remote state reconciled?
- Which IDs allow an incident to be traced end-to-end?
Reliable integrations assume networks and dependencies fail. The architecture makes those failures recoverable instead of surprising.
Sources and further reading
- AWS Builders' Library — Making retries safe with idempotent APIs
- AWS — Retry with backoff pattern
- AWS — Asynchronous communication patterns
- OWASP — API Security Top 10
For implementation, review OSYSTIC Product Engineering, Cloud & DevOps Engineering, or send an integration brief.