Where External System Integrations Fail — Principles of Interface Design

When an incident is reported on a live service and we open the logs, the cause turns out to be the business logic we wrote less often than you might expect. Most of the time it is a point where our system touches another one.

The payment provider responded more slowly than usual, a channel partner's API added a field without notice, or the other system was under maintenance and we did not know. Our code has not changed, and the incident happens anyway.

SI projects have an unusually large number of these boundaries. Even when building a single new system, it almost always has to be connected to existing in-house systems, external solutions, and public APIs. In the lodging operations system we worked on alone, the booking channels, payments, room control, and the accounting system were each connected in a different way.

This article summarizes the principles we find ourselves checking again and again when designing those boundaries. It focuses on material that is not tied to a particular technology stack.

---

1. Fix the contract before writing code

Decide the integration method first

"We'll connect through an API" means nothing has been decided yet. There are several integration methods you run into in practice, and each carries a different design and operational burden. When the other side is an old in-house system in particular, a REST API is often not an option.

Of these, file transfer is an old method but is still widely used for settlement work. The point that causes trouble repeatedly is "how do we know the file has been uploaded completely?" If you read a file that is still being written, the data is truncated. This is solved by uploading a completion marker file alongside it, or by uploading under a temporary name and then renaming the file.

Direct database access is the fastest way to connect, but we do not recommend it. If the other system changes a single column, our service stops, and there is no channel through which we would be notified of that change in advance. If it cannot be avoided, it is better to at least request a read-only account and a dedicated view.

Do not start without a document

The most common failure pattern in integration work is starting without a document. Developing while exchanging fields with the other system's contact over a messenger looks fast at first, but once the system is in operation, "what did we agree on back then?" comes up again and again.

What the interface specification must contain

The format does not matter, whether it is a document or a schema file. But if any of the following items is missing, it becomes a problem later.

The last item, operations, is often left out. Finding out a month into operation that the other system goes into maintenance in the early hours of every Tuesday is a situation that can be avoided.

The need for the result lookup item often becomes apparent only late in development. If you send a request, receive no response, and have no way to check whether it was processed, the timeout and retry design discussed below does not hold. If the other side has no lookup method, it is better to request one at the start of the project.

For the errors item, check where the error is carried. Some older systems respond with HTTP 200 even on failure and report success or failure through a result code in the body. In that case, if you treat the request as successful based on the status code alone, failed requests are stored as normal ones.

For the authentication item, do not tie the key to a person's account. If a key issued under a staff member's account is used for the integration, the integration breaks on the day that person leaves and the account is cleaned up. In B-Saver, which we built, the daemon and backend services call the API with separately issued API tokens rather than a personal account.

Write value formats as examples, not descriptions

The places where a specification becomes ambiguous are largely predictable.

• Date and time — Whether it is 2026-09-21T14:30:00+09:00 or 20260921143000, and whether the time zone is KST or UTC. If the other side sends UTC and we interpret it as KST, the value is off by nine hours
• Amounts — Whether it is an integer in KRW, whether there is a decimal point, and whether a currency code comes with it. Once overseas payments are involved, a currency code is mandatory
• Character set and length — Whether the length is in bytes under UTF-8 or in characters. The result differs once Korean text is involved
• Null and empty string — When there is no value, whether the field is omitted, null is sent, or "" is sent. Cases where all three are mixed do occur
• Boolean — Whether it is true, "Y", or 1
• Identifiers — Length limit and allowed characters. If the other side accepts only up to 20 characters and we send a 36-character UUID, it is truncated

Writing down an actual example value for each item removes almost all differences in interpretation.

Agree on the versioning policy first

An integration contract will change. The key is to decide in advance what happens when it does.

• Adding a field must not break compatibility. The receiving side is implemented to ignore fields it does not know
• Removing a field or changing its meaning is split into a new version
• Set an advance notice period for changes
• State the end-of-support date for the old version

On the receiving side, it is better to avoid an implementation that "raises an error when a field not in the specification arrives." Our service would then stop simply because the other side added one field.

---

2. Design with failure as the default

External calls fail. This has to be designed as one of the normal execution paths, not as an exceptional situation.

Set timeouts in three places

An external call without a timeout propagates failures. When the other system slows down, our server's threads or connections are tied up as they are, and eventually unrelated features stop along with it.

• Connection timeout — The time to wait for the TCP connection to be established. It can be kept short
• Read timeout — The time to wait for response data. Set it by measuring the other system's typical response time
• Total timeout — The upper limit on total elapsed time, including retries

What matters here is setting it smaller than the time budget of the upstream caller. If a user request has to finish within 3 seconds and it makes a 5-second external call internally, the timeout is meaningless.

Connection and read timeouts are separated because the two failures mean different things. A connection timeout means the request did not reach the other side, so sending it again is safe. A read timeout means the request was already delivered and only the response was not received, so we cannot know whether the other side processed it.

The read timeout value is not chosen by feel; it is picked from the other side's response time distribution. AWS describes an approach in which you first decide the rate at which normal requests may be cut off wrongly by a timeout (for example, 0.1%) and then use the corresponding response time percentile (p99.9) as the timeout. (AWS Builders' Library, Timeouts, retries, and backoff with jitter) Early in an integration there are no measurements, so record response times in the test environment to set an initial value, and adjust it again once production data has accumulated.

Do not leave library defaults as they are. The default timeouts of HTTP clients are often far too long for integration purposes. Taking undici, which the built-in fetch of Node.js uses, as an example, according to the current documentation the default connection timeout is 10 seconds, and the defaults for waiting for the response headers and for the body are 300 seconds each. (undici Client documentation) Used without configuration, a single request holds a connection for 5 minutes when the other side does not respond.

Below is an example that sets the three timeouts separately. It assumes Node.js 20 or later with the undici package installed, and the figures are illustrative.

import { Agent } from 'undici';

export const partnerAgent = new Agent({
  connect: { timeout: 2_000 }, // until the connection is established
  headersTimeout: 5_000,       // until the response headers arrive
  bodyTimeout: 5_000,          // interval between body chunks
  connections: 20,             // upper limit on concurrent connections to one partner
});

export const TOTAL_DEADLINE_MS = 12_000; // overall upper limit including retries

The error code differs by the type of timeout. undici distinguishes a connection failure as UND_ERR_CONNECT_TIMEOUT, exceeding the header wait as UND_ERR_HEADERS_TIMEOUT, and exceeding the body wait as UND_ERR_BODY_TIMEOUT. (undici Errors documentation) If you keep this distinction as it is in the integration log, you can later tell "whether we failed to reach the other side, or reached it and got no answer."

A timeout is not a failure; it is an unconfirmed result

Recording a timed-out request as a failure right away is the most common mistake in integrations. We only failed to receive the response; the other side may have finished processing. In the case of a payment, the approval went through but it remains a failure on our side, and when the user tries again it becomes a double payment.

So instead of having only two states, success and failure, we add a pending confirmation state.

Requests pending confirmation are settled through the other side's result lookup. If the lookup also fails, hand the request over to a job that rechecks periodically, and if it is still not settled after a certain time, notify an operator. When the other side is something like a payment terminal, where responses are often late or cut off midway, this procedure becomes the center of the design. In the payment terminal (VAN) integration of B-Saver as well, the polling interval, timeouts, and retry conditions are documented separately and managed. How the same problem appears in kiosk payments is covered in What Takes Longer Than Development in a Kiosk Project — A Checklist for Building Unmanned Systems.

Retry by condition

Not every failure should be retried. When the other system is already struggling, a flood of retries deepens the outage.

The meaning of the status codes is based on the HTTP specification. 503 means the request cannot be handled because of temporary overload or maintenance, and the server can indicate the wait time with Retry-After. (RFC 9110, 15.6.4) 408 means the server did not receive the complete request message within the time it was prepared to wait, and the client can repeat the request. (RFC 9110, 15.5.9) 429 is defined not in RFC 9110 but in a separate document, and it can likewise include Retry-After. (RFC 6585, Section 4)

However, there is no guarantee that the other system uses status codes according to the specification. Some systems return 500 for input errors, so treat the table above as the default, obtain the other side's list of error codes, and finalize the retry targets again.

Retry-After comes in two formats

The Retry-After value is either an integer number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Fri, 31 Dec 1999 23:59:59 GMT). (RFC 9110, 10.2.3) If the implementation handles only seconds, it cannot calculate the wait time when the date format arrives, and depending on the implementation it retries immediately with no wait. Parse both formats, and if the indicated time is longer than our total timeout, do not wait and pass the request on as a failure.

How to add jitter to backoff

The baseline for retry intervals is exponential backoff with a random delay (jitter) added. Retrying at a fixed interval makes the failed requests pile up again at the same moment. Even if the interval is simply doubled each time, requests that failed at the same instant are still resent at the same instant, so the wait time itself has to be spread out.

An article on the AWS Architecture Blog compared this problem through simulation. The approach of picking the wait time at random between 0 and the exponential cap (full jitter) resulted in fewer total calls, and the approach of keeping half of the cap fixed and spreading only the rest (equal jitter) had the worst results among the jittered approaches. (AWS Architecture Blog, Exponential Backoff And Jitter)

Below is an example that adds retries to the timeout settings above. It assumes the same environment, and the figures are examples.

import { setTimeout as sleep } from 'node:timers/promises';
import { fetch, type Response } from 'undici';
import { partnerAgent, TOTAL_DEADLINE_MS } from './partner-agent';

const RETRYABLE_STATUS = new Set([408, 429, 500, 502, 503, 504]);
const BASE_DELAY_MS = 200;
const MAX_DELAY_MS = 5_000;
const MAX_ATTEMPTS = 4;

function backoffDelay(attempt: number): number {
  const ceiling = Math.min(MAX_DELAY_MS, BASE_DELAY_MS * 2 ** attempt);
  return Math.random() * ceiling; // full jitter
}

function retryAfterMs(header: string | null): number | null {
  if (header === null) return null;
  if (/^\d+$/.test(header)) return Number(header) * 1000;
  const retryAt = Date.parse(header);
  return Number.isNaN(retryAt) ? null : Math.max(0, retryAt - Date.now());
}

export async function postWithRetry(
  url: string,
  body: string,
  idempotencyKey: string,
): Promise<Response> {
  const deadline = AbortSignal.timeout(TOTAL_DEADLINE_MS);

  for (let attempt = 0; ; attempt += 1) {
    const isLastAttempt = attempt === MAX_ATTEMPTS - 1;
    let waitMs = backoffDelay(attempt);
    try {
      const response = await fetch(url, {
        method: 'POST',
        headers: { 'content-type': 'application/json', 'idempotency-key': idempotencyKey },
        body,
        dispatcher: partnerAgent,
        signal: deadline,
      });
      if (!RETRYABLE_STATUS.has(response.status) || isLastAttempt) return response;
      waitMs = retryAfterMs(response.headers.get('retry-after')) ?? waitMs;
      await response.body?.cancel(); // discard the unread body to release the connection
    } catch (error) {
      if (deadline.aborted || isLastAttempt) throw error;
    }
    await sleep(waitMs, undefined, { signal: deadline });
  }
}

There are three things to note in this example. The idempotency key is created outside the function and passed in, so the same value is kept throughout the retries. A single total-timeout signal is applied to both the calls and the waits, so execution stops once the limit is exceeded even if retries remain. Retrying on 500 and on response timeouts holds only on the premise that the other side supports idempotency keys, so if it does not, they must be removed from the retry targets.

Retry at only one layer

The number of retries must not be unlimited. Once the limit is reached, the request must be settled as a failure and passed on to the next procedure.

Just as important as the count is where the retry happens. If the screen, the API server, and the integration module each retry, the counts multiply. The AWS Builders' Library article cited earlier gives the example that if five layers each retry three times, the load on the database at the bottom becomes 243 times, and recommends retrying at a single point only. For integrations, a safe arrangement is to retry only in the one module that makes the external call directly, with the layers above passing the result through as is. Also check whether the HTTP client library or SDK retries by default. There are cases where retries are happening twice over without anyone knowing.

Without idempotency, you cannot retry

This is the precondition for retrying. Sending the same request several times must produce the same result as sending it once.

The HTTP specification defines GET, HEAD, PUT, DELETE, OPTIONS, and TRACE as idempotent methods, and POST is not idempotent. (RFC 9110, Section 9.2.2) The same section advises that a client should not automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent or a means to detect that the original request was never applied. The problem is that the calls that matter in practice, such as payment requests and order creation, are mostly POST.

A method being classified as idempotent does not mean the other side's implementation is idempotent either. Some APIs accept PUT but are implemented internally to add to a quantity on every call, so confirm with the other side before including them in the retry targets.

The solution is to use a unique key per request.

1. The calling side generates a unique key for each request
2. It sends that key in the request header or body
3. The receiving side stores the key, and when the same key comes in again, it returns the previous result instead of processing it anew

The key must be kept identical across retries. Generating a new key on each retry removes the idempotency. A common mistake is generating the key inside the function that makes the HTTP call. A new key is then created every time the function is called again. Generate the key once at the point where the business request is created, store it together with the business data, and read the stored value when sending. This is because the same key has to go out even on a resend after the process has restarted.

The Idempotency-Key header is not yet a standard

Idempotency-Key is widely used as the name of the header that carries the key, but as of October 2026 this header has not been finalized as an RFC. It progressed as an Internet-Draft of the IETF HTTPAPI working group, and the last revision (07, October 2025) expired in April 2026. (IETF Datatracker, draft-ietf-httpapi-idempotency-key-header) So you cannot assume that "it is a standard, so the other side will support it," and the header name and behavior have to be confirmed in the other side's documentation. The implementation status section of the draft also lists services that use a header of the same name alongside services that use a different name.

Even though it is not a standard, the draft is useful as a basis for organizing the items the two sides need to agree on. The draft specifies the following. (draft-ietf-httpapi-idempotency-key-header-07)

In the draft's syntax the value is sent as a string enclosed in double quotes (Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"), but the actual notation follows the format the other side gives in its documentation.

Implement receiver-side idempotency with a unique constraint

When we are the side receiving requests, the most common implementation error is splitting "look up whether the key exists, and process if it does not" into two steps. If two requests with the same key arrive almost simultaneously, both confirm "not found" and both are processed. Retries often arrive while the original request has not yet finished, so this race is not rare.

Instead of a lookup, the key must be claimed with an insert under a unique constraint. Below is an example that assumes PostgreSQL 9.5 or later. ON CONFLICT was added in 9.5. (PostgreSQL 9.5 release notes)

CREATE TABLE idempotency_keys (
  idempotency_key text        PRIMARY KEY,
  request_hash    text        NOT NULL,
  status          text        NOT NULL DEFAULT 'PROCESSING',
  response_code   int,
  response_body   jsonb,
  created_at      timestamptz NOT NULL DEFAULT now()
);

-- Run this first for every request. If one row is returned, the key is being seen for the first time.
INSERT INTO idempotency_keys (idempotency_key, request_hash)
VALUES ($1, $2)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key;

If no row is returned, the key already exists, so read the existing row and branch.

Points to watch in operation:

• Do not leave behind the key of a request that died during processing. If the key insert and the business processing are placed in one transaction, the key disappears together when processing fails and rolls back, so a retry is processed again normally. In this case, a concurrent request with the same key waits at the insert until the earlier transaction ends. If the transaction has to be split because there is an external call during processing, put an expiry time on the PROCESSING row and allow expired rows to be claimed again. Without either, only a key in the PROCESSING state remains and subsequent retries keep receiving 409
• Set the retention period longer than the other side's maximum retry period. If the other side resends for a day and the key is kept for only an hour, a resend that arrives after that is processed as a new request
• Define the scope of the key. If there are several callers, make the caller identifier and the key together the unique condition. This is to prevent the case where different callers send the same value
• Do not skip the body comparison. If the previous result is returned based on the key alone, then when a different amount arrives under the same key because of a bug in the caller, the wrong request is answered as a success

For webhooks, verify the signature and check for duplicates together

The same principle applies to callbacks and webhooks coming in from outside. They can be delivered more than once, so a step is needed before processing to check whether the event has already been processed.

Webhooks come with one more requirement. The receiving address is open to the outside, so you have to verify that an incoming request was actually sent by the other side. If you process without verifying, anyone who knows the address can create and send a "payment completed" notification. The usual approach is to compute the HMAC-SHA256 value of the body with a shared secret and carry it in a header. GitHub puts a value starting with sha256= in the X-Hub-Signature-256 header (GitHub Docs, Validating webhook deliveries), and Stripe signs a string made by concatenating the timestamp and the body and carries it in the Stripe-Signature header together with the timestamp. (Stripe Docs, Webhooks) What is signed and the header format differ for each counterpart, so finalize them in the specification.

Below is a verification example that assumes a scheme in which the timestamp and the body are joined with . and signed. It is based on Node.js 18 or later.

import { createHmac, timingSafeEqual } from 'node:crypto';

const TOLERANCE_SECONDS = 300;

interface WebhookSignatureInput {
  rawBody: Buffer;
  timestamp: string;
  signatureHex: string;
  secret: string;
}

export function isValidWebhookSignature(input: WebhookSignatureInput): boolean {
  const ageSeconds = Math.abs(Date.now() / 1000 - Number(input.timestamp));
  if (!Number.isFinite(ageSeconds) || ageSeconds > TOLERANCE_SECONDS) return false;

  const expected = createHmac('sha256', input.secret)
    .update(`${input.timestamp}.`)
    .update(input.rawBody)
    .digest();
  const received = Buffer.from(input.signatureHex, 'hex');

  return received.length === expected.length && timingSafeEqual(received, expected);
}

The causes of verification failure are largely predictable.

The reason an ordinary string comparison is not used for comparing signatures is that the value could be guessed from the time the comparison takes. crypto.timingSafeEqual in Node.js compares using a constant-time algorithm and throws an error if the byte lengths of the two values differ. (Node.js crypto documentation) The reason the timestamp is included in the signature is to prevent replay attacks, in which an intercepted request is sent again later. Stripe's official libraries set the default tolerance to 5 minutes.

Even when the signature is valid, the order of processing remains.

• Respond quickly and defer the processing. Store the raw payload as soon as it is received, return 2xx, and process it in a separate job. If the response is late, the other side treats it as a failure and sends it again
• Filter duplicates by event identifier. Put a unique constraint on the event ID in the same way as the idempotency key table above
• Do not assume ordering. Stripe does not guarantee that events are delivered in the order in which they were generated, and states in its documentation that the same event may be delivered more than once. When you receive a notification, it is safer to query the current state again through the other side's API than to trust the values in the body as they are
• Do not rely on webhooks alone. A notification we failed to receive because of an outage on our side disappears once the other side's redelivery period has passed. Omissions have to be caught by the reconciliation discussed below

Do not go down when the other side goes down

When an external system is in a prolonged outage, continuing to attempt calls causes two problems. Our resources are consumed, and the other side's recovery is delayed.

The circuit breaker pattern addresses this problem. When the failure rate exceeds a threshold, calls themselves are blocked for a certain time, and afterwards a small number of trial calls are used to check whether the other side has recovered.

There are three states. Normally, calls are passed through as they are while failures are counted (closed). When failures reach the threshold, subsequent calls are not sent to the other side and are returned immediately as errors (open). After the set time has passed, trial calls are sent; if they succeed the breaker returns to closed, and if they fail it waits again (half-open). (Martin Fowler, Circuit Breaker)

The names differ from library to library, but the values that have to be decided are similar.

The last row is often set wrongly. Responses such as insufficient balance or a missing required value mean the other side is operating normally, so they are not counted as failures.

The unit of blocking also has to be decided. If the entire counterpart system is treated as one unit, a single slow lookup API causes even the healthy payment API to be blocked. It is better to split by the other side's functional units. If there are several servers, also be aware that the breaker state is usually determined separately on each server. There will be periods in which one server is open and another is closed, so to the user the same feature may appear to work at one moment and not the next.

What to return in the blocked state is the core of the design.

• Can the last cached value be used?
• Can processing proceed with a default value?
• Can only that feature be disabled while the rest keeps working?
• What message will be shown to the user?

"Return an empty list when the lookup fails" can be dangerous, because to the user it looks as though there is no data. It is safer to show a failure as a failure.

We control how much we send

Most external systems have a rate limit. Exceeding it gets us blocked, and even where there is no explicit limit, the other system slows down. We need a mechanism on our side to control the rate at which we send.

• Rate limiting — Limit the number of calls per second or per minute on our side first. It is better for us to throttle before the other side blocks us
• Concurrency limiting — Set an upper limit through the connection pool size. If connections open without limit, it burdens both the other side and us
• Spreading bulk work — If tens of thousands of items have to be sent as a batch, send them in portions rather than all at once
• Separating priorities — Separate real-time calls that a user is waiting on from synchronization jobs running in the background, so that they do not compete for the same resources

The last item is a frequent problem in practice. User requests slowing down while a nightly batch is running is usually caused by the lack of this separation.

Do not discard requests that ultimately failed

If a request that has exhausted its retries is left only in the logs and disappears, there is no way to recover it later. A structure that keeps failed messages in a separate store is needed.

• Record the failure cause, the time, and the retry count together
• Keep the full original request so that it can be reprocessed
• Provide a screen where operators can review and reprocess
• Trigger an alert when more than a certain number have accumulated

Without this structure, incident recovery becomes a matter of editing the database directly.

---

3. Manage the points where data diverges

The hardest problem to deal with in integrations is not outages but inconsistency: both systems ran without errors, and yet the data differs.

Distinguish between synchronous and asynchronous

Not every integration needs to be handled as a real-time call.

For work where the screen can show "Processing," running it asynchronously is more stable. The user is not kept waiting, and there is room for retries. The ship operation system management platform we developed also, for the same reason, puts execution requests in a queue and has a separate process carry them out. Remote tasks take a long time and can fail midway, so if the part that receives the request and the part that executes it are coupled, the screen is tied up for that time and it is also hard to track the state of a failed task separately.

The moment you write to two places, they diverge

Saving to our database and also sending to an external system cannot be bound into a single transaction. Situations in which only one of the two succeeds will occur.

① DB save succeeds → external send fails → only we know, the other side does not
② External send succeeds → DB save fails → only the other side knows, we do not

② is more dangerous. There is no record in our system, so there is no clue to trace it by.

In practice, we use an approach in which the save and the send are handled within one transaction, but the send is only scheduled. The business data and a pending-send record are saved in the same transaction, and a separate process reads the pending records and performs the actual send. If the send fails, the pending record remains, so it can be retried. This structure is called the transactional outbox pattern. (microservices.io, Transactional outbox)

Even with this approach, duplicate sends can occur. This is the case where the send succeeded but the process died before the completion mark was written. The idempotency discussed earlier is needed here.

Example outbox table layout

Below is a minimal layout that assumes PostgreSQL 9.5 or later. The intervals and counts are examples.

CREATE TABLE outbox (
  id              bigserial   PRIMARY KEY,
  idempotency_key uuid        NOT NULL UNIQUE,
  destination     text        NOT NULL,
  payload         jsonb       NOT NULL,
  status          text        NOT NULL DEFAULT 'PENDING', -- PENDING, SENT, DEAD
  attempts        int         NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  last_error      text,
  created_at      timestamptz NOT NULL DEFAULT now(),
  sent_at         timestamptz
);

CREATE INDEX outbox_due_idx ON outbox (next_attempt_at) WHERE status = 'PENDING';

Within the transaction that saves the business data, insert one row into outbox as well. The idempotency key is created at this point and stored with it. The sending process works in the following order.

-- 1) Claim the rows to send. The next attempt time is pushed back, so other workers do not pick up the same rows.
UPDATE outbox
SET attempts = attempts + 1,
    next_attempt_at = now() + interval '5 minutes'
WHERE id IN (
  SELECT id FROM outbox
  WHERE status = 'PENDING' AND next_attempt_at <= now()
  ORDER BY id
  LIMIT 50
  FOR UPDATE SKIP LOCKED
)
RETURNING id, idempotency_key, destination, payload, attempts;

-- 2) Send to the external system outside the transaction.

-- 3-a) On success, mark the row as done.
UPDATE outbox SET status = 'SENT', sent_at = now() WHERE id = $1;

-- 3-b) On a retryable failure, reset the next attempt time using backoff.
UPDATE outbox SET next_attempt_at = $2, last_error = $3 WHERE id = $1;

-- 3-c) On a non-retryable failure, or when the attempt limit is reached, quarantine the row.
UPDATE outbox SET status = 'DEAD', last_error = $2 WHERE id = $1;

SKIP LOCKED is an option that skips rows locked by another transaction instead of waiting for them, and the PostgreSQL documentation explains that it can be used to avoid lock contention on tables that multiple consumers use like a queue. (PostgreSQL documentation, SELECT) It is the mechanism that keeps multiple sending processes from picking up the same row at the same time.

These are the parts that are often set up wrongly.

• Do not make the external call inside the transaction. If you send while holding the row lock, the database connection and the lock are tied up together when the other side slows down. As in the example above, finish only the claim in a short transaction and do the send outside it
• The claim hold time must be longer than the total timeout. In the example it is 5 minutes. If this time passes before the send finishes, another worker picks up the same row again and sends it in duplicate
• Do not leave a path in which business code sends directly. Once an exception such as "this one is urgent, so send it right away and also put it in the outbox" appears, the results of the two paths diverge
• Clean up completed rows. If SENT rows keep piling up, the table grows. Set a retention period and move or delete them periodically. The partial index in the example above contains only pending rows, so query performance is maintained
• Work that requires ordering needs a separate mechanism. A later item can go out first while an earlier one is waiting to be retried. This is covered in the ordering section below

The DEAD state serves as the storage place for the "requests that ultimately failed" mentioned earlier. The original request, the failure cause, and the attempt count remain in one row, so it can be looked up on the operations screen and reprocessed by setting it back to PENDING.

If you have to call the external system first, write the record first

There are also calls, such as payment approval, that cannot be deferred to the outbox because the result has to be received immediately. In this case, commit a record in a "request planned" state before the call. Then call the external system and update the record according to the result. Even if the save fails or the process dies after the call, the "request planned" record remains, so by finding records that have stayed in this state for a long time and querying the other side for the result, situation ② can be recovered.

Reconcile on a regular basis

However carefully the design is done, data diverges over long-term operation. Assume this and put a reconciliation procedure in place.

• Compare the counts and amount totals of both sides on a daily basis
• When a discrepancy is found, match item by item
• Distinguish the types that can be corrected automatically from the types a person has to check
• Keep a record of the reconciliation results

For a system that involves payments or settlement, reconciliation is not optional. If anything, the absence of reconciliation becomes a bigger cost later. Matching several months of data retroactively is far harder.

For item-by-item matching, load the data the other side provides into a table and join the two sides with a full outer join. Below is an example that matches our payments table (payments) against a table loaded with the other side's settlement data (partner_settlement) by the other side's transaction ID. The table and column names are for illustration, and it is assumed that the other side's status values were converted to our code scheme at load time. :from, :to, and :settle_date are values passed at execution time.

SELECT COALESCE(o.partner_tx_id, p.tx_id) AS tx_id,
       CASE
         WHEN p.tx_id IS NULL         THEN 'ONLY_OURS'
         WHEN o.partner_tx_id IS NULL THEN 'ONLY_PARTNER'
         WHEN o.amount <> p.amount    THEN 'AMOUNT_MISMATCH'
         ELSE 'STATUS_MISMATCH'
       END AS diff_type,
       o.amount AS our_amount,
       p.amount AS partner_amount
FROM (
  SELECT partner_tx_id, amount, status
  FROM payments
  WHERE approved_at >= :from AND approved_at < :to
) o
FULL OUTER JOIN (
  SELECT tx_id, amount, status
  FROM partner_settlement
  WHERE settle_date = :settle_date
) p ON p.tx_id = o.partner_tx_id
WHERE p.tx_id IS NULL
   OR o.partner_tx_id IS NULL
   OR o.amount <> p.amount
   OR o.status <> p.status;

Each type in the result has different causes and actions.

When reconciliation produces a large number of discrepancies, it is more often a mismatch in the comparison basis than actual discrepancies. For example, we cut off at midnight by approval time while the other side goes by capture time or closes at 23:30. The same thing happens when one side is in UTC and the other in KST. That is why the aggregation cutoff has to be written into the specification, and why items in the boundary time window should be matched once more in the next day's reconciliation. Items still pending confirmation are not excluded from the comparison but marked as a separate type, so that items that remain unsettled for days become visible.

Start with a narrow scope for automatic correction. A reasonable division is to align items for which the other side's data is the system of record (whether approved, the other side's transaction ID) to the other side, and to handle items with differing amounts after a person has checked them. For corrected items, keep a record of who changed what, when, and on what basis. The procedure for matching the old and new systems during a system replacement is covered separately in Legacy Data Migration — What to Check Before You Move Anything.

First decide whether ordering matters for the work

With asynchronous messages, delivery order is sometimes not guaranteed. A situation in which the order cancellation message arrives before the order has been created does occur in practice.

The first thing to check is whether this work really depends on ordering. If state is overwritten, ordering matters less, but for increment and decrement operations, ordering changes the result.

If ordering is needed, there are three options.

• Per-key ordering — Send messages with the same order number to the same processing unit to preserve order
• Using a version number — Put a sequence number in the message, and have the receiving side ignore messages with a number lower than the previous one
• Reordering after arrival — Collect messages for a certain time and process them in order. This introduces delay, so it does not suit work that needs to be real-time

The second approach has the lowest implementation burden. However, which value to use as the version has to be agreed with the other side. If the modification time is used, it becomes a problem as soon as the two systems' clocks are out of sync.

Validate rather than trust the other system

It is better not to assume that incoming data arrives as the specification says.

• Whether required fields are present
• Data types and ranges
• Whether code values are in the defined list
• Whether referenced identifiers actually exist on our side
• Sign and upper limit of amounts and quantities

Data that fails validation is quarantined, not processed. Storing it for now and fixing it later spreads bad data across the whole system.

---

4. You must be able to see what happened when something goes wrong

In responding to integration incidents, the step that takes the most time is not recovery but identifying the cause. Sometimes it takes half a day just to determine whether the problem is on our side or the other side.

Make requests traceable end to end

When a single user request passes through several systems, the logs of each system end up disconnected from one another. An identifier that ties them together is needed.

• Generate a trace ID at the point where the request enters
• Include this ID in every log
• Carry it in a header on external calls
• If the other system uses the same convention, the logs of both sides can be viewed together

If the two sides have to define a new convention, W3C Trace Context can be used as the basis. It is a format in which a single traceparent header carries the version, the trace ID (32 hexadecimal characters), the caller's span ID (16 hexadecimal characters), and flags joined with - (traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01), and it has been finalized as a W3C Recommendation. (W3C Trace Context) Using the standard format means the header does not have to be redefined when a tracing tool is introduced.

If the other side will not accept our trace ID, storing the transaction ID issued by the other side can serve instead. When making an inquiry, having this number or not makes a large difference.

Manage the logs of the integration segment separately

If they are mixed in with general application logs, they are hard to find when needed. It is better to record external calls separately.

These are the items to record.

• Call target and purpose
• Request and response times and elapsed time
• Response code and error message
• Retry count
• Trace ID and the other side's transaction ID
• Idempotency key
• Failure category (connection failure, response timeout, error response)

Whether to keep the full request and response requires judgment. It is useful for debugging but may contain personal information or card data. If you keep it, you must define the masking rules first. A situation in which resident registration numbers, card numbers, account numbers, and contact details remain in the logs as they are has to be avoided. It is common to mask only the body and miss the headers. Include headers that carry authentication tokens, API keys, and webhook signatures in the masking targets as well.

Set alert criteria in advance

Logs are meaningless if no one looks at them. Decide in which situations to notify whom.

Thresholds cannot be accurate from the start. Begin on the premise that they will be adjusted during operation, but it is important to decide who adjusts them and how often. If alerts come too often, people start ignoring them, and from then on the alerts might as well not exist. How to set the criteria for operator alerts is covered in detail in The Point Where Notifications Start Getting Ignored — Designing a Delivery System.

Prepare the procedure for an outage on the other side in advance

A considerable share of integration incidents are problems we cannot fix. Sometimes there is nothing we can do other than wait until the other system recovers. Even so, with no preparation at all, the confusion grows during that time.

These are the items worth organizing in advance.

• Decision criteria — How to tell whether it is our problem or the other side's. If the other side provides a status page or a health check endpoint, register it as a place to check
• Contact channel — The other side's contact details and how to file a report. Check whether there is a separate channel for nights and holidays
• Internal sharing — The channel for passing the situation on so that the customer support department knows about it
• Degraded operation — Whether we can operate with the remaining features alone when that integration is blocked, and what has to be turned off to do so
• Post-recovery handling — In what order to reprocess the items backed up during the outage, and whether users need to be informed retroactively

Post-recovery handling in particular is often left out. It is not over just because the other system has come back up. Sending the backed-up requests all at once can bring down a counterpart that has only just recovered. A mechanism to control the reprocessing rate is needed.

Define what to show the user

Apart from the technical recovery, the screen the user sees during an outage is also something to design.

• Tell the user clearly whether it is a situation where retrying will work or one where they have to wait
• Avoid "Unknown error." Show a code or number alongside it that can be referenced when making an inquiry
• For operations that involve money, such as payments, the current state must be made clear. "We cannot tell whether the payment went through" is the worst possible message
• Manage the button state so as not to invite duplicate attempts

---

5. Pre-project checklist

Contract

• Does the interface specification cover endpoints, authentication, fields, code values, errors, and limits in full?
• Are the handling of dates, amounts, and nulls specified with actual example values?
• Does it state whether errors arrive as HTTP status codes or as codes in the body, and whether each error code is retryable?
• Have you received the typical and maximum response times, or measured them in the test environment?
• Is the means of preventing duplicate requests (idempotency key field, format, retention period) in the specification?
• Is there a lookup method for checking the result of a request whose response was not received?
• Are the webhook signature method, redelivery policy, ordering guarantee, and secret rotation procedure written down?
• Are the delivery method of reconciliation data and the aggregation cutoff written down?
• Is the authentication key for the integration free of any tie to a particular staff member's account?
• Have you confirmed where the test environment behaves differently from production?
• Do you know the other system's scheduled maintenance windows and incident notification channel?
• Have the notice period for contract changes and the versioning policy been agreed?
• Is the implementation built to keep working when unknown fields arrive?

Failure handling

• Are the connection, read, and total timeouts all set? (Are you not using the library defaults as they are?)
• Are the timeout values smaller than the time budget of the upstream caller?
• Are response timeouts recorded as pending confirmation rather than as failures, and is there a procedure to settle them?
• Are retryable and non-retryable cases distinguished by error type?
• Does retrying happen at only one layer? (Including the library's default retries)
• Are backoff and jitter applied, and are both Retry-After formats parsed?
• Is an idempotency key defined, and is it kept identical across retries?
• Is receiver-side idempotency implemented with a unique constraint rather than lookup-then-process?
• Are webhook signatures verified against the raw body, and is the timestamp tolerance checked?
• Is duplicate processing of incoming callbacks prevented?
• Does the circuit breaker not count business errors as failures, and is the unit of blocking split by function?
• Are the value to return and the user message when the circuit is open defined?
• Is there a means to keep and reprocess requests that ultimately failed?

Consistency

• Have you chosen among synchronous, asynchronous, and batch the method that fits the nature of the work?
• Is there a recovery path for when our save and the external send diverge?
• Are external calls made outside the database transaction?
• Is the claim hold time for pending sends longer than the total timeout?
• Are the reconciliation procedure and cycle defined?
• Are the aggregation cutoff and time zone for reconciliation the same on both sides?
• Are the targets of automatic correction and the targets a person must check distinguished?
• Is incoming data validated, and are failed items quarantined?

Observability

• Is the trace ID included in every log and external call?
• Are you storing the other side's transaction ID?
• Are log masking rules defined, and are authentication headers included in the targets?
• Are connection failures and response timeouts distinguished in the integration log?
• Are the alert criteria and recipients defined?
• Has the screen shown to users during an outage been designed?

---

Closing

The principles that recur in integration design ultimately come down to one: assume that the other system can slow down, stop, or send values different from what you expect at any time.

Accepting this assumption changes the design. You set timeouts, split retry conditions, add idempotency keys, keep failed requests, and reconcile regularly. None of these is a large task on its own, but the cost of not having them accumulates throughout the operating period.

We have carried out several projects that tie together systems of different natures, such as booking, payments, and room control. If you are preparing a build that includes integration with existing systems, feel free to get in touch through Contact us.

---

Related articles

• What Takes Longer Than Development in a Kiosk Project — A Checklist for Building Unmanned Systems
• Concurrency Design for Booking Systems: Where Double Bookings and Overbooking Come From
• Legacy Data Migration — What to Check Before You Move Anything

References

• RFC 9110, HTTP Semantics, Section 9.2.2 Idempotent Methods — https://www.rfc-editor.org/rfc/rfc9110.html#name-idempotent-methods
• RFC 9110, HTTP Semantics, Section 10.2.3 Retry-After — https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after
• RFC 9110, HTTP Semantics, Section 15.5.9 408 Request Timeout — https://www.rfc-editor.org/rfc/rfc9110.html#name-408-request-timeout
• RFC 9110, HTTP Semantics, Section 15.6.4 503 Service Unavailable — https://www.rfc-editor.org/rfc/rfc9110.html#name-503-service-unavailable
• RFC 6585, Additional HTTP Status Codes, Section 4 429 Too Many Requests — https://www.rfc-editor.org/rfc/rfc6585.html#section-4
• IETF Datatracker, draft-ietf-httpapi-idempotency-key-header (expired Internet-Draft) — https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
• The Idempotency-Key HTTP Header Field, draft 07 text — https://www.ietf.org/archive/id/draft-ietf-httpapi-idempotency-key-header-07.txt
• AWS Architecture Blog, Exponential Backoff And Jitter — https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• AWS Builders' Library, Timeouts, retries, and backoff with jitter — https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter
• undici, Client documentation (timeout options and defaults) — https://github.com/nodejs/undici/blob/main/docs/docs/api/Client.md
• undici, Errors documentation (timeout error codes) — https://github.com/nodejs/undici/blob/main/docs/docs/api/Errors.md
• Node.js documentation, crypto.timingSafeEqual — https://nodejs.org/api/crypto.html#cryptotimingsafeequala-b
• GitHub Docs, Validating webhook deliveries — https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries
• Stripe Docs, Receive Stripe events in your webhook endpoint — https://docs.stripe.com/webhooks
• Martin Fowler, Circuit Breaker — https://martinfowler.com/bliki/CircuitBreaker.html
• microservices.io, Pattern: Transactional outbox — https://microservices.io/patterns/data/transactional-outbox.html
• PostgreSQL 9.5 release notes (ON CONFLICT and SKIP LOCKED added) — https://www.postgresql.org/docs/release/9.5.0/
• PostgreSQL documentation, SELECT (locking clause) — https://www.postgresql.org/docs/current/sql-select.html
• W3C Trace Context — https://www.w3.org/TR/trace-context/