Backend · APIs · Reliability

Building Reliable Pipelines Around Third-Party APIs

Third-party APIs are part of your system whether you control them or not. They can slow down, rate-limit, return stale data or fail completely. A reliable integration is not just an HTTP request that succeeds most of the time; it is a boundary that absorbs external failure without letting it spread through the rest of the application.

September 29, 2026

Treat the provider as an unreliable dependency

The first useful assumption is that every external API will eventually fail in a way your happy-path code did not expect. That does not mean the provider is bad. Networks fail, quotas are exhausted, maintenance happens and response times change.

The integration boundary should therefore define what happens when the provider is slow, unavailable, partially wrong or temporarily rate-limited. Reliability starts when those cases are explicit instead of being handled as generic exceptions.

Separate provider adapters from application behavior

Application code should not need to know the response shape, authentication quirks or error vocabulary of every upstream service. I prefer a provider adapter that translates each external API into an internal contract.

That makes it possible to change providers, combine multiple sources or disable one temporarily without rewriting the rest of the application. Forecast evaluation, alerts and persistence can consume one normalized model while provider-specific behavior stays at the boundary.

interface WeatherProvider {
  getForecast(input: ForecastRequest): Promise<NormalizedForecast>;
}

Use timeouts before retries

A request that never fails but takes too long can be more damaging than a fast error. Every external call should have a bounded amount of time in which it is allowed to consume a worker or scheduled-job slot.

Retries only make sense after the first attempt has a clear timeout. Otherwise multiple slow calls can overlap and amplify the original problem.

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 8_000);

try {
  return await fetch(url, { signal: controller.signal });
} finally {
  clearTimeout(timeout);
}

Retry only failures that can improve with time

Not every error deserves a retry. A temporary network failure or a 5xx response may succeed on a later attempt. Invalid credentials, malformed input or a hard quota limit usually will not.

Retries should be bounded and delayed, preferably with exponential backoff and jitter so multiple instances do not retry at exactly the same time. The goal is recovery, not persistence at any cost.

delay = min(base * 2 ** attempt + jitter, maxDelay)

Circuit breakers stop repeated failure

When a provider is already failing or rate-limiting, repeatedly calling it on every scheduled run creates more load and wastes capacity. A circuit breaker records that recent failures have crossed a threshold and temporarily stops new attempts.

The open period should reflect the failure. A short cooldown can be enough for transient 429 responses, while an explicit daily quota error may justify staying open much longer. After the cooldown, a limited probe can decide whether normal traffic should resume.

CLOSED -> failures exceed threshold -> OPEN
OPEN -> cooldown expires -> HALF_OPEN
HALF_OPEN -> success -> CLOSED
HALF_OPEN -> failure -> OPEN

Freshness can be more valuable than a new request

A reliable pipeline should know how old its existing data is before calling an upstream service again. If a forecast is still fresh enough for the product behavior, reusing it may be better than making another external request.

Freshness is a business decision, not only a cache TTL. Different consumers may tolerate different ages: an interactive forecast, an alert evaluation and a daily summary can each have their own acceptable window.

Reuse one provider response across multiple consumers

Duplicate upstream calls often appear because different parts of the application independently ask for the same data. A scheduled forecast refresh may fetch data, then an alert process or comparison job fetches the same provider again minutes later.

Where possible, fetch once, normalize once and persist or share the result so forecasts, comparisons, alerts and summaries can reuse it. This reduces quota usage and also makes those features reason about the same snapshot.

Make persistence idempotent

Retries and duplicate invocations are unavoidable at some layer, so writing the same result twice should be safe. Idempotency can be enforced with unique keys, upserts, deduplication tokens or by comparing the new normalized snapshot with the current one before writing.

In data pipelines I prefer not to create a new event or snapshot when nothing meaningful changed. That reduces database growth and makes downstream activity easier to interpret.

if (sameMeaningfulData(previous, incoming)) {
  return previous;
}

return persist(incoming);

Fallbacks need explicit semantics

Multiple providers can improve availability, but only if the system knows what a fallback means. Provider B may have a different resolution, update cadence or field set from Provider A.

Normalize what can be normalized, preserve source metadata and avoid pretending that all providers are interchangeable. The application should know whether it is serving primary data, fallback data or the last known good value.

Scheduled pipelines need coordination too

Retries, circuit breakers and freshness checks do not prevent two application instances from starting the same scheduled job simultaneously. That is a separate coordination problem.

For Vienor I use PostgreSQL advisory locks to make scheduled work non-blockingly single-owner across instances. The lock protects concurrent execution, while the pipeline-level controls handle external failures and duplicate effects.

Observe the boundary, not just the final error

A single error count does not explain why an integration is unhealthy. Useful signals include request latency, timeout rate, response status, retry count, circuit state, provider freshness, skipped duplicate work and how often fallback data is used.

Those signals make it possible to distinguish an application regression from an upstream incident and to tune retry or freshness policies based on real production behavior instead of guesses.

The design principle

An external API should be treated as a source of uncertain data and uncertain availability, not as a local function call. The integration layer owns that uncertainty.

Bound the calls, retry selectively, stop calling when failure is already known, reuse fresh results, make writes idempotent and keep provider-specific details out of the rest of the system. That turns an API integration from a fragile dependency into a controlled boundary.