A client came to us a few weeks ago with a complaint that sounded minor at first: every so often, an order would go missing at the warehouse. Not a disaster — three or four a week. But the customer calls, the manager finds the order in the CRM just fine, and the warehouse team swears they never saw it. Turned out the integration between the store and the warehouse system was quietly failing for a few hours every night, and the error just sat in logs nobody was reading.
It’s a common story. On paper, an API integration looks simple: here’s the endpoint, here’s the token, here’s a sample request. The real problems don’t show up on day one. They show up a month or two into production, once both systems are living their own lives and one of them starts going down, slowing to a crawl, or returning something nobody expected.
Webhooks or polling — the choice isn’t as obvious as it looks
The first decision in any integration is how one system finds out about changes in the other. There are really only two options: wait for the other system to push an event (a webhook), or ask it yourself on a schedule (polling).
Webhooks look elegant. Something happens, a request lands, you process it, done — practically instant. But there’s a catch: a webhook only works if the receiving side happens to be available at the exact moment it’s sent. Your server goes down for forty seconds during a deploy, and whatever event fired in that window is simply gone. Most systems won’t retry forever — one or two attempts, and they give up.
Polling is the opposite. Ugly, but predictable. A request every five minutes asking “what’s changed since this timestamp” will always catch up on anything it missed, because the next poll picks up whatever the last one didn’t finish. The cost is latency — data isn’t instant — and load on both systems, which gets noticeable once the catalog is large.
In practice, the setup that actually holds up is usually both at once: webhooks for fast reaction, plus a periodic reconciliation poll — hourly or daily — that catches anything the webhook dropped. It feels redundant. It’s really just a few lines of insurance, and it’s exactly what would have caught the missing orders in the story above.
Why the same request should never create two orders
Networks are unreliable by nature. A request goes out, the server processes it, the order gets created in the CRM — and the response never makes it back to the store because of a timeout. The store sees an error and, following normal retry logic, sends the same request again. The CRM gets a second request and, unless something was built to prevent it, creates a second order. The customer gets called twice, the warehouse deducts stock twice, and accounting ends up untangling the mess.
The fix is called idempotency, and it’s simpler than the word suggests. Every request that changes something — creates an order, adjusts inventory, updates a balance — carries a unique idempotency key. Before processing, the CRM checks: has this key already come through? If yes, it just returns the result of the earlier attempt without creating anything new. If not, it processes the request and remembers the key.
The key needs to be generated once, when the order is first put together — not regenerated on every retry attempt. Generate a new key on each retry and you’re back to square one, duplicates and all.
Retrying isn’t just “try again”
A naive retry — fire the same request again the instant it fails — sounds reasonable but can take down a server that’s already struggling. It responds slowly, gets hit with another request, responds even slower, and within minutes the queue of pending requests snowballs.
The pattern that actually works is exponential backoff: retry after one second, then two, then four, then eight, up to some sensible cap, with a bit of random jitter thrown in so a hundred clients don’t all hammer the server at exactly the same second.
And then there’s the question of what happens when retries run out and nothing worked. That’s where a dead-letter queue comes in — instead of quietly dropping the event after the fifth failed attempt, it gets parked in a separate queue for someone to look at. Once a day, someone on the team reviews that queue and decides: retry, skip, or dig in manually. Without it, the event just vanishes, and you find out about the missing order from the customer instead of from the system.
Who’s right when the data disagrees
Say a customer updates their phone number in the CRM during a call with a manager. An hour later, the same customer updates it again in their account on the store’s site. Which value is correct?
Without a clear rule, the answer just depends on whichever sync happened to run last — which is luck, not logic. So before writing any code, it’s worth pinning down explicitly which system is the source of truth for which data. Usually it breaks down something like this: the CRM owns order status and communication history, the store owns the catalog and pricing, the warehouse owns stock levels. Customer details like phone or address are usually the messiest case, and someone needs to decide outright — does the most recent update win, or does one side simply not accept changes from the other at all.
That’s not really an engineering decision, it’s a business one. A developer can build whichever logic gets chosen, but deciding who wins requires understanding the process — whether it’s fine for the website to silently overwrite something a manager just typed in by hand while on the phone with a customer.
How to tell a sync broke without anyone noticing
The worst failure mode isn’t the one that throws a loud error. It’s the one where the integration keeps “working” while doing nothing useful — the endpoint returns 200 OK, the queue keeps processing, and the data hasn’t actually updated in days because the vendor changed their response format, a token expired, or someone changed a URL by mistake.
We run into this constantly during audits: the client is sure the integration is fine because nobody’s seen any errors, while stock levels on the site haven’t moved in three weeks.
- A timestamp for the last successful sync, with an alert if it’s older than expected.
- Monitoring for suspicious silence, not just errors — a sudden drop in processed events is a signal on its own.
- A periodic sanity check comparing counts: how many orders the store logged today versus how many showed up in the CRM over the same window.
None of this is technically difficult. The hard part is having someone think about it before the integration goes live, not after the third customer complaint about a lost order.
Bottom line
An integration that technically works on launch day and one that survives a year of real traffic are two different levels of engineering. The gap isn’t how many endpoints are wired up — it’s what happens when one of them goes down for a minute, two requests land at the same time, or the other side changes its data format without telling anyone. Plan for those cases up front, and the integration just runs for years, quietly, which is really the whole point.
