The spreadsheet had the same bill on two rows, a few seconds apart, and the folder had two copies of the file. Nobody had uploaded anything twice. Duplicate Slack event processing is what it looked like from the outside, and the cause turned out to be less interesting than the fix: Slack had retried a delivery my endpoint was too slow to acknowledge, and everything downstream ran a second time. Storing the event id would have stopped that particular case. It would not have stopped the one that arrived a week later, when someone re-uploaded the same document by hand.
Two different duplicates, one symptom
Both problems produce an identical mess — two rows, two files, and a person trying to work out which one is real. They arrive by completely different routes, and a defence against one is no defence at all against the other.
The first is a redelivery. Slack’s Events API expects an acknowledgement within three seconds and retries when it does not get one, so a single upload can reach the endpoint several times carrying the same event identifier. This is the case every webhook article covers, and it is genuinely the common one.
The second is a resubmission. A person forwards the document again, or drops it into the channel a second time after a failure they were not sure about. That arrives as a brand new event with a new identifier, a new timestamp, possibly a different filename — and identical bytes. An event-id check waves it straight through, because as far as the platform is concerned nothing has been seen before. The document intake system I run needed to survive both, so it has one defence per route, plus a third that catches whatever gets past them.
Layer one: the Slack event key
The first check keys on the delivery rather than the payload. The key is built from four parts — team, channel, message timestamp and file id — which together identify one file arriving in one message, and stay stable across every retry of that delivery.
Message timestamp is the load-bearing piece. Slack’s ts is unique per message and does not change when the platform retries, so the same upload always produces the same key no matter how many times it is delivered. If the key were built from a received-at time instead, every retry would look new, which is the failure mode worth naming: a key derived from anything the receiver generates is not an idempotency key.
This layer is cheap and it handles the majority of real duplicates. It is also the only one of the three that the platform effectively documents for you, which may be why so much advice stops here.
Layer two: a hash of the document itself
The second check keys on content. The downloaded bytes are hashed, and a document whose hash has already completed is skipped with a reply in the thread rather than filed a second time.
Filenames deliberately play no part in this. A phone camera roll produces names that collide constantly, a re-download from an email client produces invoice(1).pdf, and a person renaming a file has not created a new document. Content is the only property that survives all of that. The hash is computed after download and before the model is called, which also means a resubmitted document costs one download instead of a full analysis — the expensive step never runs.
The order matters here in a way that is easy to get backwards. Hashing before the vision model runs turns a duplicate into a cheap no-op; hashing afterwards would still prevent the duplicate row while paying for the extraction anyway. If you are enforcing a rule twice for safety, as I do with the idempotent scripts I use for WordPress edits, it is worth checking that the cheaper guard is the one that runs first.
Layer three: look before you create
The third check does not ask whether this job has run. It asks whether this specific artefact already exists, immediately before creating it. Before uploading to Drive, the pipeline looks up a file by the record id it would have been given; if one is already there, the upload is skipped and the existing file is used.
This is the layer that makes replay safe, and it is the one most often left out. The first two layers answer “have I seen this?”, which is a question about history and therefore about a store that might be stale, missing, or not yet written. The third answers “does this exist?”, which is a question about the world as it is right now. When a job is re-run after a partial failure — and partial failures are the normal case, not the exotic one — history is exactly the thing you cannot trust.
Why I mark completion last, not first
Most webhook guidance, including Hookdeck’s idempotency guide, tells you to record the idempotency key before running side effects. The reasoning is sound: if you send the email and then record the key, a crash in between means the retry sends the email again. Recording first makes the duplicate impossible.
This pipeline does the opposite. Completion is recorded at the very end, after the file is in Drive, the row is in the sheet, and the thread has been replied to — and there is a second such point for documents routed to review instead of filed automatically. Marking first would trade one failure for another I like much less. A crash between the mark and the work would leave a document permanently recorded as handled and permanently absent from the spreadsheet, with nothing to detect it: retries would be refused because the key exists, and no row would ever appear. The document would simply be gone, quietly.
Marking last means a crash produces a retry that does the work again — which is only acceptable because layer three exists. The Drive lookup makes the re-run harmless, so I get the recovery without the duplicate. The standard advice optimises for never doing something twice; this optimises for never silently losing a document. Both are defensible, and which one is right depends entirely on whether your side effects can be made repeatable. Mine could.
The audit log is the idempotency store
There is no separate deduplication table. Every stage of every job already writes a row to an audit tab — received, downloaded, analysed, validated, saved, confirmed, failed — and the idempotency lookups query that same log for a completed record matching an event key or a document hash. The durable store’s write method is empty, because the audit append that already happened is the write.
Keeping one artefact rather than two removes a category of bug I would otherwise have introduced. Two stores drift: the dedupe table says handled, the audit log says failed, and now you are debugging your own bookkeeping instead of the pipeline. It also means the answer to “why was this skipped?” is a row you can read, next to the row it matched, rather than a key in a store nobody can inspect.
| Layer | Keys on | Stops | Misses |
|---|---|---|---|
| Slack event key | team, channel, message ts, file id | Platform retries of one delivery | The same file uploaded again |
| Document hash | The downloaded bytes | Resubmission under any filename | A genuinely edited document |
| Resource lookup | Whether the artefact exists now | Duplicate side effects on replay | Nothing downstream of the check |
What the in-memory cache does not do
Reads hit an in-process cache before they hit the sheet, because a Slack retry can arrive faster than a spreadsheet query returns and the cache absorbs that. It is a latency optimisation and I want to be careful not to describe it as more than that.
On a serverless platform, that cache belongs to one instance. Two retries landing on two concurrent invocations do not share it, so both fall through to the durable lookup, and if they arrive close enough together neither may see the other’s completion recorded yet. The audit sheet is not a transaction; it will not arbitrate a race. In practice a personal intake system processing a handful of documents a day never lands in that window, which is precisely why it would be dishonest to present this as exactly-once processing. It is not. It is three independent reductions in the probability of a duplicate, backed by a fourth property — that doing the work twice is not harmful — and that combination is what actually holds.
The upgrade path is deliberately short. The store is an interface with one durable implementation behind it, so replacing the sheet with Redis, Vercel KV or Postgres is a swap rather than a rewrite. I have not made that swap, because the failure it prevents has not happened at this volume, and I would rather carry a documented limit than an unused dependency.
FAQ
Why does Slack send the same event more than once?
Because it expects an acknowledgement within three seconds and retries when it does not arrive in time. The retry carries the same event identifier, so a key built from the delivery — team, channel, message timestamp, file id — stays stable across all of them.
Is storing the event id enough to prevent duplicates?
It stops platform retries and nothing else. A person re-uploading the same file produces a new event with a new id and identical bytes, which an event-id check passes straight through. That case needs a content hash.
Should I record the idempotency key before or after doing the work?
It depends on which failure you prefer. Recording first guarantees no duplicate but can lose an item permanently if you crash in between. Recording last guarantees no loss but requires every side effect to be safe to repeat. I record last, because a look-up before each side effect makes re-running harmless.
Why hash the file instead of comparing filenames?
Filenames collide constantly and change for reasons that have nothing to do with content — camera roll numbering, a browser appending a suffix, a person renaming something. Identical bytes are the only reliable signal that a document has already been handled.
Does an in-memory cache work for deduplication on serverless?
Only as a latency shortcut. Instances do not share memory, so concurrent invocations each miss the cache and fall through to the durable store. Treat it as an optimisation over a durable check, never as the check itself.
My Thoughts
What changed my thinking here was noticing that the three layers answer different questions rather than the same question three times. Have I seen this delivery. Have I seen these bytes. Does this thing already exist. Only the last one is about the present tense, and it is the one that makes everything else recoverable.
I would still reach for the conventional ordering on a system where the side effects are not repeatable — sending money, sending mail, anything a person receives. The moment you can make each effect safe to repeat, marking last is the better trade, because losing a document silently is a worse outcome than doing tidy work twice. Deciding which of those you are more afraid of turns out to be the actual design decision, and it is not one the guides make for you.