Dedup & Ordering

Webhooks and event-driven systems often send duplicates or deliver events out of order. fyrn provides built-in deduplication and ordering at the source level.

Deduplication

Dedup prevents the same message from being processed twice within a time window.

Configuration

Add a dedup block under source::

source:
  connector: shopify-production
  trigger: webhook
  event: orders/create
  dedup:
    key: source.id
    window: 24h

Field	Type	Required	Description
`key`	string or string[]	Yes	Field path(s) for the dedup key
`window`	duration	Yes	Time window for dedup check

Single key

dedup:
  key: source.id
  window: 24h

Messages with the same source.id within 24 hours are dropped after the first one.

Composite key

Use an array for composite dedup keys when a single field isn’t unique enough:

dedup:
  key: [source.entity_id, source.event_type]
  window: 1h

How it works

Dedup runs as the first operation in the processing pipeline — before ordering, transforms, or delivery. This means duplicate messages are dropped before they consume any downstream resources.

Under the hood, fyrn hashes the dedup key value(s) using SHA-256 and stores the hash in Redis with a TTL matching your window duration. When a message arrives, the runtime checks for an existing key: if found, the message is dropped and logged as a duplicate; if not, the key is written and processing continues. For composite keys, field values are concatenated before hashing.

Performance is bounded by the Redis lookup — typically sub-millisecond latency per message. The TTL-based expiration means no manual cleanup is required, and memory usage scales with your unique message rate multiplied by the window duration, not total message volume.

Duration format

Pattern: <number><unit> — 30s, 5m, 1h, 24h, 7d

Message ordering

Ordering ensures messages for the same entity are processed in sequence.

Configuration

source:
  connector: shopify-production
  trigger: webhook
  ordering:
    key: source.entity_id
    mode: strict

Field	Type	Required	Default	Description
`key`	string or string[]	Yes	—	Field path(s) for ordering key
`mode`	string	No	`strict`	`strict` or `best-effort`
`lock_ttl`	duration	No	`60s`	Lock timeout

Strict mode

Guarantees in-order delivery. Messages for the same key are queued and processed one at a time.

ordering:
  key: source.order_id
  mode: strict

Use when: order of operations matters (e.g., create → update → delete for the same entity).

Best-effort mode

Attempts ordering but does not block on lock contention. If the lock is held, the message proceeds anyway.

ordering:
  key: source.sku
  mode: best-effort
  lock_ttl: 30s

Use when: ordering is preferred but not critical, and you want higher throughput.

Performance tradeoffs

Strict mode acquires a distributed lock per ordering key and holds it until processing completes. If two messages arrive for the same order_id, the second waits until the first finishes. This guarantees order but limits throughput to one concurrent message per key. Choose strict when operations are non-commutative — for example, a create followed by an update must execute in that sequence.

Best-effort mode attempts to acquire the lock but proceeds immediately if it is already held. This means two messages for the same key can process concurrently, with occasional out-of-order delivery. Use this for high-throughput scenarios where ordering is preferred but not critical, such as IoT telemetry or analytics events.

Set lock_ttl to slightly longer than your expected step execution time. If a step typically takes 2 seconds, a lock_ttl of 10s gives enough headroom for retries without holding the lock indefinitely. Too short and the lock expires mid-processing, allowing a second message through; too long and a crashed worker holds the lock, delaying subsequent messages until TTL expiry.

Mode	Throughput	Ordering guarantee	Lock contention
`strict`	Lower	Guaranteed	Blocks on same key
`best-effort`	Higher	Best-effort	No blocking

Combining dedup and ordering

When both are configured, dedup runs first, then the ordering lock is acquired:

source:
  connector: shopify-production
  trigger: webhook
  event: orders/create
  dedup:
    key: source.id
    window: 24h
  ordering:
    key: source.customer_id
    mode: strict

Pipeline: receive → dedup (filter) → ordering (lock) → transform → deliver

Common patterns

Idempotent webhook processing

source:
  connector: stripe
  trigger: webhook
  dedup:
    key: source.id
    window: 48h

Ordered entity updates

source:
  connector: crm
  trigger: webhook
  ordering:
    key: source.contact_id
    mode: strict

High-throughput with soft ordering

source:
  connector: iot-gateway
  trigger: webhook
  dedup:
    key: source.message_id
    window: 1h
  ordering:
    key: source.device_id
    mode: best-effort
    lock_ttl: 10s

What’s next

Aggregation — Buffer and aggregate messages over time or count windows
DSL Reference — Full dedup and ordering specification