Skip to main content

The Myth of Exactly-Once Delivery

Sagaweaw Team
Written bySagaweaw TeamPlatform Engineering

The promise of "exactly-once delivery" is the Holy Grail of distributed systems. Messaging tools frequently claim to support this behavior, but in practice, when we consider the laws of network physics and possible hardware failures, exactly-once is a mathematical illusion at the infrastructure level.

Why Exactly-Once Fails in Practice​

In distributed systems, there are two guarantees that are easy to achieve:

  1. At-most-once: The message is sent and, if the network fails, it is lost.
  2. At-least-once: The message is sent with an acknowledgment (ACK). If the acknowledgment is lost due to a network timeout, the producer resends the message. This guarantees delivery, but can generate duplicate messages.

To achieve exactly-once, the system needs to know perfectly whether the other end received and processed the message, even when the connection dies at the exact millisecond of the ACK. This requires infinite coordination that contradicts the CAP Theorem.

The Solution: At-Least-Once + Idempotency​

The only safe way to guarantee exactly-once behavior in critical business flows (such as payment processing or refunds) is not to require exactly-once from the network infrastructure, but instead implement Idempotency at the application layer.

[!TIP] Idempotency is the property of an operation that allows it to be applied multiple times without changing the result beyond the first application.

How Sagaweaw Solves This​

Sagaweaw was designed assuming that failures and duplications will happen. Instead of fighting the network, it focuses on Idempotency and predictable Retries:

  • Idempotency Keys: Each Saga and Step receives an idempotency key (IdempotencyKey). If a step is re-executed due to a timeout, Sagaweaw ensures that the execution will not generate duplicate side effects if the target system supports key validation.
  • Database Transactions: Sagaweaw's persistence in PostgreSQL ensures that the state change of your Saga is atomic. The framework checks the database before re-executing any Step business logic, absorbing the duplication at the orchestrator layer.

Example: Balance Compensation​

If your Saga fails after charging the customer (a PIVOT step) and you need to do the rollback (refund), it doesn't matter if the rollback message drops from the network or is sent three times by a RetryPolicy.exponential(). Your unblockBalance step will use the Saga key to ensure the customer is only refunded once, regardless of obscure network corners.

By embracing chaos and providing the tools for structured idempotency, Sagaweaw removes the burden of ensuring perfect infrastructure, delegating consistency to the transactional model.

Join the discussion!

Architecture is about trade-offs. What do you think about the decisions made in "The Myth of Exactly-Once Delivery"? Share your scenarios, ask questions, and debate with other engineers in the Sagaweaw community.

Comment on GitHub Discussions