Redis AOF and BullMQ: Durability Is a System, Not a Switch

BullMQ stores its queues in Redis. That makes Redis persistence an important part of running BullMQ in production, but it also creates a tempting oversimplification:

Enable AOF and the jobs are safe.

AOF improves durability. It does not guarantee that every job survives every failure, runs exactly once, or produces a correct business result. Those guarantees depend on several layers working together.

What Each Layer Is Responsible For

The basic relationship is straightforward:

Application → BullMQ queue → Redis
                              ↓
                         AOF on disk

Each component has a different job:

• BullMQ manages job states, scheduling, retries, and worker coordination.
• Redis stores the queue data and coordination state.
• AOF records Redis write operations so Redis can reconstruct data after a restart.
• Persistent storage keeps the AOF files across machine or container lifecycles.

Removing any layer changes the failure modes of the system.

AOF Records Writes for Recovery

Redis is primarily an in-memory data store. AOF, or Append Only File, logs write operations as they arrive. On startup, Redis can replay the log to rebuild its dataset.

This is generally more durable than relying only on occasional point-in-time snapshots. It also costs more disk space and I/O, and recovery may take longer because operations need to be replayed.

The important detail is that enabling AOF does not define one fixed durability level. The appendfsync policy does.

Policy	Behavior	Trade-off
always	Sync each batch of writes to disk	Stronger durability, higher overhead
everysec	Sync approximately once per second	Common balance of performance and durability
no	Leave flushing to the operating system	Lower overhead, larger loss window

Redis documents everysec as the default and generally recommended policy. In a failure, it may still lose roughly one second of writes. That might be acceptable for a cache and unacceptable for a critical settlement queue.

The right setting depends on the actual recovery objective, not on a generic “production-ready” checklist.

AOF Inside an Ephemeral Container Is Still Ephemeral

Suppose Redis writes a perfectly valid AOF file to /data, but /data exists only in the container’s writable layer. Restarting the same container may preserve the file. Removing and recreating the container may not.

The deployment therefore needs persistent storage:

Redis container → /data → persistent volume

AOF and a volume solve different problems:

• AOF defines how Redis records in-memory changes on disk.
• The volume defines whether those files outlive the container.

Configuring one without the other leaves a gap.

A Queue Must Not Use Cache-Style Eviction

Redis is often introduced as a cache, where evicting an old key under memory pressure can be reasonable. A job queue is different. Its keys describe work that may still need to run.

BullMQ’s production guide recommends setting Redis maxmemory-policy to:

noeviction

With an eviction policy, memory pressure could remove queue data or metadata. With noeviction, writes fail instead of Redis silently treating queue state as disposable cache content.

That failure still needs monitoring and capacity planning, but it is visible. For this reason, separating queue Redis from a best-effort cache Redis can make both systems easier to operate.

Persistence Does Not Mean Exactly-Once Processing

Consider a payment worker:

1. Worker charges the payment provider.
2. Worker crashes before marking the job complete.
3. BullMQ retries the job.
4. The payment is charged again.

Redis may have persisted every queue transition correctly. The business result is still wrong.

Distributed workers cannot always atomically combine an external side effect with a queue acknowledgement. A worker can fail in the small gap between the two. The practical response is to make important jobs idempotent.

async function settleMarket(job: Job<{ marketId: string }>) {
  const key = `settlement:${job.data.marketId}`;

  await database.transaction(async (tx) => {
    const existing = await tx.settlement.findUnique({ where: { key } });
    if (existing) return;

    await tx.applySettlement(job.data.marketId);
    await tx.settlement.create({ data: { key } });
  });
}

The database should enforce the uniqueness rule as well. A check followed by an insert without a transaction or unique constraint still has a race condition.

Retries then become safe: the second execution observes that the business operation has already completed and returns without applying it again.

Keep Authoritative Business Data Outside the Queue

For a trading or payment system, Redis should not be the final source of truth for balances, orders, trades, or positions merely because BullMQ uses it.

PostgreSQL   authoritative business state
Redis        queue and coordination state
BullMQ       delivery, scheduling, and retries

A job can carry identifiers and instructions, while the worker loads current authoritative data from the database. This avoids treating a queued payload as permanently correct after the surrounding business state has changed.

It also gives recovery a clearer shape. If queue data is lost, a reconciliation process can inspect authoritative records and recreate missing work. If Redis is the only record that the work ever existed, reconciliation has nothing to compare.

A More Complete Production Checklist

For BullMQ jobs that matter, check the system rather than a single Redis flag:

1. Enable an appropriate Redis persistence strategy.
2. Choose appendfsync based on the acceptable loss window.
3. Store persistence files on durable storage.
4. Use maxmemory-policy=noeviction for the queue Redis.
5. Monitor memory, disk, failed jobs, stalled jobs, and Redis connectivity.
6. Make worker side effects idempotent.
7. Enforce invariants with database transactions and constraints.
8. Keep authoritative business state in the primary database.
9. Design reconciliation for missing or uncertain work.
10. Test restart and failure scenarios instead of assuming the configuration works.

Replication and backups can reduce additional risks, but they do not remove the need for these application-level protections.

The Short Version

A useful way to remember the responsibilities is:

PostgreSQL keeps the books. BullMQ assigns the work. Redis stores and coordinates the queue. AOF and durable storage help Redis recover. Idempotency keeps retries correct.

Durability is not a switch. It is the combined behavior of storage, queue semantics, worker design, database constraints, monitoring, and recovery procedures.