# Choose a queue provider

**Scope:** how to pick the right backend for `@netscript/queue` — the in-memory test adapter, the zero-config Deno KV default, Redis, RabbitMQ (AMQP), and PostgreSQL — and how to either let auto-discovery select one or pin one explicitly with `provider` + `connection`. This is the *decision* recipe; for the enqueue/consume/cron mechanics see [Queue / KV / cron](https://rickylabs.github.io/netscript/data-persistence/how-to/queue-kv-cron/).

A NetScript queue is provider-agnostic by design: the same `createQueue("jobs")` runs on an in-memory adapter in a unit test, on Deno KV on your laptop, and on a real broker under Aspire — without touching the call site. The only thing that changes is which backend the factory resolves. This recipe is about making that choice deliberately.

## Prerequisites

**Before you start**

| Name | Type | Description |
| --- | --- | --- |
| `A NetScript workspace` | `netscript init` | Created per the Quickstart. Import @netscript/queue from any workspace member. |
| `Start Aspire (for real backends)` | `cd aspire && aspire start` | Aspire provisions Postgres, Redis/Garnet, and any AMQP broker BEFORE your service connects. The in-memory and Deno KV adapters need nothing extra; Redis, RabbitMQ, and the Postgres queue all expect Aspire up first. |
| `Deno KV unstable flag` | `--unstable-kv` | The Deno KV backend (the auto-discovery fallback) uses Deno KV. The scaffold's deno.json sets unstable: ['raw-imports', 'kv']; add --unstable-kv to any ad-hoc deno run/check that touches the queue. |

## The five backends at a glance

There are **four** selectable production providers in the `QueueProvider` enum — Deno KV, Redis, RabbitMQ, and PostgreSQL — plus the `MemoryQueueAdapter` from `@netscript/queue/testing` for tests and examples. Pick by the row that matches your deployment, then jump to the matching step.

**Queue backends in @netscript/queue**

| Backend | How you select it | Reach for it when |
| --- | --- | --- |
| `MemoryQueueAdapter` | `new MemoryQueueAdapter()` from `@netscript/queue/testing` (never auto-discovered) | Unit tests and isolated examples — volatile, in-process, zero setup; data is lost on restart. |
| Deno KV `(provider: 'deno-kv')` | Auto-discovery **fallback**, or pin `QueueProvider.DenoKv` | Local dev and single-instance apps with no broker; durable to the Deno KV store. Needs `--unstable-kv`. |
| Redis `(provider: 'redis')` | Auto-discovered when a Redis/Garnet connection is present; or pin `QueueProvider.Redis` | High-throughput production with a Redis/Garnet resource already provisioned by Aspire. |
| RabbitMQ `(provider: 'rabbitmq')` | Auto-discovered **first** when an AMQP broker is present; or pin `QueueProvider.RabbitMQ` | Broker-grade routing and reliability via Fedify's AMQP adapter; the top of the auto-discovery probe. |
| PostgreSQL `(provider: 'postgres')` | **Explicit only** — pass `QueueProvider.Postgres`; never auto-discovered | You already run Postgres and want one fewer moving part, or want the queue to share your transactional store. Row-claim via `FOR UPDATE SKIP LOCKED`. |

## Step 1 — Default: let auto-discovery pick

If you pass no `provider`, the factory probes the Aspire environment and selects the first available backend in a fixed order. Write provider-neutral code and let the environment decide:

```ts
// queue.ts
import { createTypedQueue } from "@netscript/queue";
import { z } from "zod";

const JobSchema = z.object({ id: z.string(), kind: z.string() });

// No provider → auto-discovery. Probe order: RabbitMQ → Redis → Deno KV.
export const jobs = createTypedQueue("jobs", JobSchema);
```

The probe order is **RabbitMQ (AMQP) → Redis → Deno KV**. Deno KV is the always-available fallback when no broker is discoverable, so the same code runs unchanged from laptop to production as you provision real backends under Aspire.

> PostgreSQL is the deliberate exception
>
> Auto-discovery probes
>
> RabbitMQ → Redis → Deno KV only
>
> . It will
>
> never
>
> fall through to PostgreSQL — even when a Postgres connection is present in the environment. The SQL-durable queue is opt-in: you must pass
>
> provider: 'postgres'
>
> (Step 3) or you will quietly land on the Deno KV adapter.

## Step 2 — Pin a provider explicitly

To take auto-discovery out of the loop, pass `provider` from the `QueueProvider` enum. Connection details go under `connection.<provider>`; every URL is optional and falls back to Aspire service discovery when omitted.

```ts
// queue.ts
import { createQueue, QueueProvider } from "@netscript/queue";

const jobs = createQueue("jobs", {
  provider: QueueProvider.Redis,
  connection: {
    redis: {
      // url optional → falls back to getRedisConnectionFromEnv() / Aspire.
      url: Deno.env.get("REDIS_URL"),
      // options are passed through to ioredis.
      options: {},
    },
  },
});
```

```ts
// queue.ts
import { createQueue, QueueProvider } from "@netscript/queue";

const jobs = createQueue("jobs", {
  provider: QueueProvider.RabbitMQ,
  connection: {
    rabbitmq: {
      // url optional → derived from Aspire's rabbitmq service (amqp://).
      url: Deno.env.get("AMQP_URL"),
      // queueName defaults to the queue's logical name.
      queueName: "jobs",
    },
  },
});
```

```ts
// queue.ts
import { createQueue, QueueProvider } from "@netscript/queue";

const jobs = createQueue("jobs", {
  provider: QueueProvider.DenoKv,
  connection: {
    denoKv: {
      // path optional → Aspire discovery or the default shared Deno KV.
      path: Deno.env.get("DENO_KV_URL"),
      verbose: false,
    },
  },
});
```

## Step 3 — Select the PostgreSQL backend

The PostgreSQL provider gives you a SQL-durable queue with row-claim semantics (`FOR UPDATE SKIP LOCKED`), a visibility timeout, ack/nack, and a dead-letter store — so concurrent consumers never double-claim a message. Because it is never auto-discovered, you must name it:

```ts
// queue.ts
import { createQueue, QueueProvider } from "@netscript/queue";

// provider: QueueProvider.Postgres is the ONLY way to select this backend.
const jobs = createQueue("jobs", {
  provider: QueueProvider.Postgres,
  connection: {
    postgres: {
      // url optional → falls back to Aspire's getPostgresUri().
      url: Deno.env.get("DATABASE_URL"),
      // tableName defaults to 'message_queue'.
      tableName: "message_queue",
    },
  },
});

await jobs.enqueue({ id: "job-1", kind: "reindex" });
```

Reach for Postgres when you already run it under Aspire and want the queue and your application data in one transactional store, rather than standing up a dedicated broker.

## Step 4 — Use the in-memory backend in tests

For unit tests and examples, construct the `MemoryQueueAdapter` directly from the testing sub-path. It is volatile and in-process — nothing to provision, and it is never selected by auto-discovery, so production code paths stay untouched:

```ts
// jobs_test.ts
import { MemoryQueueAdapter } from "@netscript/queue/testing";

const queue = new MemoryQueueAdapter<{ id: string }>();

await queue.enqueue({ id: "job-1" });
await queue.listen(async (message) => {
  // assert on `message` here
});
```

## Step 5 — Add concurrency with createParallelQueue

Any provider can be wrapped for concurrent processing with `createParallelQueue`. Pass a `concurrency` greater than 1 and a single listener processes that many messages at once — ideal for I/O-bound work (HTTP calls, DB queries). At `concurrency: 1` (the default) it is identical to `createQueue`.

```ts
// queue.ts
import { createParallelQueue, QueueProvider } from "@netscript/queue";

// 8 messages processed concurrently on one listener, pinned to Redis.
const queue = createParallelQueue("notifications", {
  concurrency: 8,
  provider: QueueProvider.Redis,
});

await queue.listen(async (message) => {
  await deliver(message); // these run in parallel
});
```

For CPU-bound work, prefer Web Workers over queue concurrency — see [Tune the worker runtime](https://rickylabs.github.io/netscript/background-processing/how-to/tune-worker-runtime/).

## Prove it end to end — same call site, two backends

The pluggability claim is only convincing if you can watch one file run unchanged on two backends. Write the producer/consumer against the factory (no `provider`), then run it first on the Deno KV fallback locally and again on Redis under Aspire — the file never changes.

```ts
// jobs.ts — the resulting file: provider-neutral producer + consumer
import { createTypedQueue } from '@netscript/queue';
import { z } from 'zod';

const JobSchema = z.object({ id: z.string(), kind: z.string() });

// No provider → auto-discovery. Probe order: RabbitMQ → Redis → Deno KV.
export const jobs = createTypedQueue('jobs', JobSchema);

if (import.meta.main) {
  await jobs.enqueue({ id: 'job-1', kind: 'reindex' });
  await jobs.listen(async (message) => {
    console.log('processed', message.id, message.kind);
  });
}
```

```bash
# From the workspace root, in order:

# 1. Local run — no Aspire, no broker. Auto-discovery falls through to Deno KV.
#    The scaffold's deno.json already sets unstable: ['raw-imports', 'kv'];
#    add --unstable-kv to any ad-hoc run that touches the queue.
deno run --unstable-kv -A jobs.ts
#    -> processed job-1 reindex   (on the Deno KV fallback)

# 2. Provision the real backend, then run the SAME file untouched.
cd aspire && aspire start && cd ..   # brings up Redis/Garnet
deno run --unstable-kv -A jobs.ts
#    -> processed job-1 reindex   (now auto-discovered onto Redis)
```

The call site — `createTypedQueue('jobs', JobSchema)`, `enqueue`, `listen` — is identical in both runs; only the resolved backend changed. Pin a specific provider with `provider: QueueProvider.*` (Step 2) when you want to take auto-discovery out of the loop.

## In-production pitfalls

> Choosing a provider — read before you ship
>
> - **Local default is Deno KV, not a broker.** If you never start Aspire, auto-discovery silently lands on the Deno KV fallback. Fine for dev — but don't assume broker-grade ordering or fan-out until you've provisioned and confirmed the real backend.
> - **PostgreSQL is opt-in, never auto-selected.** The probe is RabbitMQ → Redis → Deno KV; it will **not** choose Postgres even when a Postgres connection exists. Pin `provider: QueueProvider.Postgres` or you'll quietly run on Deno KV.
> - **Remote Deno KV (KV Connect) switches adapters.** A Deno KV path that is an `http(s)://` URL is treated as KV Connect, which has no native queue ops — the factory transparently swaps in a polling adapter (configurable `pollInterval`, `visibilityTimeout`, `maxRetries` under `connection.denoKv`). A local/SQLite KV path uses native queue ops. Know which one you're on before tuning.
> - **Make handlers idempotent.** A thrown `QueueHandlerError` follows each backend's own ack/retry policy, so the same message can be redelivered. An idempotent handler can't double-charge or double-send on a retry.

> Validate at the boundary
>
> Prefer
>
> createTypedQueue
>
> (Zod-validated at enqueue and dequeue) over the untyped
>
> createQueue
>
> whenever the payload shape matters. An unvalidated queue can carry malformed messages until the consumer fails inside the handler; validation at enqueue time fails fast at the producer, regardless of which provider you chose.

## See also

[KV, queues & cron](https://rickylabs.github.io/netscript/netscript/data-persistence/kv-queues-cron/) — the concept-level tour of queues, KV, and cron, and how Aspire wires the four queue backends.

[Background jobs](https://rickylabs.github.io/netscript/netscript/background-processing/workers/) — when you want a managed worker job (persistence, retries, an HTTP trigger) instead of a raw queue.

[Queue / KV / cron](https://rickylabs.github.io/netscript/netscript/data-persistence/how-to/queue-kv-cron/) — the mechanics recipe: enqueue, consume, and schedule. This page pairs with it: choose the backend here, run it there.

For the full `QueueProvider` enum, the `QueueConnectionOptions` shape, `createParallelQueue` / `ParallelQueueOptions`, the `MessageQueue` interface, and the `QueueError` hierarchy, see

[queue](https://rickylabs.github.io/netscript/netscript/reference/queue/) .

[Use a second database](https://rickylabs.github.io/netscript/netscript/data-persistence/how-to/use-a-second-database/) [Tune the worker runtime](https://rickylabs.github.io/netscript/netscript/background-processing/how-to/tune-worker-runtime/)