# Graceful shutdown

Drain in-flight requests and jobs, run teardown hooks, and close DB/queue connections when your app receives `SIGINT`/`SIGTERM` — so a deploy or `Ctrl+C` never drops work mid-flight.

## Prerequisites

**What you need**

| Name | Type | Description |
| --- | --- | --- |
| `@netscript/service` | `package` | Provides createService().onShutdown().serve() — signal handling and request draining live here. |
| `@netscript/plugin-workers-core/shutdown` | `subpath export` | ShutdownManager for draining worker/scheduler resources; the worker runtime also exposes a runtime.shutdown handle. |
| `@netscript/queue` | `package (if you consume a queue)` | MessageQueue.listen(handler, { signal }) + stop() let an AbortSignal stop consumption gracefully. |
| `A long-running entrypoint` | `main.ts` | A service or worker process you start with deno run / aspire start that needs to stop cleanly. |

> Signals are wired for you (services)
>
> You do
>
> not
>
> call
>
> Deno.addSignalListener
>
> yourself for a service.
>
> serve()
>
> installs
>
> SIGINT
>
> /
>
> SIGTERM
>
> (or
>
> SIGINT
>
> /
>
> SIGBREAK
>
> on Windows) automatically and drains the HTTP listener before the process exits. You only register
>
> what to tear down
>
> via
>
> .onShutdown()
>
> .

## Step 1 — Drain a service with `.onShutdown()`

`serve()` already drains in-flight HTTP requests. Add `.onShutdown(hook)` to close the things the framework does not own — your database client, an external connection, a flush buffer. Each `ShutdownHook` receives a `ShutdownContext` (`reason`, optional `signal`) and runs during the drain, in LIFO order (last registered, first to run).

```ts
// services/users/src/main.ts
import { createService } from '@netscript/service';
import { router } from './router.ts';
import { db } from '@database';

const running = await createService(router, { name: 'users', version: '1.0.0' })
  .withRPC()
  .withHealth()
  .onShutdown(async ({ reason, signal }) => {
    // reason: 'signal' | 'manual' | 'startup-failure'
    audit.record({ event: 'shutdown', reason, signal });
    await db.$disconnect();
  })
  .serve({
    port: 3001,
    drainTimeoutMs: 10_000, // wait up to 10s for in-flight work
    handleSignals: true,    // SIGINT/SIGTERM (SIGBREAK on Windows) — the default
  });

// In tests or a supervisor, trigger the same drain manually:
await running.stop();
```

The drain is bounded by `drainTimeoutMs` (default `30_000`). When it elapses the service stops anyway and the returned `ShutdownReport` records `timedOut: true` plus a per-hook `ShutdownHookOutcome`. `running.stop()` runs the identical drain with reason `'manual'`; both paths are idempotent, so a double `Ctrl+C` will not double-run hooks.

**Shutdown contract (@netscript/service)**

| Name | Type | Description |
| --- | --- | --- |
| `.onShutdown(hook)` | `ServiceBuilder method` | Registers an async teardown ShutdownHook. Hooks run LIFO during the drain. |
| `ShutdownHook` | `(context: ShutdownContext) => Promise \| void` | Your teardown callback. Throwing is captured, not fatal — it lands in the report as a failed hook. |
| `ShutdownContext` | `{ reason: ShutdownReason; signal?: Deno.Signal }` | signal is set only when reason is 'signal'. |
| `ShutdownReason` | `'signal' \| 'manual' \| 'startup-failure'` | An OS signal, a manual stop() call, or a failed startup hook. |
| `ShutdownReport` | `{ reason; timedOut: boolean; hooks: readonly ShutdownHookOutcome[] }` | Returned by stop(); timedOut is true when the drain budget elapsed first. |

## Step 2 — Configure the drain budget via `serve()`

The drain budget and signal behavior are `ServeOptions` keys. Pass an external `AbortSignal` to stop the listener from anywhere — a parent controller, a test harness, or a higher-level orchestrator — without an OS signal.

**ServeOptions — shutdown-relevant keys**

| Name | Type | Description |
| --- | --- | --- |
| `drainTimeoutMs` | `number?` | Max time to wait for in-flight requests and shutdown hooks before forcing exit. Defaults to 30_000. |
| `handleSignals` | `boolean?` | Install SIGINT/SIGTERM (or SIGBREAK) handlers. Defaults to true — set false only if a parent process owns signals. |
| `signal` | `AbortSignal?` | External signal that stops the listener (and runs the drain) when aborted. Reason is reported as 'manual'. |
| `port` | `number?` | Preferred listener port; 0 for an ephemeral port. |

```ts
// supervisor.ts — stop a service from a parent AbortController
const controller = new AbortController();

const running = await createService(router, { name: 'users' })
  .withRPC()
  .serve({ port: 3001, signal: controller.signal, drainTimeoutMs: 15_000 });

// Anywhere in the parent: triggers the same bounded drain.
controller.abort();
```

## Step 3 — Drain a worker runtime

The worker runtime carries its own `ShutdownManager` and exposes it as `runtime.shutdown`. The runtime pre-registers its worker (priority `20`) and, when present, its scheduler (priority `30`); higher-priority resources stop first. Register your own long-lived resources — a queue consumer, a connection pool — and call `runtime.shutdown.shutdown(reason)` to stop them all under one bounded budget.

```ts
// workers/src/main.ts — drain workers + a queue consumer on signal
import { ShutdownManager } from '@netscript/plugin-workers-core/shutdown';

// runtime.shutdown is a RuntimeShutdownManager: { register, shutdown }
runtime.shutdown.register({
  id: 'queue-consumer',
  priority: 10, // stops after worker (20) and scheduler (30)
  stop: async () => {
    await queue.stop(); // let in-flight messages ack before exiting
  },
});

// Workers do NOT auto-install signal handlers — wire them yourself:
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
  Deno.addSignalListener(signal, () => {
    void runtime.shutdown.shutdown(signal);
  });
}
```

> Workers need their own signal wiring
>
> Unlike a service's
>
> serve()
>
> , the worker runtime does
>
> not
>
> register OS signal handlers for you. Add
>
> Deno.addSignalListener
>
> in your worker entrypoint and route the signal into
>
> runtime.shutdown.shutdown()
>
> . On Windows, listen for
>
> SIGBREAK
>
> instead of
>
> SIGTERM
>
> (the service listener already does this internally for its own signals).

You can also build a standalone `ShutdownManager` directly. Its `createAbortController()` returns an `AbortController` that aborts the moment shutdown starts — feed that signal into `queue.listen(handler, { signal })` so consumption stops cleanly, and `register()` each resource you want stopped (higher `priority` stops first).

**ShutdownManager (@netscript/plugin-workers-core/shutdown)**

| Name | Type | Description |
| --- | --- | --- |
| `register(resource)` | `(ShutdownResource) => void` | Adds a resource: { id, priority?, stop(reason?) }. Higher priority stops first; default 0. |
| `shutdown(reason?, options?)` | `=> Promise` | Stops all registered resources concurrently under timeoutMs (default 30_000); idempotent — returns the same report on repeat calls. |
| `createAbortController()` | `=> AbortController` | An AbortController that aborts when shutdown begins. Pass its .signal to queue.listen({ signal }). |
| `waitForShutdown()` | `=> Promise` | Resolves once shutdown has started — await it to gate your own cleanup. |
| `state` | `'running' \| 'shutting-down' \| 'stopped'` | Current lifecycle state. |

## Step 4 — Stop a queue consumer cleanly

A `MessageQueue` consumer is a long-running `listen()` loop. Two ways to stop it without dropping a message mid-process: pass an `AbortSignal` to `listen()`, or call `queue.stop()` (which lets in-flight messages finish before shutting down). Wiring the `AbortSignal` from a `ShutdownManager` ties consumption directly to your drain.

```ts
// workers/src/consumer.ts — stop consuming when shutdown begins
import { ShutdownManager } from '@netscript/plugin-workers-core/shutdown';

const manager = new ShutdownManager({ timeoutMs: 15_000 });
const { signal } = manager.createAbortController();

// listen() returns when the signal aborts; in-flight messages ack/nack first.
await queue.listen(
  async (message, ctx) => {
    await handle(message);
    await ctx.ack();
  },
  { signal, concurrency: 5 },
);
```

```ts
// workers/src/consumer.ts — drain via stop()
// stop() halts the listen loop and lets in-flight messages complete.
await queue.stop();
```

## In-production pitfalls

> Footguns before you ship
>
> - **Set `drainTimeoutMs` below your platform's kill grace.** The drain defaults to `30_000`. If your orchestrator sends `SIGKILL` sooner (Kubernetes `terminationGracePeriodSeconds` defaults to 30s), in-flight work is cut off — pick a budget comfortably under the grace period.
> - **Workers do not auto-handle signals.** Only a service's `serve()` installs `SIGINT`/`SIGTERM`. A standalone worker entrypoint that omits `Deno.addSignalListener` will be killed ungracefully — register the listener and route it into `runtime.shutdown.shutdown()`.
> - **Use `SIGBREAK`, not `SIGTERM`, on Windows.** Deno does not deliver `SIGTERM` on Windows; the service listener already uses `SIGBREAK` internally, but your own worker signal wiring must too.
> - **A throwing hook does not abort the drain.** A `ShutdownHook` that rejects is captured as a failed `ShutdownHookOutcome` in the report, not re-thrown — inspect the returned `ShutdownReport` (or the logged warning) to catch teardown failures.
> - **Close the DB in a hook, not at module scope.** Call `db.$disconnect()` inside `.onShutdown()` so it runs *after* in-flight requests drain — disconnecting earlier breaks requests still being served.

> No single app-wide shutdown orchestrator yet
>
> Planned The framework drains **each runtime independently**: `serve()` drains a service, `ShutdownManager` drains workers, `queue.stop()` drains a consumer. There is **no** single top-level `host.shutdown()` that orchestrates a service + its workers + its queue + its DB together under one budget. Until that lands, *you* compose them: register every long-lived resource with the worker runtime's `runtime.shutdown`, and put service-owned teardown in `.onShutdown()`. Co-locating a service and workers in one process means wiring both drains in the same entrypoint by hand.

## See also

[service](https://rickylabs.github.io/netscript/netscript/reference/service/)

[Background jobs](https://rickylabs.github.io/netscript/netscript/background-processing/workers/)

[Look up — @netscript/service  The full shutdown contract: onShutdown, ServeOptions (drainTimeoutMs, handleSignals, signal), ShutdownReport, and the running.stop() drain.](https://rickylabs.github.io/netscript/netscript/reference/service/)

[Look up — @netscript/workers  ShutdownManager, the runtime.shutdown handle, and the worker/scheduler resources it pre-registers.](https://rickylabs.github.io/netscript/netscript/reference/workers/)

[Do — Tune the worker runtime  Recipe: concurrency, retry, and the runtime knobs that decide how much in-flight work a drain has to wait on.](https://rickylabs.github.io/netscript/netscript/background-processing/how-to/tune-worker-runtime/)

[Understand — The durability model  Why queue messages and saga steps survive a restart — the guarantees a clean drain protects.](https://rickylabs.github.io/netscript/netscript/explanation/durability-model/)

[Run a polyglot task](https://rickylabs.github.io/netscript/netscript/background-processing/how-to/run-a-polyglot-task/) [Deploy locally with Aspire](https://rickylabs.github.io/netscript/netscript/orchestration-runtime/how-to/deploy-local-aspire/)