Add a queue and a cron schedule

Your import job runs one file at a time. A migration does not: the day you backfill Dynamics with SAP's historical exports, twenty files land in the hand-off folder at once, and until cutover the two systems only stay aligned if a full re-sync runs every night whether or not a file arrived. This chapter adds the two pieces that make both durable: a queue provider and a worker concurrency so bursts drain in parallel, and a cron schedule that fires on a cadence rather than an event.

1. 1 · Scaffold
2. 2 · Import job
3. 3 · Polyglot transform
4. 4 · Queue & cron
5. 5 · Deploy

What you will build

By the end of this chapter your workers config will name a queue provider and a worker concurrency so bursts of imports drain in parallel instead of one-by-one, and a new scheduled trigger (defineScheduledTrigger) will enqueue a re-sync job on a cron cadence. You will also know the one concurrency naming gotcha to set explicitly so it does not bite you under Aspire.

Before you begin

You need the my-erp/ workspace from Chapter 2 with the import-products job and product-import-trigger working, and aspire start healthy. Confirm the job is registered:

curl '<workers-endpoint>/api/v1/workers/jobs'

Expected: import-products appears in the list. If not, return to Chapter 2 and re-run netscript generate plugins.

Step 1 — Choose a queue provider

A NetScript queue is provider-agnostic: the same job-enqueue path runs on the zero-config Deno KV default on your laptop and on a real broker under Aspire, without touching your job code. The QueueProvider enum has four selectable production backends, and auto-discovery picks one for you when you do not pin one:

For the ERP sync, the right default is to let the environment decide: write provider-neutral config and you get Deno KV locally, and the Aspire cache once it is up — redis by default, or garnet via --cache-backend. The auto-discovery probe order is RabbitMQ → Redis → Deno KV.

Step 2 — Size worker concurrency in config

concurrency on the workers config is the size of the worker pool the runner spins up — each slot is its own V8 isolate (~20–40 MB), so raising it buys parallelism at a memory cost. This is where a burst of imports gets drained in parallel. Set it in the generated worker config:

// config/official-plugins/mod.ts
import { defineWorkers } from '@netscript/plugin-workers-core/config';

export const workers = defineWorkers({
  jobsDir: './workers/jobs',
  tasksDir: './workers/tasks',
  queueProvider: 'auto', // Deno KV locally; the Aspire cache once up (redis default, garnet alt).
  queueName: 'jobs',
  concurrency: 4, // pool size: 4 isolates → ~80–160 MB. Raise for throughput, lower to bound memory.
  enabled: true,
  groups: [],
});

For per-topic control — a hot imports topic at concurrency 10 while a heavy reports topic stays at 1 — use a WorkerGroup with its own scaling: { mode, concurrency }. The full per-topic and runner-mode knobs are in Tune the worker runtime.

Step 3 — Add a cron schedule

Some migration work is time-driven, not file-driven: the nightly full re-sync that keeps Dynamics aligned with SAP until cutover, an hourly cleanup of stale staging files. That is a scheduled trigger — defineScheduledTrigger(handler, spec) from @netscript/plugin-triggers-core/builders. The add-scheduled verb uses the spaced add scheduled shell syntax and can scaffold the job action already wired:

ns-triggers add scheduled daily-resync-schedule \
  --cron="0 6 * * *" \
  --timezone=UTC \
  --job=import-products \
  --description="Runs a full product re-sync every morning." \
  --tags=resync,products,scheduled

The command writes triggers/daily-resync-schedule-trigger.ts and refreshes the trigger registry. Like the file-watch trigger, its generated handler returns an array of effects; here the --job flag supplies the enqueue action without hand-authoring the definition.

// triggers/daily-resync-schedule-trigger.ts (generated excerpt)
import { defineScheduledTrigger, enqueueJob } from '@netscript/plugin-triggers-core/builders';
import type { JobDefinition } from '@netscript/plugin-workers-core';

const importProductsJob = {
  id: 'import-products' as JobDefinition<'import-products'>['id'],
  name: 'Import Products',
  topic: 'default',
  entrypoint: './workers/jobs/import-products.ts',
} satisfies JobDefinition<'import-products'>;

export const dailyResyncSchedule = defineScheduledTrigger(
  (event) => Promise.resolve([enqueueJob(importProductsJob, { payload: event.payload })]),
  {
    id: 'daily-resync-schedule',
    cron: '0 6 * * *', // every day at 06:00
    timezone: 'UTC',
    description: 'Runs a full product re-sync every morning.',
    tags: ['resync', 'products', 'scheduled'],
  },
);

export default dailyResyncSchedule;

Step 4 — Load the new trigger

The scaffold command already refreshed the registry. Restart aspire start (or let it hot-reload) so the triggers processor loads daily-resync-schedule.

Verify your progress

First confirm the config type-checks with the new concurrency and provider settings:

deno task check

Expected: a clean check. Then confirm the cron schedule registered — for a fast feedback loop you can temporarily set its cron to */2 * * * * (every two minutes), regenerate, restart Aspire, wait, and read the executions feed:

ns-workers executions --limit=10 --json

Expected: an import-products execution appears on the cron cadence with no file dropped — the schedule, not a file event, enqueued it. Set the cron back to 0 6 * * * when you are done testing.

• [ ] config/official-plugins/mod.ts sets concurrency and queueProvider.
• [ ] deno task check is clean.
• [ ] daily-resync-schedule is registered (after netscript generate plugins + Aspire restart).
• [ ] A scheduled import-products execution appears on the cron cadence with no file dropped.
• [ ] You set WORKERS_CONCURRENCY explicitly (or rely on config concurrency) rather than the ignored Aspire WORKER_CONCURRENCY.

What you built

A workers config that names a queue provider and a worker concurrency so a SAP backfill burst drains in parallel, and a defineScheduledTrigger cron that enqueues the nightly re-sync — plus the knowledge to set concurrency where it actually takes effect. The sync now absorbs bursts and runs its recurring work unattended. The last chapter runs the whole thing under Aspire.