# Add OpenTelemetry

**Scope:** turn on, extend, and read distributed traces in an existing NetScript workspace. OpenTelemetry is not a bolt-on here — `@netscript/telemetry` wraps `@opentelemetry/api` and is wired into the services, the worker dispatcher, the scheduler, and the subprocess task runtime the scaffold generates. Aspire stands up the OTLP collector and the trace UI for you. This recipe shows where the instrumentation already lives, how to add your own spans and structured logs, how `traceparent` propagates across the oRPC/HTTP boundary and into job subprocesses, and how to watch it all land in the Aspire dashboard at [https://localhost:18888](https://localhost:18888).

This is a task recipe, not a deep-dive. For the mental model behind spans, structured logs, and the per-capability health endpoints, read [Observability](https://rickylabs.github.io/netscript/explanation/observability/). For the generated API surface, follow the [`telemetry`](https://rickylabs.github.io/netscript/reference/telemetry/) and [`logger`](https://rickylabs.github.io/netscript/reference/logger/) reference pages, and the [Telemetry capability](https://rickylabs.github.io/netscript/capabilities/telemetry/) hub for the Learn / Do / Reference triplet.

> What works today, and the one gap
>
> Worker tracing is built in and real.
>
> Job dispatch, job execution, the scheduler, and the task subprocess all emit real OpenTelemetry spans automatically — they appear in the Aspire dashboard with no code from you.
>
> task.execute
>
> spans are real too. The
>
> only
>
> gap is the scaffold's
>
> createJobTools(ctx)
>
> handler helpers:
>
> trace.addEvent
>
> ,
>
> trace.withChildSpan
>
> , and
>
> trace.recordProgress
>
> are currently
>
> no-op stubs
>
> in the generated
>
> job-tools.ts
>
> (a tracked debt with a fix planned). To emit custom spans from inside a handler today, call the
>
> @netscript/telemetry
>
> instrumentation helpers directly — shown in Step 3. Structured logging via
>
> log.*
>
> is real now.

## Prerequisites

**What this recipe assumes**

| Name | Type | Description |
| --- | --- | --- |
| `netscript workspace` | `netscript init` | An existing workspace. If you have none, scaffold one first — see the tutorials. |
| `Start Aspire` | `cd aspire && aspire start` | The AppHost provisions Postgres, Redis, the OTLP collector, and the dashboard. Start it BEFORE you expect traces. Dashboard at https://localhost:18888. |
| `@netscript/telemetry` | `OTel facade` | Wraps @opentelemetry/api and ships the worker/scheduler/queue/SSE instrumentation. Already wired into the generated handlers — no install step. |
| `A service or plugin to trace` | `services/users or plugins/workers` | The users service (:3001) and the workers/sagas/triggers/auth plugins all emit health + trace data once running. |

> Aspire first, always
>
> The OTLP endpoint and the trace UI are
>
> Aspire resources
>
> . Aspire is step 2 of the workflow:
>
> cd aspire && aspire start
>
> brings up Postgres, Redis, the collector, and the dashboard
>
> before
>
> any
>
> netscript db
>
> command or service. If
>
> aspire start
>
> is not up, there is nowhere for spans to go and nothing to view. Start orchestration from the
>
> aspire/
>
> folder before you run a service, trigger a job, or open the dashboard.

## How the telemetry is already wired

Three layers ship instrumented out of the box. Knowing which is which tells you where you get spans for free and where you add your own.

**Where instrumentation lives**

| Name | Type | Description |
| --- | --- | --- |
| `Service layer (real spans)` | `@netscript/service` | RPC trace context (header extraction into ctx.traceHeaders) is ON by default — traceContext defaults to true when withRPC() is called without arguments. The OTel TracingPlugin that creates real spans is also always active; it is independent of the traceContext option. |
| `Worker runtime (real spans)` | `job dispatcher + scheduler` | Job dispatch and execution, scheduler runs, and the task subprocess emit real OTel spans automatically via @netscript/telemetry — traceJobExecution, scheduler spans, task.execute. Traces show up in Aspire with no handler code. |
| `Scaffold job tools (stub spans today)` | `createJobTools(ctx)` | log / progress / trace handed to defineJobHandler bodies. log.* is REAL; trace.addEvent / withChildSpan / recordProgress are no-op stubs in the scaffold (tracked debt, fix planned). For custom handler spans, call @netscript/telemetry helpers directly. |
| `OTLP export + UI` | `http://localhost:4318 → :18888` | The Aspire profile points OTLP at http://localhost:4318; the dashboard renders the collected traces and correlated structured logs at :18888. |

> Two senses of "worker tracing"
>
> The framework/dispatcher layer emits real spans —
>
> traceJobExecution
>
> , scheduler spans, and subprocess
>
> traceparent
>
> propagation are all live, so you get job traces in Aspire automatically. The
>
> scaffold-facing
>
> createJobTools(ctx)
>
> helpers your handler calls are the only stubs. Never read "the job-tools helpers are stubs" as "worker tracing is a no-op" — the runtime around your handler is fully instrumented.

## Step 1 — Bring up Aspire and confirm the collector

From the workspace root, start orchestration. The AppHost registers the OTLP collector and the dashboard, then boots Postgres, Redis, and every service/plugin resource.

```bash
cd aspire
aspire start
# dashboard: https://localhost:18888  (login token printed in the console)
```

Open [https://localhost:18888](https://localhost:18888), authenticate with the token Aspire printed, and select the **Traces** tab. With nothing exercised yet it is empty — that is expected. Leave it open; it updates live.

> Resources you should see
>
> The Aspire resource graph lists
>
> postgres
>
> ,
>
> redis
>
> ,
>
> workers-api
>
> (
>
> :8091
>
> ),
>
> workers
>
> ,
>
> sagas-api
>
> (
>
> :8092
>
> ),
>
> sagas
>
> ,
>
> triggers-api
>
> (
>
> :8093
>
> ),
>
> triggers
>
> , the
>
> auth-api
>
> (
>
> :8094
>
> ), and the durable-streams service (
>
> :4437
>
> ). Each resource exports telemetry to the OTLP endpoint at
>
> http://localhost:4318
>
> ; the dashboard reads from there.

## Step 2 — Generate a trace without writing any code

Both service-layer and worker-runtime tracing are real and need no code from you. Exercise a running surface and a trace appears.

> ns-workers is a shorthand you install once
>
> ns-workers
>
> is a name
>
> you
>
> give the workers plugin's CLI — the scaffold does not create it. Install it once, globally, and every
>
> ns-workers
>
> command on this page works as written:
>
> ```bash
> deno install -gArf -n ns-workers jsr:@netscript/plugin-workers@0.0.1-beta.11/cli
> ```
>
> Rather not install it? Each
>
> ns-workers <verb …>
>
> is exactly
>
> deno x -A jsr:@netscript/plugin-workers@0.0.1-beta.11/cli <verb …>
>
> — run that full form instead.

```bash
# Workers API on :8091 — enqueue the sample health-check job by id.
# Dispatch + execution + scheduler spans are emitted automatically.
ns-workers trigger workers-plugin-health-check

# then list recent executions
ns-workers executions --limit=10 --json
```

```bash
# Users oRPC service on :3001 — the request is traced at the service layer.
# oRPC services are served under /api/rpc/* (not /rpc).
curl -s -X POST http://localhost:3001/api/v1/users/list \
  -H 'content-type: application/json' \
  -d '{}'
```

```bash
# Triggers API on :8093 (raw Hono routes, not oRPC) — resolves the inbound
# trigger, whose enqueueJob action enqueues the workers health-check job,
# producing a connected dispatch + execution trace.
curl -s -X POST http://localhost:8093/api/v1/webhooks/inbound/generic \
  -H 'content-type: application/json' \
  -d '{}'
```

Refresh the **Traces** tab in the dashboard. You will see a trace for the request, with the service or worker resource as the root span. Click it to expand the span tree, attributes, and the structured logs correlated to that trace. A webhook that enqueues a job shows the inbound request span and the resulting job-dispatch and job-execution spans — the dispatcher propagates `traceparent` into the worker subprocess, so they share one trace.

> Where the real job spans come from
>
> The worker dispatcher wraps each run in
>
> traceJobExecution
>
> and emits
>
> job.started
>
> /
>
> job.completed
>
> /
>
> job.failed
>
> events, a duration, and a status; the scheduler emits its own start and per-run spans; the multi-runtime task executor emits
>
> task.execute
>
> spans. All of that is automatic — you are seeing real instrumentation from
>
> @netscript/telemetry
>
> , not the scaffold helpers.

## Step 3 — Add your own spans in a job handler (the real way today)

The scaffold hands you `createJobTools(ctx)` so you can author against `log`, `progress`, and `trace`. `log.*` is real today. The `trace.*` helpers are **no-op stubs in the scaffold** — if you want a real custom span around a unit of work right now, call the `@netscript/telemetry` instrumentation helpers directly. Keep the `trace.*` calls if you like authoring against the forward-compatible shape, but do not rely on them for live spans yet.

```ts
import {
  createFailureResult,
  createSuccessResult,
  defineJobHandler,
} from '@netscript/plugin-workers-core';
// Call the instrumentation helpers directly for a REAL child span today.
import {
  recordJobProgress,
  withChildSpan,
} from '@netscript/telemetry/instrumentation';
import { createJobTools } from './job-tools.ts';

const handler = defineJobHandler(async (ctx) => {
  const { log } = createJobTools(ctx); // log.* is real today
  log.info('Starting workers plugin health check');

  // withChildSpan opens a real span as a child of the active job span
  // and gives you a handle to attach queryable attributes.
  const envOk = await withChildSpan('check.environment', async (span) => {
    span.setAttribute('check.name', 'environment');
    return Boolean(Deno.env.get('PORT'));
  });

  if (!envOk) return createFailureResult('environment check failed');

  // Emit a real job.progress event (current / total / percentage).
  recordJobProgress(1, 1);
  return createSuccessResult({ status: 'healthy' });
});

export default Object.assign(handler, {
  id: 'workers-plugin-health-check' as const,
});
```

```ts
import { defineJobHandler } from '@netscript/plugin-workers-core';
import { createJobTools } from './job-tools.ts';

// createJobTools(ctx) returns:
//   log          -> console.* wrappers: info/warn/error/debug (plain stdout today, NOT trace-correlated)
//   progress     -> progress(percent, message) (forwards to ctx.reportProgress)
//   trace        -> { addEvent, recordProgress, withChildSpan } (STUB spans today)
//   traceContext -> { traceparent, tracestate } for manual propagation
//
// Authoring against trace.* is fine — your code is ready for when the
// scaffold helpers are upgraded — but these calls emit NO real spans today.
// Prefer @netscript/telemetry helpers (other tab) for spans you need now.
const handler = defineJobHandler(async (ctx) => {
  const { log, trace, traceContext } = createJobTools(ctx);
  log.info('health check', { traceparent: traceContext.traceparent });
  trace.addEvent('health_check.started'); // no-op in the scaffold today
  return { ok: true } as const;
});

export default handler;
```

```ts
// @netscript/telemetry/instrumentation — the real worker helpers:
//   traceJobExecution(...)  wrap a whole job run in a span (dispatcher uses this)
//   withChildSpan(name, fn) open a child span under the active context
//   addJobStepEvent(...)    emit a job.step.* event
//   recordJobProgress(c, t) emit a job.progress event (current / total / %)
//   runTracedJob(...)       run a job body inside subprocess trace context
//   startWorkerSpan(...)    worker lifecycle span
//
// Companion subpaths:
//   @netscript/telemetry/context     active trace context + traceparent helpers
//   @netscript/telemetry/attributes  canonical OTel attribute keys
//   @netscript/telemetry/orpc        oRPC client/server trace interceptors
// Authoritative export map: /reference/telemetry/
```

> Where the scaffold log.* helpers actually go
>
> log.info
>
> /
>
> log.warn
>
> /
>
> log.error
>
> in the scaffold's
>
> createJobTools(ctx)
>
> forward to
>
> console.*
>
> — they appear in process stdout (surfaced in the Aspire dashboard's
>
> Console logs
>
> view per resource),
>
> not
>
> as OTel-correlated structured log records tied to the active trace. For trace-correlated structured logging from a handler, use
>
> @netscript/logger
>
> directly. Use real spans (the
>
> withChildSpan
>
> helper) when you want queryable attributes and durations on a unit of work.

## Step 4 — Extend service tracing and propagate `traceparent`

The services keep trace context across the oRPC/HTTP boundary. The workers service builds its app with the fluent builder and opts the RPC layer into trace context explicitly:

```ts
import { createService } from '@netscript/service';
import { router } from './router.ts';

// The plugin API services use the fluent builder. withRPC({ traceContext: true })
// threads the incoming traceparent through to handlers. oRPC is served under
// /api/rpc/* by default.
await createService(router, { name: 'workers', version: '1.0.0', port: 8091 })
  .withCors()
  .withLogger()
  .withOpenAPI({ title: 'Workers API' })
  .withDatabase(dbClient)
  .withRPC({ traceContext: true })
  .withHealth()
  .serve();
```

```ts
import { defineService } from '@netscript/service';
import { router } from './router.ts';

// Local services use the one-call form. Tracing is enabled by the framework;
// debug: true surfaces verbose request/trace logs while you wire things up.
await defineService(router, {
  name: 'users',
  version: '1.0.0',
  port: parseInt(Deno.env.get('PORT') || '3001'),
  openapi: { title: 'Users API', description: 'users service' },
  debug: true,
});
```

```bash
# When one service calls another over HTTP, forward the W3C traceparent
# header so both spans land in ONE trace in the dashboard.
curl -s http://localhost:3001/api/v1/users/list \
  -X POST -H 'content-type: application/json' \
  -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
  -d '{}'
```

Aspire injects `PORT` at runtime, so the entrypoint reads it from the environment; the typed source of truth is your `netscript.config.ts` `services.<name>.port` field, which the scaffold wires as the fallback default — set the port there rather than editing this line.

> traceparent is the join key
>
> NetScript follows the W3C Trace Context standard. When an inbound request carries a
>
> traceparent
>
> header, the service continues that trace instead of starting a new one — and the worker dispatcher propagates
>
> traceparent
>
> /
>
> tracestate
>
> into the job subprocess. So a webhook on
>
> :8093
>
> that enqueues a job and the worker subprocess that runs it appear as a
>
> single correlated trace
>
> , live, today. Always forward
>
> traceparent
>
> on service-to-service calls.

## Step 5 — Read the traces

Back in the dashboard at [https://localhost:18888](https://localhost:18888):

1. **Traces** — every request and job as a waterfall of spans. Click a root span to drill into children, durations, and attributes (the ones you set via `span.setAttribute(...)`). Job runs show `job.started` / `job.completed` events and, where you added them, `job.progress` and `job.step.*` events.
2. **Structured logs** — filter by resource (e.g. `workers-api`) or by trace id to see the `log.info`/`log.error` lines correlated to a span.
3. **Resources** — health and console output per resource; cross-reference a failing span with the resource that produced it.

Confirm liveness independently of the UI by hitting the per-capability health endpoints:

```bash
curl -s http://localhost:8091/health        # workers
curl -s http://localhost:8092/health/live   # sagas
curl -s http://localhost:8093/health        # triggers
curl -s http://localhost:8094/health        # auth
```

## Production pitfalls

> Custom handler spans: use the helpers, not the stubs
>
> The scaffold
>
> createJobTools(ctx)
>
> trace.addEvent
>
> ,
>
> trace.withChildSpan
>
> , and
>
> trace.recordProgress
>
> are
>
> no-op stubs
>
> today (tracked debt, fix planned). The dispatcher, scheduler, and task subprocess spans around your handler are real and need nothing from you — but for a
>
> custom
>
> span inside the handler, call
>
> @netscript/telemetry/instrumentation
>
> helpers (
>
> withChildSpan
>
> ,
>
> recordJobProgress
>
> ,
>
> addJobStepEvent
>
> ) directly so the span actually emits.

> No collector, no traces
>
> Spans are only collected while Aspire is running and the OTLP endpoint (
>
> http://localhost:4318
>
> ) is reachable. Running a service standalone with
>
> --no-aspire
>
> means tracing has nowhere to export to. If the Traces tab is empty, check that
>
> aspire start
>
> is up first — Aspire provisions the collector and dashboard used by the examples.

> Set attributes, not log lines, for span data
>
> Data you want to filter or aggregate on (ids, counts, statuses) belongs on the span via
>
> span.setAttribute(...)
>
> , not buried in a log string. Attributes are queryable in the dashboard; freeform log text is not. Reserve
>
> log.*
>
> for human-readable narrative and errors.

## See also

[Telemetry capability  The OTel-in-handlers capability hub — headline API, ports, and the Learn / Do / Reference triplet.](https://rickylabs.github.io/netscript/netscript/capabilities/telemetry/)

[Observability (explanation)  The mental model: spans, structured logs, health endpoints, and how Aspire renders distributed traces across services and jobs.](https://rickylabs.github.io/netscript/netscript/explanation/observability/)

[telemetry reference  The generated @netscript/telemetry API surface — the full, authoritative export map (instrumentation, context, attributes, orpc).](https://rickylabs.github.io/netscript/netscript/reference/telemetry/)

[logger reference  The structured logger used by log.info / log.warn / log.error inside handlers and jobs.](https://rickylabs.github.io/netscript/netscript/reference/logger/)