Add OpenTelemetry

Scope: turn on, extend, and read distributed traces in an existing NetScript workspace. OpenTelemetry is not a bolt-on here — @netscript/telemetry wraps @opentelemetry/api and is wired into the services, the worker dispatcher, the scheduler, and the subprocess task runtime the scaffold generates. Aspire stands up the OTLP collector and the trace UI for you. This recipe shows where the instrumentation already lives, how to add your own spans and structured logs, how traceparent propagates across the oRPC/HTTP boundary and into job subprocesses, and how to watch it all land in the Aspire dashboard at https://localhost:18888.

This is a task recipe, not a deep-dive. For the mental model behind spans, structured logs, and the per-capability health endpoints, read Observability. For the generated API surface, follow the telemetry and logger reference pages, and the Telemetry capability hub for the Learn / Do / Reference triplet.

Prerequisites

How the telemetry is already wired

Three layers ship instrumented out of the box. Knowing which is which tells you where you get spans for free and where you add your own.

Step 1 — Bring up Aspire and confirm the collector

From the workspace root, start orchestration. The AppHost registers the OTLP collector and the dashboard, then boots Postgres, Redis, and every service/plugin resource.

cd aspire
aspire start
# dashboard: https://localhost:18888  (login token printed in the console)

Open https://localhost:18888, authenticate with the token Aspire printed, and select the Traces tab. With nothing exercised yet it is empty — that is expected. Leave it open; it updates live.

Step 2 — Generate a trace without writing any code

Both service-layer and worker-runtime tracing are real and need no code from you. Exercise a running surface and a trace appears.

# Workers API — enqueue the sample health-check job by id (on its assigned port).
# Dispatch + execution + scheduler spans are emitted automatically.
ns-workers trigger workers-plugin-health-check

# then list recent executions
ns-workers executions --limit=10 --json

# Users oRPC service — the request is traced at the service layer.
# Replace <users-port> with the port assigned to your users service.
# oRPC services are served under /api/rpc/* (not /rpc).
curl -s -X POST http://localhost:<users-port>/api/v1/users/list \
  -H 'content-type: application/json' \
  -d '{}'

# Triggers API on :8093 (raw Hono routes, not oRPC) — resolves the inbound
# trigger, whose enqueueJob action enqueues the workers health-check job,
# producing a connected dispatch + execution trace.
curl -s -X POST http://localhost:8093/api/v1/webhooks/inbound/generic \
  -H 'content-type: application/json' \
  -d '{}'

Refresh the Traces tab in the dashboard. You will see a trace for the request, with the service or worker resource as the root span. Click it to expand the span tree, attributes, and the structured logs correlated to that trace. A webhook that enqueues a job shows the inbound request span and the resulting job-dispatch and job-execution spans — the dispatcher propagates traceparent into the worker subprocess, so they share one trace.

Query or export a detached AppHost

Generated workspaces include task routes that query traces and logs so you never have to discover or remember the dashboard URL:

# Query traces or logs for a specific resource
deno task aspire:otel -- traces <resource>
deno task aspire:otel -- logs <resource>

# Query with timestamp search filter (ISO format or date)
deno task aspire:otel -- traces <resource> --search "timestamp:>=2026-08-30"

# Export collected telemetry to a structured zip archive
deno task aspire:export -- -o telemetry.zip

Generate traffic before querying traces; a healthy but idle AppHost legitimately returns an empty array. The generated task wrappers forward your arguments, try Aspire's automatic discovery first, and on failure query aspire ps --format Json for the matching AppHost path and retry with its active dashboardUrl. Bare aspire otel exits with code 12 when no dashboard is reachable; the task wrappers catch that exit and resolve the running instance automatically.

AI agents and external tools that connect through the NetScript MCP telemetry server (@netscript/mcp) resolve the endpoint by a fixed precedence — options first, then environment variables, then the default:

1. The explicit --endpoint option.
2. The NETSCRIPT_TELEMETRY_ENDPOINT environment variable.
3. The ASPIRE_DASHBOARD_PORT environment variable (e.g. 18888).
4. The default fallback endpoint, http://localhost:18888.

Step 3 — Add your own spans in a job handler

The scaffold hands you createJobTools(ctx) so you can author against log, progress, and trace. The trace and progress helpers emit through the active job telemetry context; direct @netscript/telemetry/instrumentation imports remain available for lower-level composition.

import {
  createFailureResult,
  createSuccessResult,
  defineJobHandler,
} from '@netscript/plugin-workers-core';
// Call the instrumentation helpers directly for a REAL child span today.
import {
  recordJobProgress,
  withChildSpan,
} from '@netscript/telemetry/instrumentation';
import { createJobTools } from './job-tools.ts';

const handler = defineJobHandler(async (ctx) => {
  const { log } = createJobTools(ctx); // log.* is real today
  log.info('Starting workers plugin health check');

  // withChildSpan opens a real span as a child of the active job span
  // and gives you a handle to attach queryable attributes.
  const envOk = await withChildSpan('check.environment', async (span) => {
    span.setAttribute('check.name', 'environment');
    return Boolean(Deno.env.get('PORT'));
  });

  if (!envOk) return createFailureResult('environment check failed');

  // Emit a real job.progress event (current / total / percentage).
  recordJobProgress(1, 1);
  return createSuccessResult({ status: 'healthy' });
});

export default Object.assign(handler, {
  id: 'workers-plugin-health-check' as const,
});

import { defineJobHandler } from '@netscript/plugin-workers-core';
import { createJobTools } from './job-tools.ts';

// createJobTools(ctx) returns:
//   log          -> console.* wrappers: info/warn/error/debug (plain stdout, NOT trace-correlated)
//   progress     -> emits job.progress and forwards to ctx.reportProgress
//   trace        -> real addEvent, recordProgress, and withChildSpan helpers
//   traceContext -> { traceparent, tracestate } for manual propagation
const handler = defineJobHandler(async (ctx) => {
  const { log, progress, trace, traceContext } = createJobTools(ctx);
  log.info('health check', { traceparent: traceContext.traceparent });
  trace.addEvent('health_check.started');
  await progress(50, 'half complete');
  await trace.withChildSpan('health_check.verify', async (span) => {
    span.setAttribute('check.kind', 'health');
  });
  return { ok: true } as const;
});

export default handler;

// @netscript/telemetry/instrumentation — the real worker helpers:
//   traceJobExecution(...)  wrap a whole job run in a span (dispatcher uses this)
//   withChildSpan(name, fn) open a child span under the active context
//   addJobStepEvent(...)    emit a job.step.* event
//   recordJobProgress(c, t) emit a job.progress event (current / total / %)
//   runTracedJob(...)       run a job body inside subprocess trace context
//   startWorkerSpan(...)    worker lifecycle span
//
// Companion subpaths:
//   @netscript/telemetry/context     active trace context + traceparent helpers
//   @netscript/telemetry/attributes  canonical OTel attribute keys
//   @netscript/telemetry/orpc        oRPC client/server trace interceptors
// Authoritative export map: /reference/telemetry/

Step 4 — Extend service tracing and propagate traceparent

The services keep trace context across the oRPC/HTTP boundary. The workers service builds its app with the fluent builder and opts the RPC layer into trace context explicitly:

import { createService } from '@netscript/service';
import { router } from './router.ts';

// The plugin API services use the fluent builder. withRPC({ traceContext: true })
// threads the incoming traceparent through to handlers. oRPC is served under
// /api/rpc/* by default.
await createService(router, { name: 'workers', version: '1.0.0', port: 8091 }) // note: your scaffold's port will differ
  .withCors()
  .withLogger()
  .withOpenAPI({ title: 'Workers API' })
  .withDatabase(dbClient)
  .withRPC({ traceContext: true })
  .withHealth()
  .serve();

import { defineService } from '@netscript/service';
import { router } from './router.ts';

// Local services use the one-call form. Tracing is enabled by the framework;
// debug: true surfaces verbose request/trace logs while you wire things up.
await defineService(router, {
  name: 'users',
  version: '1.0.0',
  port: parseInt(Deno.env.get('PORT') || '3001'), // note: your scaffold's port will differ
  openapi: { title: 'Users API', description: 'users service' },
  debug: true,
});

# When one service calls another over HTTP, forward the W3C traceparent
# header so both spans land in ONE trace in the dashboard.
# Replace <users-port> with your service's assigned port.
curl -s http://localhost:<users-port>/api/v1/users/list \
  -X POST -H 'content-type: application/json' \
  -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
  -d '{}'

Aspire injects PORT at runtime, so the entrypoint reads it from the environment; the typed source of truth is your netscript.config.ts services.<name>.port field, which the scaffold wires as the fallback default — set the port there rather than editing this line.

Step 5 — Read the traces

Back in the dashboard at https://localhost:18888:

1. Traces — every request and job as a waterfall of spans. Click a root span to drill into children, durations, and attributes (the ones you set via span.setAttribute(...)). Job runs show job.started / job.completed events and, where you added them, job.progress and job.step.* events.
2. Structured logs — filter by resource (e.g. workers-api) or by trace id to see the log.info/log.error lines correlated to a span.
3. Resources — health and console output per resource; cross-reference a failing span with the resource that produced it.

Confirm liveness independently of the UI by hitting the per-capability health endpoints:

# Replace ports with the ones dynamically assigned to your services:
curl -s http://localhost:<workers-port>/health        # workers
curl -s http://localhost:<sagas-port>/health/live   # sagas
curl -s http://localhost:<triggers-port>/health        # triggers
curl -s http://localhost:<auth-port>/health        # auth

Production pitfalls

See also