# Telemetry & logging

The most expensive turn in any change loop — whether the author is you or an AI agent — is the verification turn: *did the change actually work, across every process it touched?* NetScript closes that loop inside the framework. The runtimes emit real OpenTelemetry spans on their own, a single W3C `traceparent` groups a cross-process flow into **one distributed trace**, and a typed query surface reads the resulting spans back — so "did it work" is a trace you open (or query programmatically), not four console logs you correlate by eye.

Concretely, observability is a built-in, not a bolt-on. Every service and plugin runtime is wired for **OpenTelemetry** — services serve their RPC handlers with trace-context propagation (first-party oRPC and Hono instrumentation), the worker runtime wraps job dispatch, execution, scheduling, subprocess hand-off, and task execution in real OTel spans, and structured logs flow through the framework `logger`. The viewing surface is the **Aspire dashboard at `https://localhost:18888`**, which collects OTLP traces, metrics, and structured logs from every resource in the app graph (services, plugin APIs, background processors) the moment you run `aspire start`. You do not stand up Jaeger, Grafana, or a log shipper to get started — the AppHost provisions the OTLP collector and the dashboard for you.

![A W3C traceparent header propagates from a service request through the worker runtime into a subprocess and across a saga boundary, keeping every span under one trace id.](https://rickylabs.github.io/netscript/netscript/assets/diagrams/otel-traceparent.svg)

*Trace-context propagation: a single traceparent (W3C) flows service → worker → subprocess → saga, so spans from every runtime join one distributed trace in Aspire.*

This page is the capability hub: what telemetry exists, how to emit it, where to view it, and the one place that is still a scaffold stub. For the full generated API of each unit, follow the reference links: the telemetry primitives live at [`/reference/telemetry/`](https://rickylabs.github.io/netscript/reference/telemetry/) and the structured logger at [`/reference/logger/`](https://rickylabs.github.io/netscript/reference/logger/).

> Aspire first, then telemetry
>
> The dashboard, the OTLP collector, and the per-resource trace/log views all come up with the orchestrator — they are not separate processes you start by hand. Run
>
> cd aspire && aspire start
>
> before
>
> any
>
> netscript db
>
> command (Aspire provisions Postgres and Redis first), then open the dashboard URL printed in the console (
>
> https://localhost:18888
>
> , with a one-time auth token). Until Aspire is running there is no
>
> :18888
>
> surface to view traces or logs on. See
>
> Database & migration
>
> for the full startup order.

## The story: one grouped trace, end to end

Picture the failure mode that eats an afternoon. A trigger fires, the scheduler dispatches, a queue hands the job to a worker, the job reports success — and the downstream callback never lands. With per-process logs you tail one console per resource and correlate timestamps by eye; an AI agent in the same position burns its turns asking you to check logs it cannot see.

In NetScript that whole flow is **one grouped trace**: trigger → scheduler dispatch → queue enqueue/dequeue → `job.execute` → the SDK's `rpc.client` callback span, all under a single trace id, because the runtimes propagate W3C trace context across every process boundary for you. Where many producers feed one consumer — a fan-in into a durable-streams processor — `createFanInLinks` attaches **span links** from each producer trace instead of re-parenting, so the consumer span shows exactly which upstream flows fed it. And this shape is not aspirational: the grouped cross-process trace is exercised by a non-mocked merge gate in the framework's own CI, which scaffolds a real project, runs the flow under Aspire, and reads the live spans back before a change can land.

The same spine covers AI workloads. Spans follow the upstream `gen_ai.*` semantic conventions (via `createGenAiAttributes` in `@netscript/telemetry/attributes`), so a traced agent turn shows the model call, the tool calls, and the durable write it produced — step by step, on one tree.

## What it is

Telemetry in NetScript is three OpenTelemetry signals — **traces**, **metrics**, and **logs** — wired through one package, `@netscript/telemetry`, and one viewing surface, the Aspire dashboard. Traces are the headline: a request enters a service, the worker runtime dispatches and executes a job, a subprocess runs a polyglot task, and a saga advances — and because a single **W3C `traceparent`** header propagates across every one of those process boundaries, the whole fan-out collapses into one trace tree. Metrics (worker counts, SSE connection stats) and structured logs ride the same OTLP export. The framework owns transport (OTLP/HTTP to the collector) and viewing (the dashboard); it now owns most of emission too — the worker, scheduler, queue, saga, and SSE runtimes are instrumented for you. The full mental model — what is framework-real versus a scaffold stub — is in [Observability](https://rickylabs.github.io/netscript/explanation/observability/).

Under the hood, `@netscript/telemetry` is structured as **telemetry ports and adapters**: application and runtime code programs against port contracts (`TracerProviderPort`, `MeterPort`, `PropagatorPort`, `SpanLinkPort`, `TelemetryQueryPort` — the `@netscript/telemetry/otel` subpath), with a zero-dependency default adapter (`OtelDenoTracerProvider`, bound to Deno's built-in OTLP exporter via `OTEL_DENO=true`) and an opt-in OpenTelemetry-SDK adapter (`OtelSdkTracerProvider`) for apps that bring the JS SDK — which also unlocks attribute-preserving span links. Attribute names follow one convention: upstream semconv keys (`rpc.*`, `messaging.*`, `gen_ai.*`, `server.*`) wherever OpenTelemetry defines them, and NetScript-owned keys under the single proprietary root `netscript.*` — correlated spans share `netscript.correlation.id`. The full rule set (span naming, SpanKind, status, propagation) is the [telemetry convention](https://rickylabs.github.io/netscript/reference/telemetry/convention/).

## Learn → / Do →

[Do — Add OpenTelemetry  Task-oriented recipe: lean on the automatic job spans, add custom spans via @netscript/telemetry withChildSpan, propagate traceparent across services and subprocesses, and point the OTLP export (:4318) at the dashboard.](https://rickylabs.github.io/netscript/netscript/observability/how-to/add-opentelemetry/)

[Understand — Observability  The mental model: how spans, structured logs, health endpoints, and Aspire traces fit together — and the precise framework-vs-scaffold map of what is real (traceJobExecution, task.execute) versus the tracked createJobTools(ctx) stub.](https://rickylabs.github.io/netscript/netscript/explanation/observability/)

[Look up — telemetry reference  The full generated deno doc API for @netscript/telemetry — tracer facade, context/traceparent helpers, the NetScript instrumentation helpers, and the oRPC tracing plugin.](https://rickylabs.github.io/netscript/netscript/reference/telemetry/)

## What you get out of the box

Telemetry in NetScript spans three layers — emission (in framework runtimes and your handler code), transport (OTLP over HTTP to the collector), and viewing (the Aspire dashboard). The framework owns transport and viewing, and it now owns most of emission too: the worker runtime traces the entire job lifecycle automatically. The only thing you still wire by hand is custom spans *inside* a job handler — and even there the `@netscript/telemetry` helpers do the work.

Structured logging

The @netscript/logger unit gives every service and plugin a level-aware, structured logger. Config is declared in netscript.config.ts (logging: { level: 'info', format: 'text' }) and threaded through the runtime — text or JSON, filterable in the dashboard.

OpenTelemetry traces (real)

Service request paths propagate W3C trace context, and the worker runtime emits real spans for job dispatch, execution, scheduling, subprocess hand-off, and task.execute. @opentelemetry/api (^1.9) is in the catalog and wired into the request and job paths so spans flow to the collector.

OTLP → Aspire dashboard

The generated AppHost configures an OTLP endpoint (http://localhost:4318) and the Aspire dashboard (https://localhost:18888) so traces, metrics, and logs from every resource land in one place — no collector to deploy.

Browser console logs (default)

Generated app resources call withBrowserLogs() automatically (the AppHost pins Aspire.Hosting.Browsers), so a Fresh/Vite app's browser console output — including a client-side error an island throws — is forwarded into the dashboard next to server logs and traces. No opt-in, nothing to wire.

Per-resource health

Each plugin API exposes a liveness probe — workers :8091 GET /health, sagas :8092 GET /health/live, triggers :8093 GET /health, auth :8094 — surfaced as resource state in the dashboard.

> Browser logs land in the dashboard too
>
> Because generated app resources emit
>
> withBrowserLogs()
>
> by default, the class of failure that is otherwise invisible from the server — a client-side
>
> TypeError
>
> a durable-streams island throws in the browser, say — shows up in the Aspire dashboard's
>
> Console logs
>
> for that app resource, correlated with the server-side spans of the same request. You do not start a separate browser-log collector; the AppHost wires it when it scaffolds the app.

## Enable tracing & see a span

Tracing turns on from the **environment** — `@netscript/telemetry/config` resolves a `TelemetryConfig` from the standard `OTEL_*` variables, and the Aspire AppHost already sets them for you. The fastest path is to run `aspire start`, trigger any job, and the runtime's automatic spans show up in the dashboard with no code change. The tab below adds a **custom** span on top of that automatic trace.

> Local vs deployed: same instrumentation, different collector
>
> Locally
>
> , Aspire provisions the OTLP collector and the dashboard and injects the
>
> OTEL_*
>
> variables into every resource, so telemetry works the moment you run
>
> aspire start
>
> — nothing to configure. Note the dashboard and OTLP endpoints bind to
>
> ephemeral ports
>
> (the generated profile uses
>
> localhost:0
>
> ), so use the URL
>
> aspire start
>
> prints rather than assuming
>
> :18888
>
> /
>
> :4318
>
> . Running
>
> aspire start --isolated
>
> (an upstream Aspire CLI flag) also randomizes the dashboard, OTLP, and resource-service ports and isolates secrets, so several apphosts can boot in parallel without colliding.
>
> Deployed
>
> , there is no Aspire and no dashboard: you point
>
> OTEL_EXPORTER_OTLP_ENDPOINT
>
> at your own collector or a hosted backend (Grafana Tempo, Honeycomb, Jaeger, …). The instrumentation code is identical — only the endpoint/protocol env and the sampler change. That single OTLP seam is the whole porting story; see
>
> Production notes
>
> below.

```ts
// netscript.config.ts — declare the structured-logging contract for the workspace.
// Tracing itself is enabled by OTEL_* env vars (the Aspire AppHost sets these); the
// config package reads them — there is no defineTelemetry() call to make.
import { defineConfig } from '@netscript/config';

export default defineConfig({
  name: 'my-app',
  version: '1.0.0',
  logging: { level: 'info', format: 'text' }, // level: debug|info|warn|error
  plugins: ['./plugins/workers/mod.ts'],
});

// Anywhere in app code you can inspect what telemetry resolved (log-safe, redacted):
import { describeTelemetryConfig, isTelemetryEnabled } from '@netscript/telemetry/config';
if (isTelemetryEnabled()) console.log(describeTelemetryConfig());
// → { enabled: true, endpoint: 'http://localhost:4318', protocol, serviceName, sampler }
```

```ts
// plugins/workers/jobs/health-check.ts — a real child span inside a handler.
// The dispatcher already wraps this job in traceJobExecution automatically;
// withChildSpan opens a child span under that parent. recordJobProgress's 3rd arg
// is a UNIT label (e.g. 'steps'), not a free-text message.
import { createSuccessResult, defineJobHandler } from '@netscript/plugin-workers-core';
import { withChildSpan, recordJobProgress } from '@netscript/telemetry/instrumentation';

const handler = defineJobHandler(async (ctx) => {
  recordJobProgress(1, 2, 'steps'); // → job.progress event on the trace

  const envCheck = await withChildSpan('check.environment', async (span) => {
    span.setAttribute('check.name', 'environment');
    return { ok: true };
  });

  recordJobProgress(2, 2, 'steps');
  return createSuccessResult({ healthy: envCheck.ok });
});

export default Object.assign(handler, { id: 'workers-plugin-health-check' as const });
```

> Lowest-effort observability path
>
> Run
>
> aspire start
>
> , trigger a job, and open
>
> :18888
>
> →
>
> Traces
>
> . You will see the dispatch span, the execution span, progress and step events, and — for subprocess or polyglot tasks — a continued trace across the process boundary. None of that requires editing the scaffold. Reach for custom instrumentation only when you want spans
>
> inside your own handler logic
>
> .

## Key types: the telemetry init/config surface

Configuration is **read from the environment**, not constructed — `@netscript/telemetry/config` resolves and caches a `TelemetryConfig` from the `OTEL_*` variables. These are the confirmed fields and the helpers that read them.

**TelemetryConfig (@netscript/telemetry/config — resolved from OTEL_* env vars)**

| Name | Type | Description |
| --- | --- | --- |
| `enabled` | `boolean` | Whether OpenTelemetry instrumentation is active (driven by OTEL_DENO / endpoint presence). Mirror it with isTelemetryEnabled(). |
| `endpoint` | `string \| undefined` | OTLP exporter endpoint URL (e.g. http://localhost:4318). Read directly via getOtlpEndpoint(). |
| `protocol` | `string` | OTLP exporter protocol (e.g. http/protobuf), from OTEL_EXPORTER_OTLP_PROTOCOL. |
| `serviceName` | `string` | Service name reported to backends, from OTEL_SERVICE_NAME (or the default). Read via getServiceName(). |
| `serviceVersion` | `string` | Service version reported to backends. |
| `resourceAttributes` | `Record` | Resource attributes parsed from OTEL_RESOURCE_ATTRIBUTES. |
| `sampler` | `string` | Trace sampler name, from OTEL_TRACES_SAMPLER (e.g. parentbased_always_on, traceidratio). |
| `debug` | `boolean` | Whether debug-level telemetry logging is on (OTEL_LOG_LEVEL). |

**Config helpers (@netscript/telemetry/config)**

| Name | Type | Description |
| --- | --- | --- |
| `getTelemetryConfig()` | `→ TelemetryConfig` | Resolve telemetry config from the OTEL_* environment variables. |
| `getConfig()` | `→ TelemetryConfig` | Return the process-cached config (resetConfig() clears the cache, mainly for tests). |
| `isTelemetryEnabled()` | `→ boolean` | Whether instrumentation is enabled for this process. |
| `getOtlpEndpoint()` | `→ string \| undefined` | The configured OTLP endpoint, when present. |
| `getServiceName()` | `→ string` | The configured service name, or the default. |
| `describeTelemetryConfig()` | `→ TelemetryConfigDescription` | A redacted, log-safe summary of the resolved config — safe to print in diagnostics. |
| `OTEL_ENV_VARS` | `const record` | The OTEL_* variable NAMES NetScript reads: OTEL_DENO, OTEL_EXPORTER_OTLP_ENDPOINT/_PROTOCOL, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, OTEL_LOG_LEVEL, the BSP/BLRP schedule delays, and OTEL_METRIC_EXPORT_INTERVAL. |

## Span & instrumentation helpers

Two subpaths cover hand-written instrumentation. `@netscript/telemetry/tracer` is the low-level facade over `@opentelemetry/api` — get a tracer, open a span, run a callback inside one. `@netscript/telemetry/instrumentation` is the NetScript-domain layer the runtimes themselves use — job, scheduler, queue, and SSE spans with the right attributes baked in. The `/context` subpath holds the **`traceparent`** propagation helpers that make the cross-boundary trace in the diagram above possible.

**Tracer facade (@netscript/telemetry/tracer)**

| Name | Type | Description |
| --- | --- | --- |
| `getTracer(name, version)` | `→ Tracer` | Cached tracer for an instrumentation name/version. Domain shortcuts exist: getJobTracer/getQueueTracer/getWorkerTracer/getSchedulerTracer/getSagaTracer/getSSETracer/getKVTracer. |
| `withSpan(tracer, name, fn, options)` | `→ Promise` | Run an async callback inside a span and close it on completion. withSpanSync is the synchronous form. |
| `createSpan(tracer, name, options)` | `→ Span` | Create a span from a tracer + CreateSpanOptions (kind, attributes, parentContext, links). You end() it yourself. |
| `setSpanAttributes / setSpanError / setSpanOk` | `(span, …) → void` | Bulk-set attributes, mark a span failed (with an optional Error), or mark it OK. addSpanEvent(span, name, attrs?) adds an event. |
| `getActiveSpan() / getActiveContext()` | `→ Span? / Context` | Read the span / OTel context active on the current async path. isTracingEnabled() gates work. |
| `SpanKind / SpanStatusCode / TracerNames` | `const` | Span-kind (INTERNAL/SERVER/CLIENT/PRODUCER/CONSUMER), status (UNSET/OK/ERROR), and the standard domain tracer-name strings. |

**NetScript instrumentation helpers (@netscript/telemetry/instrumentation)**

| Name | Type | Description |
| --- | --- | --- |
| `traceJobExecution(options, fn)` | `→ Promise` | Wrap a job run in a span with job.started/completed/failed events. Emitted automatically by the dispatcher; the supported entry point for tracing a job by hand. |
| `withChildSpan(name, fn, attributes?)` | `→ Promise` | Open a child span under the active span and run fn inside it. The go-to for custom spans in a handler. |
| `recordJobProgress(current, total, unit)` | `→ void` | Emit a job.progress event. The 3rd arg is a UNIT label (e.g. 'steps'), NOT a description. addJobStepEvent(stepName, attrs?) records job.step.* events. |
| `startJobDispatchSpan / traceJobDispatch` | `span / → Promise` | Open the dispatch span and return PropagationHeaders to carry traceparent to the executor. createJobSubprocessEnv injects traceparent/tracestate into a subprocess env. |
| `scheduler & cron spans` | `span / event` | createSchedulerStartSpan, createScheduleJobSpan, startSchedulerTickSpan, traceSchedulerTick, recordCronJobRun — cron/schedule dispatch traced end to end. |
| `traceQueue(queue, options) / TracedQueue` | `→ TracedQueue` | Wrap a MessageQueue so enqueue/consume carry trace context. SSE helpers (startSSEConnection, createSSEEventSpan, traceSSEEvent) trace Server-Sent-Event streams. |

**Traceparent / context propagation (@netscript/telemetry/context)**

| Name | Type | Description |
| --- | --- | --- |
| `formatTraceparent(spanContext)` | `→ string` | Serialize a span context to a W3C traceparent header value. parseTraceparent(value) parses one back to a ParsedTraceparent \| null. |
| `injectContext(headers, ctx?) / extractContext(headers)` | `→ headers / → Context` | Inject the active trace context into outbound PropagationHeaders, or extract a remote one from inbound headers — the core of cross-service propagation. |
| `withContext(ctx, fn) / withContextAsync(ctx, fn)` | `→ T / → Promise` | Run a callback with a given OTel context active so spans created inside attach to the right parent. |
| `createMessageHeaders / resolveParentContextFromHeaders` | `→ headers / → Context` | Build propagation headers for a queue message, and resolve the parent context back out on the consumer side. |
| `getTraceId(ctx?) / getSpanId(ctx?)` | `→ string?` | Read the current trace/span ids — handy for correlating structured log lines with a trace. |

## Service, RPC & HTTP tracing

Services opt into trace propagation through **first-party instrumentation** rather than by hand — NetScript wraps the upstream OTel packages instead of reinventing them. The `@netscript/telemetry/orpc` subpath ships an oRPC **tracing plugin** (backed by the upstream `@orpc/otel` instrumentation) that opens a SERVER span per RPC and continues any inbound `traceparent`, plus an error-handling plugin that classifies failures. The `@netscript/telemetry/hono` subpath does the same for plain HTTP: `createHonoTracingMiddleware` wraps Hono's first-party `@hono/otel` middleware and layers NetScript service naming and W3C propagation on top — the service builder registers it outermost for you. On the service builder the RPC side is the `.withRPC({ traceContext: true })` toggle (see [Services](https://rickylabs.github.io/netscript/services-sdk/services/)); the factories below are the underlying surface if you mount oRPC or Hono yourself.

**oRPC + Hono tracing surface (@netscript/telemetry/orpc, @netscript/telemetry/hono)**

| Name | Type | Description |
| --- | --- | --- |
| `createTracingPlugin(options?)` | `→ TracingPlugin` | oRPC plugin that opens a SERVER span per call and continues an inbound traceparent, following upstream rpc.* semconv. Options are TracingPluginOptions. |
| `registerORPCInstrumentation(config?)` | `→ void` | Registers the upstream @orpc/otel ORPCInstrumentation the tracing plugin is backed by — wrap-don't-reinvent, applied to oRPC. |
| `createErrorHandlingPlugin(options?)` | `→ ErrorHandlingPlugin` | Classifies errors as client \| server \| transient and records them on the span; takes an optional ErrorClassifier and ErrorLogger. |
| `createTraceContext()` | `→ TraceContext` | Build the per-call trace context the plugin threads through. addEvent / setAttributes / getTraceId / getSpanId operate on the active span. |
| `createHonoTracingMiddleware(options)` | `→ HonoTracingMiddleware` | Hono middleware wrapping @hono/otel's httpInstrumentationMiddleware with NetScript service naming + W3C propagation. app.use('*', createHonoTracingMiddleware({ serviceName: 'users' })). |

> Database & worker tracing toggles live next door
>
> The Prisma query-tracing and worker-runtime tracing toggles are documented on their own hubs to avoid duplication. Database query spans are wired through the
>
> database
>
> client; the worker runtime's automatic job spans are described in
>
> background jobs
>
> . This hub owns the
>
> cross-cutting
>
> telemetry surface — config, the tracer facade, traceparent propagation, and the oRPC plugin — that those runtimes build on. See
>
> [Database & Prisma](https://rickylabs.github.io/netscript/netscript/data-persistence/database/) for the per-query toggle.

## What is traced automatically

Before you write a single instrumentation call, the worker runtime already produces a usable trace tree in Aspire. The `@netscript/telemetry` instrumentation is wired into the dispatcher, the executor, and the scheduler, so the moment a job runs you get spans with attributes, durations, status, and lifecycle events — no scaffold changes required.

**Automatic worker traces (emitted by the runtime — see /reference/telemetry/)**

| Name | Type | Description |
| --- | --- | --- |
| `traceJobExecution` | `span` | Wraps each job's execution with attributes, duration, status, and job.started / job.completed / job.failed / job.exception events. Emitted by the dispatcher. |
| `recordJobProgress` | `event` | job.progress events carrying current / total / unit — the runtime records real progress as the job advances. |
| `addJobStepEvent` | `event` | job.step.* events for each step the runtime walks through. |
| `scheduler spans` | `span` | createSchedulerStartSpan / createScheduleJobSpan / startSchedulerTickSpan / recordCronJobRun — cron and schedule dispatch are traced end to end. |
| `subprocess traceparent` | `context` | The dispatcher injects W3C traceparent / tracestate into the subprocess env so out-of-process job runs continue the same trace (initJobTracing / runTracedJob / createJobSubprocessEnv). |
| `task.execute` | `span` | The multi-runtime task executor wraps each task run in a task.execute span — polyglot and TS tasks alike show up in the trace tree. |

## Instrument a handler

The two tabs below show the two emission paths a developer touches: structured logging through the framework logger, and **custom** spans inside a job handler. For custom spans, call the `@netscript/telemetry` helpers (`traceJobExecution`, `withChildSpan`, `recordJobProgress`) directly — they are the real, supported surface. Read the callout under the tabs to understand why you reach for the telemetry package rather than the scaffold's `createJobTools(ctx)` trace helpers.

```ts
// netscript.config.ts — declare the logging contract for the whole workspace.
import { defineConfig } from '@netscript/config';

export default defineConfig({
  name: 'my-app',
  version: '1.0.0',
  // level: 'debug' | 'info' | 'warn' | 'error'; format: 'text' | 'json'
  logging: { level: 'info', format: 'text' },
  plugins: ['./plugins/workers/mod.ts', './plugins/sagas/mod.ts'],
});

// Inside a job handler, the logger comes from the job tools (console-backed today,
// and surfaced under the resource's Console logs view in the Aspire dashboard).
// import { createJobTools } from './job-tools.ts';
const emit = (log) => {
  log.info('user provisioned', { userId: 'u_123', source: 'scaffold' });
  log.warn('rate limit approaching', { remaining: 4 });
};
```

```ts
// plugins/workers/jobs/health-check.ts — real custom spans inside a handler.
// The dispatcher already wraps this job in traceJobExecution automatically;
// withChildSpan + recordJobProgress let you add detail under that parent span.
import { createSuccessResult, createFailureResult, defineJobHandler } from '@netscript/plugin-workers-core';
import { withChildSpan, recordJobProgress } from '@netscript/telemetry/instrumentation';

const handler = defineJobHandler(async (ctx) => {
  const { log } = ctx;
  log.info('Starting workers plugin health check');

  // Real progress event — 3rd arg is a UNIT label, not a message.
  recordJobProgress(1, 2, 'steps');

  // withChildSpan opens a real child span under the job-execution span.
  const envCheck = await withChildSpan('check.environment', async (span) => {
    span.setAttribute('check.name', 'environment');
    return { ok: true };
  });

  recordJobProgress(2, 2, 'steps');
  if (!envCheck.ok) return createFailureResult('environment check failed');
  return createSuccessResult({ status: 'healthy' });
});

export default Object.assign(handler, { id: 'workers-plugin-health-check' as const });
```

> Known gap: the scaffold createJobTools(ctx) trace/progress helpers are no-op stubs
>
> Job dispatch and execution are instrumented with
>
> real OTel spans
>
> — traces show up in Aspire automatically (
>
> traceJobExecution
>
> ,
>
> recordJobProgress
>
> , scheduler spans, and subprocess traceparent propagation are all live in the runtime). The one remaining gap is narrow and specific: the
>
> trace.{addEvent,withChildSpan,recordProgress}
>
> and
>
> progress(...)
>
> helpers returned by the scaffold's
>
> createJobTools(ctx)
>
> are currently
>
> no-op stubs
>
> in the generated copy.
>
> log
>
> writes to the console; those particular trace/progress helpers do nothing. This is a
>
> known, tracked limitation with a fix planned
>
> (debt
>
> workers-scaffold-job-tools-noop
>
> ), not a permanent design choice.
>
> Workaround:
>
> for custom spans and progress, call the
>
> @netscript/telemetry
>
> helpers directly (as in the tab above) rather than the scaffold
>
> createJobTools
>
> trace surface — those are real today. See
>
> Observability
>
> for the full framework-vs-scaffold map.

## Endpoints & ports

Telemetry has no single service of its own — it is emitted by every runtime and aggregated by Aspire. These are the real addresses you interact with, validated by the CLI E2E suite.

**Observability surfaces (link to /reference/telemetry/ and /reference/logger/ for the generated APIs)**

| Name | Type | Description |
| --- | --- | --- |
| `https://localhost:18888` | `dashboard` | Aspire dashboard — traces, structured logs, metrics, and resource state for the whole app graph. Auth token printed by `aspire start`. |
| `http://localhost:4318` | `OTLP/HTTP` | OTLP ingest endpoint the AppHost configures (aspire.config.json https profile). Runtimes export spans and logs here; the dashboard reads them back. This is the seam you point at a hosted backend. |
| `GET :8091/health` | `liveness` | Workers API health probe — reported as resource health in the dashboard. |
| `GET :8092/health/live` | `liveness` | Sagas API liveness route. |
| `GET :8093/health` | `liveness` | Triggers API health probe (Hono service). |
| `:8094` | `service` | Auth API (auth-api) — its request spans propagate trace context like any other service. |

## View it in Aspire

With `aspire start` up, the dashboard at `https://localhost:18888` gives you four views over the same telemetry stream — there is no separate tool to configure.

**Aspire dashboard views (https://localhost:18888)**

| Name | Type | Description |
| --- | --- | --- |
| `Resources` | `graph` | Live state of every app-graph resource: postgres, redis, workers-api, workers, sagas-api, sagas, triggers-api, triggers, auth-api. Health probes drive the status colour. |
| `Console logs` | `stream` | Per-resource stdout/stderr — the framework logger's text/JSON output lands here in real time. |
| `Structured logs` | `OTLP` | Structured log records exported over OTLP, filterable by resource, level, and attributes. |
| `Traces` | `OTLP` | Distributed traces collected from the OTLP endpoint (http://localhost:4318) — job dispatch/execution/scheduler/task spans and cross-service request spans as trace context propagates. |

> Use this when…
>
> Reach for the dashboard whenever you need to answer
>
> "what happened and where"
>
> : which resource is unhealthy, what a service logged for a given request, why a job failed, or how a call fanned out across services and subprocesses. Because Aspire wires OTLP for you, the lowest-effort observability path is to run
>
> aspire start
>
> and open
>
> :18888
>
> — no extra dependency, no collector to deploy. When you outgrow the local dashboard, the same OTLP export (
>
> http://localhost:4318
>
> ) is the seam you point at a hosted backend.

## Close the loop: read the trace back in code

The dashboard answers *"what happened and where"* for a human. The same telemetry has a **typed, in-language read side** for programs — and for an AI agent verifying its own change without a browser. `@netscript/telemetry/query` publishes a `TelemetryQueryPort` contract with an Aspire-backed reader: `createAspireTelemetryQuery(...)` (or `createTelemetryQuery(...)` for the port type) reads the Aspire `/api/telemetry/*` surface into package-owned read models — `TelemetryTrace`, `TelemetrySpan` (with its `events` and `links`), `TelemetryLog`, `TelemetryResource`, `TelemetryMetric`. So the verification turn becomes an assertion: fetch the trace by id, check that the span you expected is present with the status you expected — no console to tail, no screenshot to read.

**Telemetry query read side (@netscript/telemetry/query)**

| Name | Type | Description |
| --- | --- | --- |
| `createAspireTelemetryQuery(options) / createTelemetryQuery(options)` | `→ AspireTelemetryQuery / TelemetryQueryPort` | Build the Aspire-backed reader over the dashboard's /api/telemetry/* surface. The first returns the concrete adapter; the second returns it typed as the port contract. |
| `queryTraces(filter?) / getTrace(traceId)` | `→ Promise / Promise` | List traces matching a TraceQueryFilter, or fetch a single trace by id — the assertion primitive for 'did my flow produce the span I expected'. |
| `querySpans / queryLogs / queryMetrics / queryResources` | `→ Promise` | Read spans, structured logs, metrics, and resource records back as package-owned models (TelemetrySpan / TelemetryLog / TelemetryMetric / TelemetryResource). |
| `validateTraceQueryFilter / validateMetricQueryFilter / validateResourceQueryFilter` | `(filter) → filter` | Standard Schema validators for the query filters — a malformed query fails in-process rather than returning silently-wrong rows. |

> One comparison: reading traces back to verify a change
>
> Encore ships an MCP server that exposes its local traces to an AI agent, so the agent can read back what its change did. NetScript closes the same loop from the other side: instead of a separate protocol server, the trace read side is a **typed in-language port** (
>
> TelemetryQueryPort
>
> ) the app — or a test, or an agent's own tool — calls directly, returning the same package-owned read models the runtimes emit. Same goal — let the author verify a change *from the trace*, not by eye — kept inside the language and the type system.

## Production notes

> Footguns before you ship telemetry
>
> **❌ Trace 100% of traffic in production.** The local Aspire default samples everything — fine for dev, ruinous at scale.
>  **✅ Set a ratio sampler via env.** Sampling is an env decision, not a code one: set `OTEL_TRACES_SAMPLER` (surfaced as `TelemetryConfig.sampler`) to `traceidratio` with a rate. *Exception:* keep always-on sampling for low-volume services or a targeted debugging window.
>
> **❌ Fork the runtime to change telemetry backends.** Editing instrumentation to point at Grafana/Honeycomb is churn you will re-do every upgrade.
>  **✅ Point the exporter.** The OTLP export (`OTEL_EXPORTER_OTLP_ENDPOINT`, default `http://localhost:4318`) is the single seam — swap endpoint/protocol via env and the instrumentation code does not change.
>
> **❌ Put unbounded values in span attributes.** User ids, raw URLs, or full payloads in `span.setAttribute` explode index cardinality on the backend.
>  **✅ Use stable, low-cardinality keys.** Correlate to a specific record with `getTraceId()` in a structured log line instead of widening the span.
>
> **❌ Pass a message as `recordJobProgress`'s 3rd argument.** Free text there inflates cardinality and misreads the API.
>  **✅ Pass a stable unit label.** `'steps'` / `'items'` — the argument is a unit, not a description.
>
> **❌ Ship code that relies on the scaffold `createJobTools(ctx)` trace/progress helpers.** Those are no-op stubs today (debt `workers-scaffold-job-tools-noop`).
>  **✅ Call the `@netscript/telemetry` helpers directly** for real spans and progress, as in the handler tab above.

## Reference

This hub is intentionally thin — the full generated API lives in the reference.

[telemetry](https://rickylabs.github.io/netscript/netscript/reference/telemetry/)

[Do — Add OpenTelemetry  Task-oriented recipe: lean on the automatic job spans, add custom spans via @netscript/telemetry withChildSpan, propagate traceparent across services and subprocesses, and point the OTLP export (:4318) at the dashboard.](https://rickylabs.github.io/netscript/netscript/observability/how-to/add-opentelemetry/)

[Understand — Observability  The mental model: how spans, structured logs, health endpoints, and Aspire traces fit together — and the precise framework-vs-scaffold map of what is real (traceJobExecution, task.execute) versus the tracked createJobTools(ctx) stub.](https://rickylabs.github.io/netscript/netscript/explanation/observability/)

[Look up — telemetry reference  The full generated deno doc API for @netscript/telemetry — tracer facade, context/traceparent helpers, the instrumentation helpers, scheduler spans, and the oRPC tracing plugin. The authority for signatures.](https://rickylabs.github.io/netscript/netscript/reference/telemetry/)

[Look up — logger reference  The full generated deno doc API for @netscript/logger — log levels, formats, and the structured-logging surface declared in netscript.config.ts.](https://rickylabs.github.io/netscript/netscript/reference/logger/)

[KV, queues & cron](https://rickylabs.github.io/netscript/netscript/data-persistence/kv-queues-cron/) [Fresh UI & design](https://rickylabs.github.io/netscript/netscript/web-layer/fresh-ui/)