Skip to content
Browse documentation
typescript sdk

Usage and inference budgets

Separate recorded usage from atomic admission of each physical model attempt.

Observe usage without treating it as a limit

Successful inference returns normalized usage and route evidence. The optional synchronous onLifecycleEvent hook reports content-free plan/attempt events for application metrics. It must not return a promise; throwing aborts the invocation.

For provider logs, use the diagnostics channel instead. It queues records and isolates sink failures from inference. A slow or failed log destination can lose records, so use durable storage for usage accounting.

RunUsageTracker records post-execution usage. It does not reserve capacity and cannot enforce a concurrent spend ceiling. Keep accounting, estimated input size and provider billing evidence distinct.

Admit attempts atomically

import {
  createInMemoryInferenceBudgetController,
  type InferenceBudgetController,
} from "@smoothie-framework/runtime/model";

Supply an InferenceBudgetController through createInference's budget option to reserve and settle every physical attempt. Production controllers need an atomic durable store; the in-memory implementation is for single-process development and tests.

Choose estimates, exact pricing, reservation lifetime and reconciliation policy explicitly. A failed dispatched call may have incurred usage even without a usable response. Budgets do not automatically refund external charges.

For run records and audit history, see Timeline and usage records.