Skip to content
Browse documentation
typescript sdk

Routing, retries and fallbacks

Declare candidate models and bound physical attempts without hiding route changes.

Declare the available routes

Use a named policy when the application needs an explicit primary/fallback set. Contextual selection may filter or reorder that set; it cannot silently introduce arbitrary models.

import { createInference, modelPolicy } from "@smoothie-framework/runtime/model";

// providers and resolveCredential are server-owned bindings.
const inference = createInference({
  providers,
  resolveCredential,
  routing: {
    policies: {
      production: {
        primary: "openai/gpt-5.4-mini",
        fallbacks: ["backup/gpt-5.4-mini"],
      },
    },
  },
  execution: {
    maxAttemptsPerCandidate: 2,
    maxTotalAttempts: 4,
    deadlineMs: 60_000,
  },
});
const selection = modelPolicy("production");

Distinguish retry from fallback

Retry repeats a candidate only for retryable failures. Fallback moves to another declared candidate only when the failure is fallback-eligible. Both consume the execution policy's bounded physical attempts.

After streaming exposes a chunk, neither retry nor fallback is automatic. Check returned route/attempt evidence when diagnosing failures; do not infer which provider ran from the original requested policy.

Every physical attempt also passes budget admission when a controller is configured. Avoid stacking hidden provider retries on top of Framework-owned attempts.