Routing, retries and fallbacks
Declare candidate models and bound physical attempts without hiding route changes.
Declare the available routes
Use a named policy when the application needs an explicit primary/fallback set. Contextual selection may filter or reorder that set; it cannot silently introduce arbitrary models.
import { createInference, modelPolicy } from "@smoothie-framework/runtime/model";
// providers and resolveCredential are server-owned bindings.
const inference = createInference({
providers,
resolveCredential,
routing: {
policies: {
production: {
primary: "openai/gpt-5.4-mini",
fallbacks: ["backup/gpt-5.4-mini"],
},
},
},
execution: {
maxAttemptsPerCandidate: 2,
maxTotalAttempts: 4,
deadlineMs: 60_000,
},
});
const selection = modelPolicy("production");Distinguish retry from fallback
Retry repeats a candidate only for retryable failures. Fallback moves to another declared candidate only when the failure is fallback-eligible. Both consume the execution policy's bounded physical attempts.
After streaming exposes a chunk, neither retry nor fallback is automatic. Check returned route/attempt evidence when diagnosing failures; do not infer which provider ran from the original requested policy.
Every physical attempt also passes budget admission when a controller is configured. Avoid stacking hidden provider retries on top of Framework-owned attempts.