prompt

Prompt pipeline for the streaming server.

• :mod:providers — LLM backend abstraction + built-in adapters
• :mod:enhancer — provider-agnostic enhance / auto-extend / rewrite operations on top of the provider layer

All of this is optional; the streaming server runs fine without it (PR 7.5's skeleton never invokes the enhancer). When the operator enables ServeConfig.streaming.prompt.enabled, the server routes each session_init_v2 curated prompt through enhance before the first segment.

fastvideo.entrypoints.streaming.prompt.PromptEnhancer

PromptEnhancer(*, providers: Sequence[LLMProvider], model: str, timeout_ms: int = 20000, temperature: float = 0.7, max_tokens: int | None = 256, system_prompt_dir: str | None = None)

Orchestrates prompt operations across a priority-ordered provider list with structured fallback + hot-reloadable system prompts.

Usage::

enhancer = PromptEnhancer(
    providers=[CerebrasProvider(), GroqProvider()],
    model="gpt-oss-120b",
    system_prompt_dir="/etc/fastvideo/prompts",
)
response = await enhancer.enhance("a fox running through snow")

fastvideo.entrypoints.streaming.prompt.PromptEnhancer.register_provider

register_provider(provider: LLMProvider, *, priority: int = -1) -> None

Insert an additional provider. priority=0 makes it primary; priority=-1 (default) appends as a fallback.

fastvideo.entrypoints.streaming.prompt.PromptEnhancer.reload_system_prompts

reload_system_prompts() -> None

Re-read the system prompt files from system_prompt_dir.

The streaming server exposes this via a management endpoint so operators can iterate on prompt templates without restarting workers.

fastvideo.entrypoints.streaming.prompt.PromptEnhancer.enhance

enhance(prompt: str) -> LLMResponse

fastvideo.entrypoints.streaming.prompt.PromptEnhancer.auto_extend

auto_extend(prior_prompts: Sequence[str]) -> LLMResponse

fastvideo.entrypoints.streaming.prompt.PromptEnhancer.rewrite

rewrite(seed_prompt: str) -> LLMResponse

fastvideo.entrypoints.streaming.prompt.PromptOperation

PromptOperation()

fastvideo.entrypoints.streaming.prompt.LLMMessage

LLMMessage(role: Literal['system', 'user', 'assistant'], content: str)

fastvideo.entrypoints.streaming.prompt.LLMProvider

LLMProvider()

Provider interface every LLM adapter implements.

Providers are async-first because every built-in implementation talks to an HTTP API. Synchronous providers can wrap their call in asyncio.to_thread internally.

fastvideo.entrypoints.streaming.prompt.LLMProvider.complete

complete(request: LLMRequest) -> LLMResponse

fastvideo.entrypoints.streaming.prompt.LLMProviderError

LLMProviderError(message: str, *, retryable: bool = True)

Raised when an LLM provider fails a request.

retryable controls whether the enhancer falls back to the next provider. It is settable per-instance so the same exception type can describe retryable transport errors (5xx, 429) and non-retryable client errors (4xx auth/bad-request) without forcing a separate subclass for every status family.

fastvideo.entrypoints.streaming.prompt.LLMRequest

LLMRequest(messages: list[LLMMessage], model: str, max_tokens: int | None = None, temperature: float | None = None, timeout_ms: int | None = None)

fastvideo.entrypoints.streaming.prompt.LLMResponse

LLMResponse(content: str, provider: str, model: str, latency_ms: float, fallback_used: bool = False)

fastvideo.entrypoints.streaming.prompt.LLMTimeoutError

LLMTimeoutError(message: str)

Raised when an LLM provider times out — always retryable.