Skip to content
FastVideo
Esc
↑↓navigate↵open⌘Jpreview
On this page

base

LLM provider protocol + DTOs used by the prompt enhancer.

Third-party users add a new provider by implementing :class:LLMProvider and registering it with a prompt enhancer instance. The shipped providers live in sibling modules (cerebras.py, groq.py) and each is ~100-200 LOC — the provider layer is intentionally thin so the enhancer stays provider-agnostic.

fastvideo.entrypoints.streaming.prompt.providers.base.LLMMessage

LLMMessage(role: Literal['system', 'user', 'assistant'], content: str)
Source
@dataclass
class LLMMessage:
    role: Literal["system", "user", "assistant"]
    content: str

fastvideo.entrypoints.streaming.prompt.providers.base.LLMRequest

LLMRequest(messages: list[LLMMessage], model: str, max_tokens: int | None = None, temperature: float | None = None, timeout_ms: int | None = None)
Source
@dataclass
class LLMRequest:
    messages: list[LLMMessage]
    model: str
    max_tokens: int | None = None
    temperature: float | None = None
    timeout_ms: int | None = None

fastvideo.entrypoints.streaming.prompt.providers.base.LLMResponse

LLMResponse(content: str, provider: str, model: str, latency_ms: float, fallback_used: bool = False)
Source
@dataclass
class LLMResponse:
    content: str
    provider: str
    model: str
    latency_ms: float
    fallback_used: bool = False

fastvideo.entrypoints.streaming.prompt.providers.base.LLMProviderError

LLMProviderError(message: str, *, retryable: bool = True)

Raised when an LLM provider fails a request.

retryable controls whether the enhancer falls back to the next provider. It is settable per-instance so the same exception type can describe retryable transport errors (5xx, 429) and non-retryable client errors (4xx auth/bad-request) without forcing a separate subclass for every status family.

Source
class LLMProviderError(RuntimeError):
    """Raised when an LLM provider fails a request.

``retryable`` controls whether the enhancer falls back to the next
provider. It is settable per-instance so the same exception type
can describe retryable transport errors (5xx, 429) and
non-retryable client errors (4xx auth/bad-request) without forcing
a separate subclass for every status family.
"""

def __init__(self, message: str, *, retryable: bool = True) -> None:
    super().__init__(message)
    self.retryable = retryable</code></pre></details>


fastvideo.entrypoints.streaming.prompt.providers.base.LLMTimeoutError


LLMTimeoutError(message: str)


Raised when an LLM provider times out — always retryable.


Source
class LLMTimeoutError(LLMProviderError):
    """Raised when an LLM provider times out — always retryable."""

def __init__(self, message: str) -&gt; None:
    super().__init__(message, retryable=True)</code></pre></details>


fastvideo.entrypoints.streaming.prompt.providers.base.LLMProvider


LLMProvider()


Provider interface every LLM adapter implements.


Providers are async-first because every built-in implementation
talks to an HTTP API. Synchronous providers can wrap their call in
asyncio.to_thread internally.


Source
@runtime_checkable
class LLMProvider(Protocol):
    """Provider interface every LLM adapter implements.

Providers are async-first because every built-in implementation
talks to an HTTP API. Synchronous providers can wrap their call in
``asyncio.to_thread`` internally.
&quot;&quot;&quot;

name: str

async def complete(self, request: LLMRequest) -&gt; LLMResponse:
    ...</code></pre></details>


fastvideo.entrypoints.streaming.prompt.providers.base.LLMProvider.complete


complete(request: LLMRequest) -> LLMResponse


Source
async def complete(self, request: LLMRequest) -> LLMResponse:
    ...

Was this page helpful?