Module chat_completion

Chat Completions functionality and route handler.

Structs

Streamer

A streaming response handler.

Enums

ChatCompletionResponder

Represents different types of chat completion responses.

Constants

DEFAULT_CHANNEL_BUFFER_SIZE

Default buffer size for the response channel used in streaming operations.

DEFAULT_KEEP_ALIVE_INTERVAL_MS

Default keep-alive interval for Server-Sent Events (SSE) streams in milliseconds.

Functions

chatcompletions

OpenAI-compatible chat completions endpoint handler.

create_chat_streamer

Creates a SSE streamer for chat completions with optional callbacks.

create_response_channel

Creates a channel for response communication.

get_keep_alive_interval

Gets the keep-alive interval for SSE streams from environment or default.

handle_chat_completion_error

Helper function to handle chat completion errors and logging them.

match_responses

Matches and processes different types of model responses into appropriate chat completion responses.

parse_request

Parses and validates a chat completion request.

process_non_streaming_chat_response

Processes non-streaming chat completion responses.

send_request

Sends a request to the model processing pipeline.

Type Aliases

OnChunkCallback

A callback function that processes streaming response chunks before they are sent to the client.

OnDoneCallback

A callback function that is executed when the streaming response completes.