thinwrap_llm (thinwrap_llm v1.1.0)

View Source

Chat completions (with streaming) and embeddings across 18 LLM providers — one normalized surface, zero dependencies.

Switching vendor is a change of the config's provider + the input's model only; the input and output shapes are identical across every provider.

Config = #{provider => openai, api_key => <<"sk-...">>},
{ok, #{message := #{content := Text}, usage := #{total_tokens := N}}} =
    thinwrap_llm:complete(Config, #{
        model => <<"gpt-4o-mini">>,
        messages => [#{role => user, content => <<"Say hi in one word.">>}]
    }).

Streaming

stream/4 folds a fun over the normalized deltas in the calling process as they arrive. The library starts no long-lived processes; the default thinwrap_llm_httpc transport runs each streaming call through one helper process that ends with the call (a transport of your own is unaffected):

{ok, Text} = thinwrap_llm:stream(Config, Input,
    fun(Delta, Acc) -> [Acc, maps:get(content_delta, Delta, <<>>)] end, []).

A stream that is cut off before the provider's terminal event, or a 2xx answer that is not text/event-stream, is {error, #{code := provider_unavailable}} rather than ending as if it were complete. Deltas already delivered to the fun stay delivered. usage arrives once, on the terminal delta.

Two kinds of failure

  • Operation failures — a vendor error, a transport failure, request data the vendor would reject — come back as {error, connector_error()}, a plain map whose code you branch on.
  • Misuse — a missing or unknown config/input key, a value of the wrong type, an enum atom outside its set, a provider that lacks the operation — raises error({badarg, Description}) before any HTTP. Folding those into {error, _} would make a bug in the calling code look like bad end-user input or, worse, a retryable outage.

Usage

usage counts every billed token: input_tokens includes cache reads and writes (Anthropic, Bedrock) and tool-use prompt tokens (Gemini), output_tokens includes thinking tokens (Gemini), and total_tokens is the vendor's total, or input + output when it reports none.

Tool loops and provider_metadata

Some providers need opaque state from a response sent back on the next turn: Gemini 3 rejects a replayed function call without its thought signature, and Anthropic keeps its thinking blocks only if they are replayed verbatim. provider_metadata on a tool call and on the assistant message carries that state, keyed by provider (#{<<"gemini">> => ...}, #{<<"anthropic">> => ...}). Treat it as opaque and copy it into the next request together with content and tool_calls. When streaming, it rides on tool_call_delta.provider_metadata and on the terminal delta. A connector reads only its own key, so a history survives a provider switch.

Passthrough

Every input accepts passthrough => #{body => map(), headers => map(), query => map()}, forwarded verbatim and overriding what the connector set (auth included): body is merged shallowly (a key replaces the connector's whole value), headers are merged, query is appended to the URL. Vendor extras on the response are in raw.

Summary

Types

assistant_message()
chat_input()
chat_result()

raw is the decoded vendor body (JSON null is the atom null).

config()

Provider configuration. provider selects the connector; the other keys depend on it

connector_error()

The operation-failure value. code is the normalized classification shared with the sibling libraries; status is the HTTP status when the provider answered (an error inside a 2xx body or a stream frame keeps the status it arrived with); cause is a binary-keyed map with the decoded vendor body under <<"raw">>, plus <<"retryAfter">> (the raw header) and <<"retryAfterSeconds">> when the provider sent a usable Retry-After.

content_part()

One part of a multimodal message.

embeddings_input()
embeddings_result()

embeddings holds one vector per input, in input order.

embeddings_usage()
finish_reason()
message()

One turn. Use content for plain text, or content_parts for multimodal (content_parts wins when non-empty). provider_metadata on an assistant turn is the opaque state copied from the result's message.

operation()
passthrough()
provider()

Provider ids, byte-identical to the sibling libraries' values. The first 15 share one OpenAI-compatible connector; anthropic, bedrock and gemini are native adapters.

reasoning()

Reasoning effort. low | medium | high, or a binary sent as given (an empty binary means "unspecified"). The one deliberate carve-out from the ≥90% baseline rule; providers that cannot map it ignore it.

response_format()

json_schema takes json_schema_name + json_schema.

result(T)
role()
secret()
stream_delta()

One normalized stream delta. Every key is optional and omitted when empty. usage, finish_reason and the message-level provider_metadata arrive on the terminal delta; raw is the decoded vendor event the delta came from.

tool_call()

A tool call. arguments is the JSON-encoded arguments text. provider_metadata is opaque per-provider state keyed by provider id (e.g. Gemini's thought signature); copy it back unchanged when replaying the call.

tool_call_delta()

A streamed tool-call fragment. Concatenate arguments_delta per index (the 0-based tool-call ordinal); id and function_name arrive on the first fragment. Copy provider_metadata onto the assembled tool call.

tool_choice()

function needs function_name.

tool_definition()
usage()

Functions

complete(Config0, Input0)

Run one chat completion.

embed(Config0, Input0)

Create embeddings for input (the OpenAI-float subset: openai, 'azure-openai', openrouter, together, fireworks, mistral, deepinfra, cloudflare, vllm, ollama, lmstudio).

stream(Config0, Input0, Fun, Acc0)

Stream one chat completion, folding Fun(Delta, Acc) over the normalized deltas in the calling process as they arrive. Returns the final accumulator, or the error that ended the stream.

supports/2

Whether Provider implements Operation.

Types

assistant_message()

-type assistant_message() ::
          #{role := assistant,
            content := binary(),
            tool_calls := [tool_call()],
            provider_metadata => map()}.

chat_input()

-type chat_input() ::
          #{model := binary(),
            messages := [message()],
            tools => [tool_definition()],
            tool_choice => tool_choice(),
            response_format => response_format(),
            temperature => number(),
            top_p => number(),
            max_output_tokens => integer(),
            stop => [binary()],
            reasoning => reasoning(),
            passthrough => passthrough()}.

chat_result()

-type chat_result() ::
          #{message := assistant_message(),
            finish_reason := finish_reason(),
            usage := usage(),
            model := binary(),
            raw := term()}.

raw is the decoded vendor body (JSON null is the atom null).

config()

-type config() ::
          #{provider := provider(),
            api_key => secret(),
            base_url => binary(),
            headers => #{binary() | atom() => binary()},
            anthropic_version => binary(),
            default_max_tokens => non_neg_integer(),
            region => binary(),
            access_key_id => secret(),
            secret_access_key => secret(),
            session_token => secret(),
            transport => thinwrap_llm_transport:transport()}.

Provider configuration. provider selects the connector; the other keys depend on it:

  • the 15 OpenAI-compatible providers — api_key (optional for the self-hosted vllm, ollama, lmstudio; sent as Authorization: Bearer, or api-key for 'azure-openai'), base_url (required for 'azure-openai' and cloudflare, which have no fixed public endpoint), headers;
  • anthropic — api_key (required, sent as x-api-key), base_url, anthropic_version (default <<"2023-06-01">>), default_max_tokens, headers;
  • bedrock — region, access_key_id, secret_access_key (required), session_token, base_url, default_max_tokens, headers;
  • gemini — api_key (required, sent as x-goog-api-key), base_url, default_max_tokens, headers.

Optional on every provider: transport (see thinwrap_llm_transport; default {thinwrap_llm_httpc, #{}}).

connector_error()

-type connector_error() ::
          #{code := thinwrap_llm_error:code(),
            message := binary(),
            status => integer(),
            provider_message => binary(),
            cause => term()}.

The operation-failure value. code is the normalized classification shared with the sibling libraries; status is the HTTP status when the provider answered (an error inside a 2xx body or a stream frame keeps the status it arrived with); cause is a binary-keyed map with the decoded vendor body under <<"raw">>, plus <<"retryAfter">> (the raw header) and <<"retryAfterSeconds">> when the provider sent a usable Retry-After.

content_part()

-type content_part() ::
          #{type := text, text => binary()} |
          #{type := image, base64 => binary(), media_type => binary()}.

One part of a multimodal message.

embeddings_input()

-type embeddings_input() ::
          #{model := binary(),
            input := [binary()],
            dimensions => integer(),
            passthrough => passthrough()}.

embeddings_result()

-type embeddings_result() ::
          #{embeddings := [[float()]], usage := embeddings_usage(), model := binary(), raw := term()}.

embeddings holds one vector per input, in input order.

embeddings_usage()

-type embeddings_usage() :: #{input_tokens := integer(), total_tokens := integer()}.

finish_reason()

-type finish_reason() :: stop | length | tool_calls | content_filter | unknown.

message()

-type message() ::
          #{role := role(),
            content => binary(),
            content_parts => [content_part()],
            tool_calls => [tool_call()],
            tool_call_id => binary(),
            name => binary(),
            provider_metadata => map()}.

One turn. Use content for plain text, or content_parts for multimodal (content_parts wins when non-empty). provider_metadata on an assistant turn is the opaque state copied from the result's message.

operation()

-type operation() :: complete | stream | embed.

passthrough()

-type passthrough() ::
          #{body => map(),
            headers => #{binary() | atom() => binary()},
            query => #{binary() | atom() => binary() | atom() | number()}}.

provider()

-type provider() ::
          openai | 'azure-openai' | openrouter | groq | together | fireworks | deepseek | xai |
          mistral | perplexity | deepinfra | cloudflare | vllm | ollama | lmstudio | anthropic |
          bedrock | gemini.

Provider ids, byte-identical to the sibling libraries' values. The first 15 share one OpenAI-compatible connector; anthropic, bedrock and gemini are native adapters.

reasoning()

-type reasoning() :: #{effort := low | medium | high | binary()}.

Reasoning effort. low | medium | high, or a binary sent as given (an empty binary means "unspecified"). The one deliberate carve-out from the ≥90% baseline rule; providers that cannot map it ignore it.

response_format()

-type response_format() ::
          #{type := text | json_object | json_schema,
            json_schema_name => binary(),
            json_schema => map()}.

json_schema takes json_schema_name + json_schema.

result(T)

-type result(T) :: {ok, T} | {error, connector_error()}.

role()

-type role() :: system | user | assistant | tool.

secret()

-type secret() :: binary() | fun(() -> binary()).

stream_delta()

-type stream_delta() ::
          #{content_delta => binary(),
            tool_call_delta => tool_call_delta(),
            finish_reason => finish_reason(),
            usage => usage(),
            raw => term(),
            provider_metadata => map()}.

One normalized stream delta. Every key is optional and omitted when empty. usage, finish_reason and the message-level provider_metadata arrive on the terminal delta; raw is the decoded vendor event the delta came from.

tool_call()

-type tool_call() ::
          #{id := binary(), name := binary(), arguments := binary(), provider_metadata => map()}.

A tool call. arguments is the JSON-encoded arguments text. provider_metadata is opaque per-provider state keyed by provider id (e.g. Gemini's thought signature); copy it back unchanged when replaying the call.

tool_call_delta()

-type tool_call_delta() ::
          #{index := integer(),
            id => binary(),
            function_name => binary(),
            arguments_delta => binary(),
            provider_metadata => map()}.

A streamed tool-call fragment. Concatenate arguments_delta per index (the 0-based tool-call ordinal); id and function_name arrive on the first fragment. Copy provider_metadata onto the assembled tool call.

tool_choice()

-type tool_choice() :: #{mode := auto | none | required | function, function_name => binary()}.

function needs function_name.

tool_definition()

-type tool_definition() :: #{name := binary(), description => binary(), parameters => map()}.

usage()

-type usage() :: #{input_tokens := integer(), output_tokens := integer(), total_tokens := integer()}.

Functions

complete(Config0, Input0)

-spec complete(config(), chat_input()) -> result(chat_result()).

Run one chat completion.

embed(Config0, Input0)

-spec embed(config(), embeddings_input()) -> result(embeddings_result()).

Create embeddings for input (the OpenAI-float subset: openai, 'azure-openai', openrouter, together, fireworks, mistral, deepinfra, cloudflare, vllm, ollama, lmstudio).

stream(Config0, Input0, Fun, Acc0)

-spec stream(config(), chat_input(), fun((stream_delta(), Acc) -> Acc), Acc) ->
                {ok, Acc} | {error, connector_error()}.

Stream one chat completion, folding Fun(Delta, Acc) over the normalized deltas in the calling process as they arrive. Returns the final accumulator, or the error that ended the stream.

supports/2

-spec supports(provider(), operation()) -> boolean().

Whether Provider implements Operation.