thinwrap_llm_httpc (thinwrap_llm v1.1.0)

View Source

Default transport, built on OTP httpc. Implements both request/2 (buffered) and stream/4 (incremental, used by thinwrap_llm:stream/4).

Options (all optional), passed as transport => {thinwrap_llm_httpc, Opts}:

  • timeout — milliseconds (default 600000, ten minutes). For request/2 it is the whole-request budget: a non-streaming reasoning call sends nothing until the model has finished, so a shorter default would cut long answers off (600 s matches the official OpenAI and Anthropic SDK defaults). For stream/4 it is the idle budget between two chunks, so a long stream that keeps producing tokens is never cut off. httpc itself defaults to infinity, so a blackholed connection would otherwise hang the calling process forever.
  • connect_timeout — milliseconds (default: timeout).
  • profile — an httpc profile you started yourself (default: default). The library never starts a profile or any long-lived process (see stream/4 below for the one short-lived helper).
  • ssl — a complete list of ssl client options. Leave it out: since OTP 26 httpc verifies the peer and its host name against the OS trust store by default, and passing any ssl list replaces those defaults rather than adding to them.

The adapter is HTTPS-only: an http:// (or any other) URL is refused with an invalid_request error instead of being sent in cleartext. The http:// defaults of the local servers (vLLM, Ollama, LM Studio) therefore need your own transport. Redirects are never followed.

stream/4 runs httpc in asynchronous streaming mode ({sync, false}, {stream, self}) inside one helper process per call, spawned and monitored by the caller, which forwards each event to the caller; the fold itself runs in the calling process. On every exit path — end of body, a halted fold, an idle timeout, a failure, or an exception raised by the fold — the helper cancels the request and the caller waits for the helper's 'DOWN', dropping whatever it forwarded meanwhile. Because messages between two processes arrive in send order, nothing can follow that 'DOWN': the caller's mailbox is left exactly as it was. The helper also monitors the caller and cancels the request if the caller dies, so it never outlives the call. An exception from the fold propagates unchanged after that cleanup.

First body bytes may be held back (an OTP httpc limitation). When the first body bytes arrive in the same socket read as the response headers, httpc decodes them but does not stream them: in OTP's httpc_handler module, handle_http_body keeps an incomplete chunked body (the {Module, Function, Args} continuation returned by http_chunk's decode) without streaming what it already decoded, and the "steal decoded chunks" path that does stream partial bodies runs only for later reads (the do_handle_info clause for socket data with a pending continuation). Checked in inets 9.2 (OTP 27.0) and 9.8.1 (OTP 29.1); the {stream, {self, once}} mode shares the same path. In practice the first SSE event of a response can therefore reach the fold only together with the next read, and the idle timeout keeps running meanwhile. A server that sends its headers, one event, and then stays silent longer than timeout ends in {error, timeout, _} even though that event was sent. Use your own transport if this matters.

Summary

Functions

Send one request with httpc. See the module doc for Opts.

Send one request and fold Fun over the response as it arrives. See thinwrap_llm_transport for the event protocol and the module doc for Opts.

Functions

request/2

-spec request(thinwrap_llm_transport:request(), map()) ->
                 {ok, thinwrap_llm_transport:response()} | {error, term()}.

Send one request with httpc. See the module doc for Opts.

stream/4

-spec stream(thinwrap_llm_transport:request(), map(), thinwrap_llm_transport:stream_fun(), term()) ->
                {ok, term()} | {error, term(), term()}.

Send one request and fold Fun over the response as it arrives. See thinwrap_llm_transport for the event protocol and the module doc for Opts.