docs(graph): llm nodes stream internally; node timeout bounds the chat loop, read_timeout only stalls
1 file changed
+12
+12
@@ -743,6 +743,18 @@ Unlike the graph-level `settings.timeout`, a node timeout still gives the
|
||||
graph a chance to recover or report through its `fallback` route.
|
||||
`timeout: 0` (or unset) disables the bound.
|
||||
|
||||
`llm` nodes stream the completion internally, quietly: chunks are
|
||||
accumulated as they arrive and nothing is rendered. This matters for the
|
||||
interplay with the client-level `read_timeout` ([Clients](Clients),
|
||||
default 300s), which fires between successive reads. Because the node
|
||||
receives bytes throughout generation, `read_timeout` only bounds a genuine
|
||||
stall between chunks, while the node `timeout` bounds the node's chat loop
|
||||
(the structured-output extraction pass that follows it is a separate call
|
||||
outside the node `timeout`, bounded only by `read_timeout` stalls). A node `timeout` above 300s therefore
|
||||
genuinely allows generations longer than 300s; without streaming, the first
|
||||
byte would not arrive until the reply was complete and `read_timeout` would
|
||||
cap it regardless.
|
||||
|
||||
### Retries (`max_attempts`)
|
||||
|
||||
`max_attempts` retries the LLM call **only on transient errors**. The
|
||||
|
||||
Reference in new issue
Block a user