nc-talk-ai/docs/pseudo_streaming.md

Pseudo Streaming (Chunked Replies) in Talk AI

This document explains how Talk AI simulates “streaming” replies in Nextcloud Talk by emitting partial chunks of the assistant response while the LLM stream is still in progress.

High-level idea

• The LLM API is called with stream=true (SSE / chunked HTTP response).
• Incoming token deltas are buffered on the server.
• The buffer is flushed periodically (paragraph / sentence boundaries) as separate Talk messages.
• Only the first chunk is sent as a reply to the original user message (replyTo=<messageId>). All subsequent chunks are sent without replyTo so they don’t all appear as nested replies.

Where this is implemented

• lib/Service/LLMClient.php::streamChatCompletion()
  • Opens the streaming connection to the OpenAI-compatible endpoint and calls a callback for each delta chunk (delta.content, and optionally delta.tool_calls).
• lib/Service/BotService.php::processMessage()
  • When no tools are used, it converts streaming delta.content into readable chunks via an onProgress(string $partial) callback.
• lib/Service/AgentExecutor.php::run()
  • When tools are enabled, it also streams via LLMClient::streamChatCompletion() and uses the same buffering/flush strategy.
  • Additionally, it can emit small progress messages (e.g. “🔧 Using tool(s): …”) via the streaming callback.
• lib/Webhook/TalkHandler.php::processNormalMessage()
  • Wires onProgress(...) to Talk by calling sendReplyToTalk(...) for every emitted chunk.
  • Manages replyTo behavior (only first chunk replies to the user message).
  • Filters model “thinking tokens” and optionally sends a placeholder while the model is thinking.

Chunking / flush strategy (server-side)

Both BotService (no-tools path) and AgentExecutor (tools path) implement the same strategy:

1. Append each incoming delta['content'] to a $buffer.
2. Flush when:
   • a paragraph boundary is detected (\n\n), or
   • enough time passed (3 seconds) and enough text accumulated (>100 chars) → then flush at the next sentence boundary (best-effort).
3. After the stream ends, flush any remaining buffer.

This avoids token-by-token “spam” and produces readable chunks.

Delivery to Talk + replyTo behavior

In TalkHandler::processNormalMessage(), the streaming callback:

• filters out “thinking tokens” (for models that emit them),
• sends each non-empty chunk to Talk via sendReplyToTalk($roomToken, $chunk, $replyTo),
• uses replyTo=<original message id> only for the first streamed chunk, then switches to replyTo=0 for all later chunks.

This prevents Talk from rendering every chunk as a reply to the original user message.

Also note:

• If no chunks were streamed ($alreadySent === false), the handler sends a single final message (normal non-streamed behavior).
• If chunks were streamed, the handler skips sending the final full response to avoid duplicate content.

Why this is “pseudo” streaming

• Nextcloud Talk bot messages are sent as discrete messages (no server-side “edit this message” API).
• The app therefore simulates streaming by sending multiple messages as the response arrives.
• Chunking happens at the PHP application level; Talk clients are not consuming the LLM stream directly.

Troubleshooting

• No streaming visible:
  • Verify your LLM provider supports stream=true.
  • Check nextcloud.log for EducAI: entries (from LLMClient, AgentExecutor, TalkHandler).
• Too many tiny chunks:
  • Adjust the flush strategy in BotService / AgentExecutor (paragraph vs time-based flush thresholds).