mirror of
https://github.com/crowetic/nc-talk-ai.git
synced 2026-09-07 07:32:31 +00:00
Pascal Kienast
0739d3da6a
Talk AI is a multi-bot AI assistant manager for Nextcloud Talk: per-bot prompts and models, agentic tool calling (MCP + built-in tools), RAG over Nextcloud files, room-document search, vision and speech-to-text attachments, persistent bot wikis, approval workflows, rate limiting, and multi-provider LLM support (any OpenAI-compatible endpoint). Developed within EDUC - the European Digital UniverCity (https://educalliance.eu), where it runs as the 'EDUC AI' assistant on the alliance-wide Nextcloud portal. This public repository is the upstream point of truth; deployment-specific tools plug in via the tool-provider extension point (docs/TOOL_PROVIDERS.md). License: AGPL-3.0-or-later.
67 lines
3.6 KiB
Markdown
67 lines
3.6 KiB
Markdown
# Pseudo Streaming (Chunked Replies) in Talk AI
|
||
|
||
This document explains how Talk AI simulates “streaming” replies in Nextcloud Talk by emitting partial chunks of the assistant response while the LLM stream is still in progress.
|
||
|
||
## High-level idea
|
||
|
||
- The LLM API is called with `stream=true` (SSE / chunked HTTP response).
|
||
- Incoming token deltas are buffered on the server.
|
||
- The buffer is flushed periodically (paragraph / sentence boundaries) as **separate Talk messages**.
|
||
- Only the **first** chunk is sent as a reply to the original user message (`replyTo=<messageId>`). All subsequent chunks are sent without `replyTo` so they don’t all appear as nested replies.
|
||
|
||
## Where this is implemented
|
||
|
||
- `lib/Service/LLMClient.php::streamChatCompletion()`
|
||
- Opens the streaming connection to the OpenAI-compatible endpoint and calls a callback for each `delta` chunk (`delta.content`, and optionally `delta.tool_calls`).
|
||
- `lib/Service/BotService.php::processMessage()`
|
||
- When **no tools** are used, it converts streaming `delta.content` into readable chunks via an `onProgress(string $partial)` callback.
|
||
- `lib/Service/AgentExecutor.php::run()`
|
||
- When **tools are enabled**, it also streams via `LLMClient::streamChatCompletion()` and uses the same buffering/flush strategy.
|
||
- Additionally, it can emit small progress messages (e.g. “🔧 _Using tool(s): …_”) via the streaming callback.
|
||
- `lib/Webhook/TalkHandler.php::processNormalMessage()`
|
||
- Wires `onProgress(...)` to Talk by calling `sendReplyToTalk(...)` for every emitted chunk.
|
||
- Manages `replyTo` behavior (only first chunk replies to the user message).
|
||
- Filters model “thinking tokens” and optionally sends a placeholder while the model is thinking.
|
||
|
||
## Chunking / flush strategy (server-side)
|
||
|
||
Both `BotService` (no-tools path) and `AgentExecutor` (tools path) implement the same strategy:
|
||
|
||
1. Append each incoming `delta['content']` to a `$buffer`.
|
||
2. Flush when:
|
||
- a paragraph boundary is detected (`\n\n`), **or**
|
||
- enough time passed (3 seconds) and enough text accumulated (>100 chars) → then flush at the next sentence boundary (best-effort).
|
||
3. After the stream ends, flush any remaining buffer.
|
||
|
||
This avoids token-by-token “spam” and produces readable chunks.
|
||
|
||
## Delivery to Talk + `replyTo` behavior
|
||
|
||
In `TalkHandler::processNormalMessage()`, the streaming callback:
|
||
|
||
- filters out “thinking tokens” (for models that emit them),
|
||
- sends each non-empty chunk to Talk via `sendReplyToTalk($roomToken, $chunk, $replyTo)`,
|
||
- uses `replyTo=<original message id>` **only for the first streamed chunk**, then switches to `replyTo=0` for all later chunks.
|
||
|
||
This prevents Talk from rendering every chunk as a reply to the original user message.
|
||
|
||
Also note:
|
||
|
||
- If **no** chunks were streamed (`$alreadySent === false`), the handler sends a single final message (normal non-streamed behavior).
|
||
- If chunks **were** streamed, the handler skips sending the final full response to avoid duplicate content.
|
||
|
||
## Why this is “pseudo” streaming
|
||
|
||
- Nextcloud Talk bot messages are sent as discrete messages (no server-side “edit this message” API).
|
||
- The app therefore simulates streaming by sending multiple messages as the response arrives.
|
||
- Chunking happens at the PHP application level; Talk clients are not consuming the LLM stream directly.
|
||
|
||
## Troubleshooting
|
||
|
||
- **No streaming visible**:
|
||
- Verify your LLM provider supports `stream=true`.
|
||
- Check `nextcloud.log` for `EducAI:` entries (from `LLMClient`, `AgentExecutor`, `TalkHandler`).
|
||
- **Too many tiny chunks**:
|
||
- Adjust the flush strategy in `BotService` / `AgentExecutor` (paragraph vs time-based flush thresholds).
|
||
|