Phase 7 — Multi-modal Support

Goal

The agent can both receive and send media, and inline button presses round-trip back to the agent. After this phase, the bot is no longer text-only.

Scope / Deliverables

Inbound:

• Photo updates → download the largest size via Telegram getFile, store metadata (file_id, dimensions, optional bytes path) in chat_messages.payload, forward normalized event to the agent
• Document updates → same pattern; respect a size cap (e.g. 20MB) before downloading
• Callback queries (inline button presses) → normalized event includes callbackData, surfaced to the agent; auto-ack with answerCallbackQuery to clear the spinner
• Decision: do we persist binary bytes or just file_id? Phase-default = file_id only (Telegram retains files), with an opt-in flag per tool to download.

Outbound:

• send_image supports {url}, {fileId}, and {bytesBase64, mimeType} sources
• send_document (same source variants)
• send_buttons already exists from Phase 2; extend to multi-row layouts and URL buttons

Automated Unit Testing

• inbound.photo.test.ts —
  • photo update with multiple sizes → normalized event uses the largest size’s file_id and dimensions
  • chat_messages.payload archived row contains file_id, width, height, and the raw normalized payload
  • by default no binary bytes are downloaded; with the opt-in flag enabled, the test exercises the download path (mock getFile)
• inbound.document.test.ts —
  • document under the size cap → normalized event with file_id, mime_type, file_size
  • document over the cap (e.g. 20MB+1) → event still surfaced to the agent (with metadata) but binary download skipped, audit/log entry noting the skip
• callback.ack.test.ts —
  • inbound callback query → normalized event includes callbackData, chatId, userId, messageId
  • answerCallbackQuery is called exactly once for each callback (auto-ack), even if the agent forwarding step later fails
• outbound.image.test.ts — send_image source-variant handling:
  • {url: string} → telegraf called with { url }
  • {fileId: string} → telegraf called with the file_id
  • {bytesBase64, mimeType} → telegraf called with a Buffer/Uint8Array constructed from the base64; Zod rejects missing mimeType
  • any other shape → INVALID_INPUT
  • all variants archive a chat_messages row with kind='photo' and the source recorded in payload
• outbound.document.test.ts — same source-variant matrix for send_document.
• outbound.buttons.test.ts —
  • multi-row layouts pass through correctly (2D Button[][])
  • URL buttons ({label, url}) and callback buttons ({label, callbackData}) coexist in the same row
  • mixed row with invalid button shape → INVALID_INPUT

Definition of Done

• Sending a photo to the bot → agent receives a normalized event with file_id and dimensions.
• Agent calls send_image({bytesBase64}) → image appears in the chat.
• Tapping an inline button on a bot message → agent receives a callback event with the configured callbackData.