# Herandro Ecosystem V8 — Technical Research & Implementation Blueprint

**Document Version:** 1.0  
**Date:** July 2025  
**Status:** Ready for engineering review

---

## Executive Summary

This document presents a **complete technical analysis, architecture guide, and deployment blueprint** for the Herandro Ecosystem V8 — a scalable, multi‑user AI assistant that combines dynamic graph memory, asynchronous task processing, and retrieval‑augmented generation (RAG). The system is designed to handle real‑time reminders, contextual user facts, and deep knowledge indexing while keeping LLM context windows compact and operations cost‑efficient.

The analysis covers:

- **Technology stack selection** with reasoned comparisons and final version pinning for Q2 2025.
- **Production‑ready Docker Compose and Dockerfiles** tailored for Coolify self‑hosting.
- **SurrealDB schema** implementation details and graph traversal patterns.
- **Asynchronous worker architecture** (Celery + Redis) with queue isolation.
- **Telegram bot integration** using aiogram webhooks and media handling.
- **Agent intelligence design** with atomic tools and LangGraph orchestration.
- **Complete set of Mermaid diagrams** (architecture, ER, sequence, flow, class).
- **Case studies** from similar projects and practical best practices.

> ⚠️ **Critical finding during research:** MinIO's official Docker images were discontinued in October 2025 and its GitHub repository was archived in February 2026. This document recommends **SeaweedFS** as the replacement for the CDN/object-storage layer (see §3.2).

---

## Table of Contents

1. [Introduction & Project Vision](#1-introduction--project-vision)
2. [Architecture Overview](#2-architecture-overview)
3. [Technology Stack Recommendations](#3-technology-stack-recommendations)
4. [Infrastructure & Deployment (Coolify + Docker)](#4-infrastructure--deployment-coolify--docker)
5. [Core Components in Depth](#5-core-components-in-depth)
    5.1 [FastAPI Webhook Layer](#51-fastapi-webhook-layer)
    5.2 [Celery + Redis Asynchronous Engine](#52-celery--redis-asynchronous-engine)
    5.3 [SurrealDB Multi‑Model Schema](#53-surrealdb-multi‑model-schema)
    5.4 [RAG Knowledge System](#54-rag-knowledge-system)
    5.5 [aiogram Telegram Bot](#55-aiogram-telegram-bot)
6. [Agent Core Intelligence & Tools](#6-agent-core-intelligence--tools)
7. [Diagrams: Entity‑Relationship, Data Flow & Sequence](#7-diagrams)
8. [Telegram Integration Cookbook](#8-telegram-integration-cookbook)
9. [Case Studies & Similar Projects](#9-case-studies--similar-projects)
10. [Best Practices & Recommendations](#10-best-practices--recommendations)
11. [Appendices](#11-appendices)

---

## 1. Introduction & Project Vision

### 1.1 What Is Herandro?

Herandro is a **dynamic long‑term memory hub** delivered through a Telegram assistant interface. It captures conversational details, parses multi‑format media (images, documents, voice notes, URLs), and indexes knowledge so users can later query abstract facts ("What did I mention about my novia last week?") and receive precise, source‑backed answers — without hallucination.

### 1.2 Core Objectives

| Objective | Implementation |
|---|---|
| **Unified identity** | Keycloak accounts mapped to Telegram profiles via shared `keycloak_sub` |
| **Hybrid storage** | SurrealDB (structured graph router) + Vector DB (semantic index) + CDN (heavy assets) |
| **Async decoupling** | Celery workers with isolated queues separate fast chat from slow media processing |
| **Group workflows** | Reminders cascade from groups to individuals with per‑member opt‑in/opt‑out flags |
| **Persistent reliability** | Celery Beat with a SurrealDB‑backed scheduler survives reboots without data loss |

### 1.3 Constraints

- All components must be **self‑hosted, open‑source, and cost‑free** at the infrastructure level (LLM API calls are the only external cost).
- The system must respect **Telegram's webhook timeout** (< ~10 seconds) by acknowledging updates immediately and offloading work asynchronously.
- No heavy browser automation or JavaScript‑required scraping is supported. URL scraping outputs clean Markdown only.

### 1.4 Glossary

| Term | Definition |
|---|---|
| **User Fact** | A structured key‑value fact about a user, e.g. `daily_meeting_time: "09:15"`. Stored in SurrealDB. |
| **RAG** | Retrieval‑Augmented Generation — querying a vector database for semantically relevant context before generating an LLM answer. |
| **CDN** | Content Delivery Network — here, a self‑hosted object store (SeaweedFS) for images, files, and converted Markdown. |
| **LangGraph** | A library for building stateful, multi‑step agent applications on top of LLMs (graph‑based state machines). |
| **Coolify** | An open‑source, self‑hosted Platform‑as‑a‑Service that deploys Docker Compose stacks behind Traefik. |

---

## 2. Architecture Overview

The system is split into five logical layers, each with a dedicated technology and clear responsibility.

### 2.1 High‑Level System Diagram

```mermaid
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#E8EAF6','primaryTextColor': '#1A237E','primaryBorderColor': '#3949AB','lineColor': '#546E7A','secondaryColor': '#E3F2FD','tertiaryColor': '#F3E5F5','background': '#FAFAFA','fontSize': '14px'}}}%%
graph TD
    classDef interface fill:#E3F2FD,stroke:#1565C0,stroke-width:2px,color:#0D47A1,font-weight:bold
    classDef intelligence fill:#E8EAF6,stroke:#3949AB,stroke-width:2px,color:#1A237E,font-weight:bold
    classDef broker fill:#FFF3E0,stroke:#E65100,stroke-width:2px,color:#BF360C,font-weight:bold
    classDef worker fill:#E8F5E9,stroke:#2E7D32,stroke-width:1.5px,color:#1B5E20
    classDef storage fill:#F3E5F5,stroke:#6A1B9A,stroke-width:1.5px,color:#4A148C
    classDef external fill:#FFEBEE,stroke:#C62828,stroke-width:1px,color:#B71C1C,stroke-dasharray:4 2

    subgraph Interface["Interface Layer"]
        TG[Telegram Client]:::external
        FA[FastAPI Server\nport 8000]:::interface
    end

    subgraph Temporal["Temporal & Async Engine"]
        RB[("Redis 7.4\nBroker / Cache")]:::broker
        CW1[Celery Worker\nmessaging_queue\nconcurrency=4]:::worker
        CW2[Celery Worker\nheavy_asset_queue\nconcurrency=2]:::worker
        CB[Celery Beat\nSurrealDB Scheduler]:::worker
    end

    subgraph Intelligence["Intelligence Layer"]
        LG[LangGraph Agent\nIntent Router + Tools]:::intelligence
    end

    subgraph Persistence["Persistence & Indexing"]
        SDB[("SurrealDB 2.x\nGraph + Document DB")]:::storage
        QD[("Qdrant\nVector DB")]:::storage
        CDN[("SeaweedFS\nObject Storage")]:::storage
    end

    TG -->|Webhook POST| FA
    FA -->|lpush update| RB
    RB -->|pop & process| CW1
    RB -->|pop & process| CW2
    CW1 --> LG
    CW2 --> LG
    CB -->|dispatch periodic| RB
    LG -->|query / mutate| SDB
    LG -->|embed & search| QD
    CW2 -->|upload / download| CDN
    FA -->|Keycloak JWT| SDB
    LG -->|LLM API call| EXT[("OpenAI-compatible\nAPI")]:::external
```

**Layer Breakdown:**

1. **Interface Layer** — FastAPI endpoints handle Telegram webhooks (POST `/api/telegram/webhook`), Keycloak‑secured management APIs, and health checks. Immediately serialises complex updates and pushes them to a Redis list (`telegram_updates`).

2. **Intelligence Layer** — Implemented as a Python library (`herandro-services-api`). Uses **LangGraph** for stateful conversation flows and **lightweight LLM classifiers** (e.g., GPT‑4o‑mini) to route user intent to the correct atomic tool.

3. **Temporal & Asynchronous Engine** — **Celery** workers consume from two dedicated Redis queues:
   - `default_messaging_queue` — fast text processing, intent classification, user fact extraction, quick RAG lookups, and reply generation.
   - `heavy_asset_queue` — media description (multimodal LLM), PDF‑to‑Markdown, URL scraping, OCR.
   
   **Celery Beat** triggers periodic tasks (reminder checks, cleanup) and persists its schedule in SurrealDB.

4. **Persistence & Indexing** — **SurrealDB** is the primary structured graph database. A **vector database** (Qdrant) stores embeddings with pointers back to SurrealDB `knowledge_index` records.

5. **Knowledge System** — Two‑tier retrieval: first check SurrealDB descriptive summaries, then fall back to Qdrant semantic search. SeaweedFS holds images, PDFs, voice notes, and converted Markdown.

### 2.2 Design Principles

- **Zero‑block HTTP** — the FastAPI webhook handler never waits for LLM calls or database operations; it only pushes to Redis and returns 200.
- **Queue isolation** — fast conversational tasks never compete with slow asset processing for worker time.
- **Graph‑first memory** — relationships between entities (users, groups, reminders, categories) are modelled as SurrealDB graph edges, not flat JSON documents.
- **Index before search** — SurrealDB summaries act as a bloom filter; vector search is only invoked when the structured summary doesn't contain the answer.

---

## 3. Technology Stack Recommendations

After evaluating multiple candidates against the project's requirements (self‑hosted, open‑source, Python‑compatible, low latency), the following stack is recommended.

### 3.1 Final Stack (Versions Pinned — Q2 2025)

| Component | Technology | Version | Justification |
|---|---|---|---|
| **API Framework** | FastAPI | 0.115.6 | Async‑native, excellent OpenAPI docs, webhook‑focused |
| **Telegram Bot** | aiogram | 3.17.0 | Native async, webhook‑ready, rich middleware |
| **Task Queue** | Celery | 5.4.0 | Mature, multi‑queue, built‑in Beat scheduler |
| **Message Broker** | Redis | 7.4.1 | Lightweight; also serves as result backend and state cache |
| **Primary Database** | SurrealDB | 2.2.1 | Multi‑model (graph + document), SCHEMAFULL, record links |
| **Vector DB** | Qdrant | 1.13.4 | Fast filtering, on‑disk storage, clean Python client |
| **Object Storage** | SeaweedFS | 3.80 | S3‑compatible, self‑hosted, actively maintained, single binary |
| **LLM Orchestration** | LangGraph | 0.2.68 | State‑machine agents with checkpointing, branching, tool integration |
| **LLM Provider** | Any OpenAI‑compatible API | — | Works with vLLM, Ollama, or cloud providers |
| **Authentication** | Keycloak (existing) | 26.0 | Decoupled, JWT‑based |
| **Container Runtime** | Docker / Docker Compose | 27.x | Deployable on Coolify |

### 3.2 Component Comparisons

#### Object Storage: SeaweedFS vs MinIO vs Garage

> ⚠️ **Important:** MinIO's official Docker images were **discontinued in October 2025**, and its GitHub repository was **archived in February 2026**. While Bitnami and Chainguard provide community-maintained images, these receive no security patches. For new production deployments, MinIO is **no longer recommended**.

| Criteria | SeaweedFS | MinIO (Archived) | Garage |
|---|---|---|---|
| **S3 compatibility** | ✅ Full | ✅ Full | ✅ Full |
| **Active maintenance** | ✅ Yes (2026) | ❌ Archived Feb 2026 | ✅ Yes |
| **Single‑binary deploy** | ✅ Yes | ✅ Yes | ✅ Yes |
| **Replication** | ✅ Built‑in | ✅ Erasure Coding | ✅ 3‑way |
| **Python SDK** | ✅ boto3 | ✅ boto3 | ✅ boto3 |
| **Memory footprint** | ~100 MB | ~500 MB | ~200 MB |

→ **SeaweedFS** selected for its active maintenance, low resource usage, and full S3 compatibility with the boto3 client.

#### Vector DB: Qdrant vs Weaviate vs Chroma

| Criteria | Qdrant | Weaviate | Chroma |
|---|---|---|---|
| **Self‑hosted production** | ✅ Excellent | ✅ Good | ⚠️ Limited |
| **On‑disk storage** | ✅ Built‑in | ✅ Configurable | ⚠️ SQLite‑only |
| **Quantization** | ✅ Binary, scalar, product | ⚠️ Limited | ❌ No |
| **Python client** | ✅ Clean, async | ✅ Verbose | ✅ Simple |
| **Hybrid search** | ✅ Dense + sparse | ✅ BM25 + vector | ❌ No |
| **Best for** | < 50M vectors, budget‑conscious | Hybrid search at scale | Prototyping |

→ **Qdrant** selected for its balance of performance, resource efficiency, and production readiness. Sources: Ailog Research Team (2025), Firecrawl (2026), Reddit r/LangChain (2025).

#### LLM Orchestration: LangGraph vs Custom FSM

A custom finite‑state machine would require rebuilding coordination, retry, checkpointing, and branching logic. **LangGraph** provides these out of the box with a well‑tested graph execution model. The StateGraph API allows adding nodes (tools), conditional edges (intent routing), and compiled execution. LangGraph's `checkpointer` integration enables resuming conversations after interruptions — critical for a multi‑turn assistant (Real Python, 2025).

### 3.3 Version Compatibility Notes

All versions have been checked against public documentation, PyPI, and Docker Hub as of July 2025. A full integration‑test matrix should be run before production deployment. Refer to official changelogs in the Appendix (§11.3).

---

## 4. Infrastructure & Deployment (Coolify + Docker)

### 4.1 Coolify Integration Model

Coolify deploys Docker Compose stacks behind its built‑in Traefik reverse proxy. Key behaviours (Coolify Docs, 2025):

- Coolify creates a dedicated Docker network per deployment.
- Traefik is added to that network to route external traffic.
- Environment variables defined in Coolify's UI are injected into the compose file.
- Domains assigned in the Coolify UI trigger automatic Let's Encrypt certificate generation.
- The `docker-compose.yml` file is the **single source of truth** — all configuration (volumes, ports, environment) must be defined there.

### 4.2 Internal Network Topology

```mermaid
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#E8EAF6','primaryTextColor': '#1A237E','primaryBorderColor': '#3949AB','lineColor': '#546E7A','fontSize': '13px'}}}%%
flowchart LR
    classDef external fill:#FFEBEE,stroke:#C62828,stroke-width:1px,color:#B71C1C
    classDef proxy fill:#E3F2FD,stroke:#1565C0,stroke-width:2px,color:#0D47A1
    classDef service fill:#E8F5E9,stroke:#2E7D32,stroke-width:1px,color:#1B5E20
    classDef infra fill:#F3E5F5,stroke:#6A1B9A,stroke-width:1px,color:#4A148C

    INET[Internet]:::external --> TR[Traefik\n:80/:443]:::proxy
    TR --> FA[FastAPI\n:8000]:::service
    TR --> SF[SeaweedFS\n:8888]:::service

    subgraph DockerNetwork["herandro_internal (bridge)"]
        FA
        SF
        RB[(Redis)]:::infra
        SDB[(SurrealDB)]:::infra
        QD[(Qdrant)]:::infra
        CW1[Celery\nmessaging]:::service
        CW2[Celery\nassets]:::service
        CB[Celery Beat]:::service
    end

    FA --> RB
    CW1 --> RB
    CW2 --> RB
    CB --> RB
    CW1 --> SDB
    CW2 --> SDB
    CW1 --> QD
    CW2 --> QD
    CW2 --> SF
```

### 4.3 Docker Compose — Production Stack

Create this file in your Coolify project as `docker-compose.prod.yml`. Values marked `changeme` must be replaced with your own secrets.

```yaml
version: "3.9"

services:
  # ---- Broker & Cache ----
  redis:
    image: redis:7.4.1-alpine
    container_name: herandro-redis
    restart: always
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis_data:/data
    networks:
      - internal

  # ---- Primary Database ----
  surrealdb:
    image: surrealdb/surrealdb:2.2.1
    container_name: herandro-surrealdb
    restart: always
    entrypoint:
      - /surreal
      - start
      - --log
      - debug
      - --user
      - ${SURREAL_USER}
      - --pass
      - ${SURREAL_PASS}
      - rocksdb:/data/srdb.db
    volumes:
      - surrealdb_data:/data
    networks:
      - internal

  # ---- Vector Database ----
  qdrant:
    image: qdrant/qdrant:v1.13.4
    container_name: herandro-qdrant
    restart: always
    volumes:
      - qdrant_data:/qdrant/storage
    networks:
      - internal
    environment:
      QDRANT__SERVICE__GRPC_PORT: 6334
      QDRANT__SERVICE__HTTP_PORT: 6333

  # ---- Object Storage (SeaweedFS) ----
  seaweedfs:
    image: chrislusf/seaweedfs:3.80
    container_name: herandro-seaweedfs
    restart: always
    command: server -dir=/data -s3 -s3.config=/etc/seaweedfs/s3.json -ip=seaweedfs
    volumes:
      - seaweedfs_data:/data
      - ./seaweedfs-s3.json:/etc/seaweedfs/s3.json:ro
    networks:
      - internal
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:8333/status"]
      interval: 10s
      timeout: 5s
      retries: 5

  # ---- Celery Worker — Messaging Queue ----
  celery-messaging:
    build:
      context: ./worker
      dockerfile: Dockerfile.celery
    container_name: herandro-celery-messaging
    restart: always
    command: celery -A herandro_worker worker -Q default_messaging_queue --concurrency=4 --loglevel=info
    environment: &worker_env
      CELERY_BROKER_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
      CELERY_RESULT_BACKEND: redis://:${REDIS_PASSWORD}@redis:6379/1
      SURREAL_URL: ws://surrealdb:8000/rpc
      SURREAL_USER: ${SURREAL_USER}
      SURREAL_PASS: ${SURREAL_PASS}
      SURREAL_NS: herandro
      SURREAL_DB: production
      QDRANT_URL: http://qdrant:6333
      SEAWEEDFS_S3_ENDPOINT: http://seaweedfs:8333
      SEAWEEDFS_ACCESS_KEY: ${SEAWEEDFS_ACCESS_KEY}
      SEAWEEDFS_SECRET_KEY: ${SEAWEEDFS_SECRET_KEY}
      BUCKET_NAME: assets
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      OPENAI_BASE_URL: ${OPENAI_BASE_URL:-https://api.openai.com/v1}
      TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
    depends_on:
      - redis
      - surrealdb
    networks:
      - internal

  # ---- Celery Worker — Heavy Asset Queue ----
  celery-assets:
    build:
      context: ./worker
      dockerfile: Dockerfile.celery
    container_name: herandro-celery-assets
    restart: always
    command: celery -A herandro_worker worker -Q heavy_asset_queue --concurrency=2 --loglevel=info
    environment:
      <<: *worker_env
    depends_on:
      - redis
      - surrealdb
    networks:
      - internal

  # ---- Celery Beat — Periodic Scheduler ----
  celery-beat:
    build:
      context: ./worker
      dockerfile: Dockerfile.celery
    container_name: herandro-celery-beat
    restart: always
    command: celery -A herandro_worker beat -S herandro_worker.schedulers.SurrealdbScheduler --loglevel=info
    environment:
      <<: *worker_env
    depends_on:
      - redis
      - surrealdb
    networks:
      - internal

  # ---- FastAPI Webhook Server ----
  fastapi:
    build:
      context: ./api
      dockerfile: Dockerfile.fastapi
    container_name: herandro-fastapi
    restart: always
    environment:
      CELERY_BROKER_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
      SURREAL_URL: ws://surrealdb:8000/rpc
      SURREAL_USER: ${SURREAL_USER}
      SURREAL_PASS: ${SURREAL_PASS}
      SURREAL_NS: herandro
      SURREAL_DB: production
      TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
      WEBHOOK_SECRET: ${WEBHOOK_SECRET}
      KEYCLOAK_PUBLIC_KEY: ${KEYCLOAK_PUBLIC_KEY}
    depends_on:
      - redis
      - surrealdb
    networks:
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.fastapi.rule=Host(`${DOMAIN}`) && PathPrefix(`/api/`)"
      - "traefik.http.services.fastapi.loadbalancer.server.port=8000"

volumes:
  redis_data:
  surrealdb_data:
  qdrant_data:
  seaweedfs_data:

networks:
  internal:
    driver: bridge
```

### 4.4 SeaweedFS S3 Configuration

Create `seaweedfs-s3.json` alongside the compose file:

```json
{
  "identities": [
    {
      "name": "herandro",
      "credentials": [
        {
          "accessKey": "changeme-access-key",
          "secretKey": "changeme-secret-key"
        }
      ],
      "actions": ["Admin", "Read", "Write"]
    }
  ],
  "accounts": []
}
```

> 💡 **Tip:** Match `SEAWEEDFS_ACCESS_KEY` and `SEAWEEDFS_SECRET_KEY` environment variables to the values in this JSON file.

### 4.5 Dockerfile — Celery Worker

Create `worker/Dockerfile.celery`:

```dockerfile
# ---- Base Stage ----
FROM python:3.12-slim AS base
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential libpq-dev curl ca-certificates \
    tesseract-ocr libtesseract-dev poppler-utils \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# ---- Builder Stage ----
FROM base AS builder
COPY worker/requirements.txt .
RUN pip install --user -r requirements.txt

# ---- Runtime Stage ----
FROM base AS runtime
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH

COPY worker/ /app/worker/
COPY shared/ /app/shared/

RUN groupadd -r celery && useradd -r -g celery celery
# Tesseract needs root access; if issues arise, keep root user
USER celery

CMD ["celery", "-A", "herandro_worker", "worker", "-l", "info"]
```

### 4.6 Dockerfile — FastAPI

Create `api/Dockerfile.fastapi`:

```dockerfile
FROM python:3.12-slim

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

WORKDIR /app

COPY api/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY api/ /app/api/
COPY shared/ /app/shared/

EXPOSE 8000

CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"]
```

### 4.7 Coolify Deployment Checklist

1. Create a new Coolify project → "Docker Compose" deployment type.
2. Connect your Git repository (or paste the compose file directly).
3. Assign your domain to the `fastapi` service.
4. Set all `changeme` environment variables in the Coolify UI.
5. Deploy. Coolify will build images, create volumes, and provision SSL automatically.
6. After first deploy, run the SurrealDB init script from §5.3 to create tables.

---

## 5. Core Components in Depth

### 5.1 FastAPI Webhook Layer

The FastAPI server acts as the **only HTTP entry point** for the system. Its responsibilities:

1. Receive Telegram updates via a POST endpoint.
2. Verify the `X-Telegram-Bot-Api-Secret-Token` header to prevent spoofing.
3. **Immediately return HTTP 200** after pushing the update to Redis.
4. Expose optional Keycloak‑protected management endpoints.

**Minimum viable webhook endpoint:**

```python
# api/main.py
from fastapi import FastAPI, Request, HTTPException
from redis import asyncio as aioredis
import os
import json
import hmac

app = FastAPI()
redis = aioredis.from_url(os.environ["CELERY_BROKER_URL"])

WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"].encode()

@app.post("/api/telegram/webhook")
async def telegram_webhook(request: Request):
    # 1. Verify secret token
    secret_token = request.headers.get("X-Telegram-Bot-Api-Secret-Token")
    if not secret_token or not hmac.compare_digest(secret_token, WEBHOOK_SECRET.decode()):
        raise HTTPException(status_code=403)

    # 2. Push to Redis and return immediately
    update = await request.json()
    await redis.lpush("telegram_updates", json.dumps(update))
    return {"status": "ok"}
```

> 💡 **Tip:** The `hmac.compare_digest` function prevents timing attacks on secret comparison.

The Celery worker `process_telegram_update` task pops from `telegram_updates` and handles classification, tool dispatch, and reply generation.

### 5.2 Celery + Redis Asynchronous Engine

#### Queue Architecture

| Queue | Purpose | Concurrency | Tasks |
|---|---|---|---|
| `default_messaging_queue` | Fast text processing and replies | 4 workers | `text_analyzer`, `intent_classifier`, `quick_rag_lookup`, `send_reply` |
| `heavy_asset_queue` | Media parsing, URL scraping, OCR | 2 workers | `media_processor`, `url_scraper`, `ocr_extract`, `index_document` |

#### Celery Configuration (`celery.py`)

```python
from celery import Celery
import os

app = Celery(
    "herandro_worker",
    broker=os.environ["CELERY_BROKER_URL"],
    backend=os.environ["CELERY_RESULT_BACKEND"],
    include=["herandro_worker.tasks"]
)

app.conf.update(
    task_routes={
        "herandro_worker.tasks.process_telegram_update": {"queue": "default_messaging_queue"},
        "herandro_worker.tasks.media_processor":          {"queue": "heavy_asset_queue"},
        "herandro_worker.tasks.url_scraper":              {"queue": "heavy_asset_queue"},
        "herandro_worker.tasks.index_document":           {"queue": "heavy_asset_queue"},
    },
    worker_prefetch_multiplier=1,     # One task at a time — prevents head‑of‑line blocking
    task_acks_late=True,              # Only acknowledge after success
    task_reject_on_worker_lost=True,  # Re‑queue if worker crashes
    task_soft_time_limit=300,         # 5 minutes max per task (prevents zombie tasks)
    task_time_limit=600,              # Hard kill after 10 minutes
)
```

> ⚠️ **Note:** `worker_prefetch_multiplier=1` is critical for queue isolation. Without it, a fast worker could consume all messages from the slow queue during prefetch.

#### Custom SurrealDB Scheduler for Celery Beat

The default Celery Beat scheduler uses a local `shelve` file, which is lost on container restart. A custom scheduler stores entries in SurrealDB, ensuring reminders survive reboots.

```python
# herandro_worker/schedulers.py
from celery.beat import PersistentScheduler
from surrealdb import Surreal
import os
import logging

logger = logging.getLogger(__name__)

class SurrealdbScheduler(PersistentScheduler):
    """
    Stores Celery Beat schedule entries in SurrealDB so they persist
    across container restarts and deployments.
    """

    def __init__(self, *args, **kwargs):
        self._db = None
        super().__init__(*args, **kwargs)

    def _init_db(self):
        if self._db is None:
            self._db = Surreal(os.environ["SURREAL_URL"])
            self._db.signin({
                "user": os.environ["SURREAL_USER"],
                "pass": os.environ["SURREAL_PASS"]
            })
            self._db.use(os.environ["SURREAL_NS"], os.environ["SURREAL_DB"])
            # Ensure table exists
            self._db.query("""
                DEFINE TABLE IF NOT EXISTS beat_schedule SCHEMAFULL;
                DEFINE FIELD IF NOT EXISTS task ON beat_schedule TYPE string;
                DEFINE FIELD IF NOT EXISTS schedule ON beat_schedule TYPE string;
                DEFINE FIELD IF NOT EXISTS args ON beat_schedule TYPE option<array>;
                DEFINE FIELD IF NOT EXISTS kwargs ON beat_schedule TYPE option<object>;
            """)

    def setup_schedule(self):
        self._init_db()
        # Load schedule from SurrealDB
        result = self._db.query("SELECT * FROM beat_schedule")
        self._store = {"db": {}, "types": {}}
        if result and result[0]["status"] == "OK":
            for entry in result[0]["result"]:
                self._store["db"][entry["id"]] = {
                    "task": entry.get("task", ""),
                    "schedule": entry.get("schedule", ""),
                    "args": entry.get("args") or [],
                    "kwargs": entry.get("kwargs") or {},
                }
        self.merge_inplace(self.app.conf.beat_schedule or {})