Multi-Agent AI Platform / observability & knowledge

What the platform does

The platform lets business teams build their own AI agents, give them skills, MCP connectors, and knowledge bases, and chat with them under approval gates and policies. It is an internal product, still before public release. I joined in mid-2026. I owned observability and the knowledge-base subsystem, and I led the integration of a browser extension that records how people do a task.

The system has three parts:

• A FastAPI backend owns the business data, meaning agents, conversations, runs, and knowledge bases. It relays agent output to browsers over authenticated SSE with replay.
• A separate agent runtime runs DeepAgents on LangGraph, with a Postgres checkpointer and Celery workers. It never touches the business database. Everything goes through an RPC API on the backend.
• A React 19 frontend, in English and Japanese.

Models run on Azure OpenAI and on Claude through Azure AI Foundry.

Tracing that never blocks a chat

I forked MLflow 3.x, trimmed it for self-hosting, and built it from source. The team needed its REST API, not its UI. A sitecustomize.py injected into the image patches Flask so every non-API route returns 404.

The first real design question was how to separate traces per agent. One MLflow experiment per agent looks natural, but the active experiment is process state, and concurrent async runs race on it. I patched the LangChain tracer in the fork instead. It copies agent_id, conversation_id, and run_id from the LangChain run metadata into trace tags. All traces share one experiment, and the backend filters with search_traces("tag.agent_id = ..."). The cost is a small patch to carry on every upgrade. I think that trade is right, because the alternative fails only under load.

When I read the existing integration, most tracking was dead code:

• init_mlflow() was never called.
• The start-run call was awaited with a three-second timeout, so a slow MLflow server delayed every chat message. It is now fire-and-forget.
• A tuple went where a run-ID string was expected.
• mlflow.start_run keeps an active-run stack per context, and overlapping runs on pool threads corrupted it. The runtime now uses a stateless MlflowClient.

Claude streams structured tokens, and OpenTelemetry rejected them with Invalid type dict in attribute token. The fork converts the token to a string at the source. The runtime adds a narrow logging filter for that one message, so every other telemetry warning still reaches the logs.

On the backend I built the trace list and detail API with an async-safe client and a check that the caller owns the agent.

Knowledge bases with three kinds of index

I led the knowledge-base subsystem end to end. It started as a requirements spec and twelve module designs, covering the data model, upload, extraction, indexes, deletion, MCP authorization, and scaling. Then I wrote the code across all three repos.

Every knowledge base has exactly one index type, and the type cannot change:

• Vector: chunks embedded into Azure AI Search.
• Graph: a Neo4j graph of documents, chunks, summaries, and entities for GraphRAG search. Entities are keyed by name, and shared nodes record which documents they came from. Deleting one document removes only what belongs to it.
• Wiki: a generated wiki of Markdown pages with reference, summary, concept, and entity pages. Every page carries YAML front matter and citations back to the source page. The prompt forbids inventing facts or sources.

The first version fanned every upload out to all three pipelines. Making the type exclusive replaced that fan-out with one Celery chain picked by the backend. The unused pipelines are marked as not configured, and the MCP tools refuse queries that do not match the type.

Knowledge bases later split from workspaces into their own table, with per-user membership and access levels on each workspace link. The runtime enforces that scope in the agent tools, the MCP server, and the API routes. MCP tokens without a workspace claim are rejected.

Video ingestion goes through Azure AI Content Understanding. The pipeline validates the MP4 container, writes time-stamped segment text with provenance, and treats most 4xx responses as permanent. Timeouts, conflicts, and rate limits are retried.

Deletion and failure you can see

Deleting a file touches three stores: the database, the search index, and blob storage. Any of them can fail.

1. The delete commits a tombstone first, so the file disappears from retrieval immediately.
2. It purges the index, then the blobs, then the job record.
3. If a step fails, the API returns 502 or 503 and keeps the job record. Sending the same DELETE again resumes where it stopped.

1. 01 Tombstone hidden from retrieval at once
2. 02 Purge index Azure AI Search
3. 03 Delete blobs Azure Blob Storage
4. 04 Drop job record kept on failure, so DELETE resumes

A workspace with active or half-purged files cannot be deleted. The API returns 409.

The other bug was quieter. Six of eleven Celery tasks wrote their status before their try block. When the status RPC failed, the task used up its retries and the file showed “Pending” forever. I added one Celery task_failure signal handler that marks the job failed. It also catches worker deaths that no except block can see.

Recording how people work

The newest direction on the platform turns a recording of a person doing a task into a reusable agent skill. A teammate started the Chrome extension that captures it, and I led its integration with the platform. The extension records clicks, screenshots, optional screen video, and spoken narration, then uploads the bundle as a versioned recording.

Live transcription uses Azure Speech real-time speech-to-text:

• The backend mints ten-minute tokens. The Azure key never reaches the browser, and the extension swaps tokens every eight minutes.
• Token minting is rate-limited per user and fails closed when Redis is down. An unlimited minute costs real money.
• Chrome kills a speech recognizer inside an offscreen document within milliseconds, so it runs in the side panel. Gaps in coverage are drawn as gaps. They are never treated as silence.

The upload records what was requested and what was actually captured, such as microphone access denied or truncated video. Downstream steps can then say plainly that narration was lost. Step cards crop each screenshot around the pointer, and narration flushes to IndexedDB every 15 seconds so a crash loses little.

Smaller hardening

• Idempotent message sends: a unique index on user and idempotency key. The losing concurrent request catches the integrity error and returns the winner’s result.
• Guarded Alembic migrations that detect drift and refuse to run, after a revision leaked into a shared database.
• A Claude Code hook that asks for confirmation before a migration targets any database that is not local.