Enterprise Document AI Assistant / the ingestion backbone

What the platform does

The platform is an AI chat product that an enterprise client’s employees sign in to with Azure AD. It has a general assistant, with web search, translation, image generation, and speech. It also has Q&A over two kinds of documents: files a user uploads, and a SharePoint library the company syncs. Answers cite the passages they come from.

The system is a set of FastAPI services on AKS, deployed through ArgoCD:

• A gateway handles identity and routes each conversation to the right backend.
• A Q&A service runs the retrieval agent, built with LangGraph.
• An assistant service handles general chat, translation, and media.
• A document service ingests, extracts, and indexes files.

All models run on Azure OpenAI. Azure AI Search and pgvector hold the embeddings.

I wrote the first commit of the document service and stayed its top contributor for over a year. I also led the Q&A agent loop, did much of the assistant-service work, and set up the UAT and production deployments. The hard problems were rarely the LLM. They were keeping a large SharePoint library in sync, respecting its permissions exactly, and keeping workers alive on files that were bigger than they looked.

Syncing SharePoint without starting over

The first version listed every file on every run. I moved it to the Microsoft Graph delta API:

1. Each sync runs per drive and folder and stores the @odata.deltaLink it receives in a token table.
2. The next run starts from that link and gets only what changed. Deleted items come back with a deleted facet and are removed from the index.
3. If Graph returns 410 or 404, or a resyncRequired error, the sync falls back to a full listing for that folder.
4. If a run fails, the token stays where it was, so the next run replays from the same point instead of skipping changes.

1. 01 Load delta link per drive and folder
2. 02 Fetch changes only 410 or resyncRequired: full listing
3. 03 Index or delete skip when quickXorHash is unchanged
4. 04 Save new token only after success

Metadata changes are not content changes. SharePoint Online exposes one content hash, quickXorHash. The service stores it and skips reprocessing when a file was renamed or re-tagged but its bytes are the same.

The delta API can still miss things. A weekly reconcile task does a full fetch and, folder by folder, deletes files that no longer exist in SharePoint. The daily coordinator started as one long task. It is now a fan-out of independent per-folder tasks, so one slow folder cannot hit a global time limit.

Resyncing permissions under throttling

Q&A over SharePoint must return only documents the user can open. That means reading permissions for every file, which is hundreds of thousands of Graph calls, and Graph throttles hard.

I redesigned the resync over several iterations:

• It runs on its own Celery queue and worker, so a long resync never blocks user uploads.
• It sends Graph $batch requests with 20 sub-requests each. That cuts HTTP round trips roughly twentyfold.
• A throttled sub-request returns 429 with its own Retry-After header. The worker collects those, sleeps for the longest one, and retries only the throttled requests, round after round, until all of them succeed. A fixed three-attempt backoff used to give up with files still missing permissions.
• Database writes are chunked, and group membership is cached for the length of one task.

The permission rules matter as much as the throughput. Group grants expand to their owners or members, with paginated member fetches. Direct user grants map to an email. A site group that Graph cannot enumerate, such as “everyone except external users”, becomes a wildcard entry, so the pipeline never guesses about that group’s members.

Permissions the model never sees

The Q&A agent reads documents through tools served by an MCP server inside the document service. The key rule: identity never passes through the LLM.

• The calling service authenticates with an internal token and sends the user’s email, allowed folders, and conversation document scope as headers.
• Middleware turns those headers into a request-scoped UserScope. Every tool filters by the intersection of the folder ACL and the conversation scope.
• No tool accepts an email or folder argument, so a prompt injection has nothing to change.
• A missing header and an empty list mean different things. An empty list returns zero rows. It never falls back to “everything”.

An agent loop with a token budget

The SharePoint Q&A agent follows the loop design of the open-source Onyx project. It runs at most six cycles with up to three concurrent tool calls. Every cycle rebuilds the message list inside a 128k-token budget:

• 16k tokens are reserved for the answer, 4k for the system prompt, and 8k for the current turn.
• History fills the rest, newest first, in whole turns.
• When a tool result is evicted, a short system note takes its place. The note tells the agent to call read_file again instead of answering from memory.

sys

turn

answer

• sys 4k system prompt
• history whole turns, newest first
• turn 8k current turn
• answer 16k reserved
• dashed evicted turn, leaves a breadcrumb

Answers must be grounded in tool output. The last pass merges citations, strips any citation that does not resolve to a real source, and repairs broken formatting.

Workers that survive real files

The worst worker failures were memory. Three fixes stand out.

• A 32 MB PowerPoint made LibreOffice peak around 2.6 GB inside a 1.5 GiB pod. I moved Office conversion out of the worker into a dedicated Gotenberg service and recorded the decision as an ADR.
• Image OCR with a vision model ran in three nested serial loops, and image-heavy decks took over 30 minutes. It now runs through asyncio.gather with a semaphore sized to the model’s token-per-minute quota, and it skips repeated slide images by content hash.
• PDFs linked in chat go to Document Intelligence capped at 60 pages. For a large file that took peak memory from about 1.7 GB to 480 MB and time from 208 s to 79 s. The link reader also validates every redirect hop against SSRF, checks the %PDF magic bytes, and caches extraction across follow-up turns.

Streaming needed a fix at three layers. Long silent agent stretches dropped HTTP/2 connections. The existing keep-alive was an empty string, which ASGI treats as a no-op. It is now a real sentinel sent every 20 seconds. On the frontend, the stripper that removes the sentinel held back a fixed number of characters and froze word-by-word output. It now holds back only characters that could start the sentinel.

What came next

In 2026 I designed and built a next-generation agent service on LangChain Deep Agents, intended to replace the separate Q&A and assistant services. It has one coordinator with knowledge, artifact, and data subagents, Postgres checkpointing, resumable SSE over Redis Streams, sandboxed code execution, and the document service’s MCP tools. It has over 300 tests and is not deployed yet.