We mapped out the 10,000-foot view of a production multi-agent system. We talked about how chaining four specialized agents together (correlation, ranking, routing, and remediation) allowed us to eliminate 87% of alert noise in an enterprise database environment.
But how do these agents actually work under the hood?
It is easy to say, “The AI correlates alerts.” It is much harder to actually engineer a system that does not hallucinate during a massive production outage.
“The engine room”: Let’s examine the precise decision-making process of the Correlation Agent during a complete infrastructure breakdown.
The Problem with “IF/THEN” Logic
Before AI, the best way to handle monitoring was to write complex, nested “if/then” rules.
IF CPU > 90% AND Memory > 80% THEN *trigger a critical alert.
The problem? Modern architectures are far too complex for simple rules. When a major service goes down, you do not get one alert. You get a cascading failure.
1. The database hits a connection limit.
2. The application servers start throwing 500 timeout errors.
3. The load balancers start redirecting traffic.
4. Hundreds of user-facing alerts trigger sequentially.
Within 60 seconds, an on-call engineer gets 50 separate text messages from 50 different monitoring systems. The engineer has to physically open multiple dashboards and manually deduce that “Alert 1” actually caused “Alert 45.”
Enter the correlation agent.
To solve this, we could not just write more rules. We needed a system capable of semantic reasoning. We needed the correlation agent.
The correlation agent’s entire job is to ingest thousands of raw, chaotic signals and group the ones that belong together into a single “incident cluster.” It uses two massive technical pillars to do this without human intervention.
The Semantic Correlation Engine: Finding needles in the Haystack
To appreciate the architecture, you have to visualize how the AI “sees” the infrastructure. It does not just look at text; it looks at the mathematical relationship between events.
This dual-layer approach-combining sequence timing with semantic similarity-is what allowed us to reach 87% noise reduction with near-zero false positives.
Pillar 1: Vector Embeddings (Semantic Similarity)
Text matching was not going to work. “High CPU” and “slow query execution” are the same issue, though they don’t share words.
We used an AI technique called “vector embeddings.” Every time an alert comes in, the agent translates the raw text and metrics into a mathematical vector (a massive array of numbers).
Because the AI model understands database administration, it mathematically maps the vector for “High CPU” incredibly close to the vector for “Slow Query Execution.”
Just knowing “what is wrong” is not enough for an enterprise system. You need to know “how to fix it.”
We integrated a retrieval-augmented generation (RAG) engine into the correlation agent. Once the AI forms a cluster of alerts, it instantly searches a historical database for previous incidents that share similar vector profiles. It then “augments” the final incident report with the exact runbook or SQL command that resolved the issue last year. This converts the agent from a “watcher” into a “senior advisor.”
The agent looks at a temporal window (did these alerts happen within 5 minutes?). It then runs the cluster through an advanced machine learning confidence scorer.
This model was trained on years of historical outage data. It has learned pattern covariance. It knows, statistically, that when a particular connection pool alert occurs on Database A, a specific timeout alert will almost always trigger 30 seconds later on Server B.
If the model returns a 95% confidence score indicating that these alerts are linked, the Correlation Agent bundles all 50 chaotic alerts into one clean, actionable incident ticket. It identifies the “Root Cause Alert” and suppresses the 49 “Symptoms.”
The Developer PM Reality: Data Engineering is Product Engineering
The biggest lesson I learned from building the Correlation Agent is that the actual AI model (the prompt and the LLM) is only 5% of the work.
The remaining 95% involves heavy data engineering tasks: cleaning the incoming telemetry, scaling the vector database to handle 10,000 alerts per minute, and building ingestion pipelines from enterprise monitoring systems.
If the data you feed the agent is garbage, the agent will confidently produce garbage.
We have now cleaned the noise and found the root cause. Tomorrow, we will look at the holy grail of agentic AI: How can you allow an AI to safely fix a production database without requiring human approval?
One prompt, traced end to end: tokens, attention, and the part where a vector database gets involved.
ONE PROMPT, ALL THE WAY DOWN
I am going to use exactly one prompt for this entire article:
What is our retention period for Slack messages?
I picked it on purpose. The word “our” is doing the work. No language model can answer this from its training, however large it is, because the answer lives in an internal policy document that did not exist when the model was trained. That one word separates “how a model generates text” from “how a model generates text grounded in private data,” and it lets me cover both without switching examples.
Every figure I quote from my own experiments was computed locally, on a corpus small enough to reproduce here in full. Where a number comes from a published source instead, I say so. And I will be explicit throughout about what my measurements do and do not demonstrate.
Stage 1. Words Becomes Numbers
The model cannot read characters. First comes tokenization: text is split into subword units, and each unit maps to an integer ID from a fixed vocabulary.
Most current models use Byte-Pair Encoding or a SentencePiece variant. BPE(Byte-Pair Encoding.) is built by starting from individual characters (or bytes, in byte-level variants like GPT-2’s) and repeatedly merging the most frequent adjacent pair, thousands of times, until the vocabulary hits a target size.
Rather than quote a production tokenizer’s IDs, I implemented BPE from scratch and trained it on a 150-word corpus of policy-style text, sixteen sentences of the kind an internal-policy assistant would see. Here are the first merges it learned, in order:
Merges 12 through 14 are the ones worth pausing on. The algorithm assembles ‘re’ plus ‘te’ into ‘rete’, then ‘reten’, then joins it to the ‘tion</w>’ it had already discovered back at step 7. It builds the whole word “retention” out of fragments it learned separately. Nobody specified that. It is pure frequency counting, applied repeatedly.
With 120 merges learned and a vocabulary of 148 symbols, here is our prompt tokenized:
Now the interesting part. Why did some words survive as single tokens while others shattered? Cross-referencing each word against how often it appears in the training corpus gives the answer:
Counts are of the exact lowercased word. “Slack” appears three times in the corpus, but twice with punctuation attached, as ‘slack.’ and ‘slack?’, so the bare form occurs once. That is exactly why it never fused into a single symbol: my merge rule requires a pair to appear at least twice, and ‘slack’ plus end-of-word only ever appeared once.
The pattern is clean. Words the tokenizer saw often compressed to one ID. Words it barely saw or never saw decomposed into fragments. This is the entire design goal of subword tokenization: the tokenizer never fails on unfamiliar input, it just spends more tokens on it.
That has three practical consequences:
Whitespace and punctuation are part of the token. The symbols ‘r</w>’ and ‘or’ differ precisely because of the end-of-word marker. In production tokenizers a leading space works the same way, which is why “Slack” and ” Slack” are different IDs, and why a stray trailing space in a prompt can occasionally shift output.
This is why models are bad at spelling tasks. A model asked to count the letters in a word struggles, because it never sees letters. It sees ‘messag’ and ‘es’. The limitation lives in the tokenizer, not in the intelligence.
There is a cost asymmetry across languages. BPE merges are learned from a training corpus that is overwhelmingly English. My toy tokenizer makes this visible in miniature: “our” cost 3 tokens because the corpus never contained it, so no merge could ever assemble it. Scaled up, the same sentence in an under-represented language consumes measurably more tokens than its English equivalent. Since nearly every API prices per token, that is a real cost difference, not a curiosity.
Note: This is my own small BPE, not GPT-4’s tokenizer. The algorithm is the same one production tokenizers use. The scale is not. Real vocabularies run to 100k or more symbols trained on enormous corpora, so their specific IDs and splits differ completely from mine. Anyone who wants production numbers should encode their own text with the tokenizer their provider publishes, and state which encoding was used. I built something small enough to verify end to end instead.
Stage 2. Numbers Become Meaning
Each token ID indexes a row in an embedding matrix, and that row is a vector of floats. For a mid-size model the matrix might be vocabulary size by 4096.
The important part: nobody assigned these vectors. Nobody wrote down that “Slack” should sit near “Teams.” That geometry emerged from gradient descent over an enormous corpus. It is learned structure, not a hand-built ontology.
That property is worth holding onto. It is the same one the vector database will exploit later, and the reason the two halves of this article are really one story.
Attention has no inherent sense of order. Without position information, “retention policy” and “policy retention” would be identical inputs. Position gets injected either by adding positional embeddings or, in most current open-weight models, by rotating the query and key vectors by a position-dependent angle, which is rotary position embedding, or RoPE. Closed models do not disclose their architecture, so “most” is the honest word.
Stage 3: Attention, Or Where – Enough Information – comes
How does the model know what a prompt means?
Each token’s vector is projected into three vectors: a query, a key, and a value. Token i’s query is compared against every token’s key by dot product, scaled, and pushed through a softmax to produce weights that sum to 1. Those weights take a weighted average of the value vectors.
In plain language: every token asks “who here is relevant to me”, receives a set of relevance scores, and pulls in a blend of the other tokens’ information.
In our prompt, the content-bearing tokens, “retention” and “slack”, would be expected to draw more weight at the final position than function words like “is” or “for”. I did not measure that here, since inspecting real attention maps is a separate exercise. The structural point is what matters: nobody programs which tokens matter. It falls out of the learned projections.
Three details that matter in practice.
Multi-head attention: This runs in parallel across many heads with separate projections, and different heads specialise. Interpretability researchers have identified heads that appear to perform specific jobs, though that is an active research area, not settled fact.
Causal masking: Positions after i are set to negative infinity before the softmax, so no token attends to the future. This is what makes left-to-right generation coherent. A model trained without it would learn to peek at the answer.
The KV cache: Keys and values for earlier tokens do not change as generation proceeds, so they are cached. This is why the first token of a response is slow, since the whole prompt must be processed, while later tokens are fast, one at a time. At long context lengths and large batch sizes the cache becomes a major memory consumer at serving time, and it is a big part of why long conversations get expensive.
Each layer pairs attention with a feed-forward network, which holds most of the model’s parameters. Current interpretability work suggests attention moves information between positions while feed-forward layers store and retrieve learned associations. I want to be careful here: that is a useful intuition from ongoing research, not a proven architectural fact.
Stage 4: A Probability for Every Possible Next Token
The final hidden state is multiplied by the unembedding matrix, producing one raw score, called a logit, per vocabulary entry. Softmax turns those into probabilities. A sampling step picks one.
temperature – Divides the logits before softmax. Below 1 sharpens toward the top candidate, above 1 flattens the distribution. Zero is greedy: always the highest-probability token, fully deterministic.
top_k – Keeps only the k highest-probability tokens, then renormalises.
top_p – Keeps the smallest set of tokens whose cumulative probability exceeds p. Adapts to model confidence.
frequency and presence penalties Subtract from the logits of tokens already used, which reduces repetition.
Here is the uncomfortable implication. Softmax always returns a valid probability distribution. If the model has no strong learned pattern for a given question, that distribution is nearly flat, but sampling still returns a token, and the resulting sentence reads exactly as fluent as a well-grounded one. The model has no separate “I do not know” output channel.
Hallucination is not a bug someone forgot to fix. It is a structural consequence of generating text by sampling from a distribution that is always defined.
Which brings us back to our prompt. Asked what “our” retention period is, a bare model produces something fluent, specific, and wrong, because no mechanism exists by which it could know.
What I Measured, And What I Did Not
Every tokenization figure above came from that 150-word corpus: the merges and their frequencies, the token IDs, the 17 tokens, the word-frequency table. I recomputed all of it from scratch a second time before publishing, because the first run is the one that flatters the author.
What it is not: my BPE is not a production tokenizer, and I make no claim that these specific IDs resemble any real model’s. Parameter counts, context window sizes and benchmark numbers are absent on purpose, since they date within months and I would have been quoting them from memory.
Where This Leaves Us
Nothing in stages 1 to 4 can fix that wrong answer. The mechanism that does fix it is not part of the model at all. It sits in front of the model, runs before inference, and is routinely described as something it is not.
How a vector database turns a question into a vector, what cosine similarity actually measures, the retrieval failure I did not expect to find in my own test corpus, and the one step most explanations skip.
Corrections are welcome. Every figure above traces to either my terminal or a other sources
