Mind Behind Machines

Category: Multi-Agent

  • The Limits of a Single AI

    Our systems were generating around 5,000 alerts every single month. Nearly 95% of those alerts were noise: false positives, duplicates, or low-priority warnings. Our engineers were spending hours manually triaging this wall of red text. Critical issues, the actual 5% that mattered, were getting buried.

    The Realisation: Refining the Strategy

    I realized that having a single “smart” predictive agent was like having one brilliant specialist trying to manage a chaotic emergency room by themselves. The sheer volume of incoming signals overwhelms single-purpose tools.

    We did not need a better dashboard. We needed an intelligent ecosystem that could triage and judge data just like a team of human experts would.

    The Multi-Agent Ecosystem

    Building a team of AI agents that collaborate is the true frontier of agentic AI.

    We evolved our strategy into a specialized multi-agent approach:

    1. The Alert Triage Agent

    This agent was specifically trained to group related alerts together and filter out the noise. It reduced alert volume by 87% almost instantly.

    2. The Compliance and Governance Agent

    In parallel, we deployed another high-impact agent focused exclusively on database governance. It continuously monitors 50+ compliance requirements like GDPR and PCI DSS. It does not just alert when there is a violation; it auto-remediates low-risk configuration drifts immediately.

    Article content

    The Orchestration Architecture: Moving from Silos to Ecosystems

    This shift required us to completely rethink our project architecture. Instead of a single monolithic AI trying to do everything, we built a decentralized multi-agent platform.

    To visualize how these agents coordinate, we moved away from rigid point-to-point connections to an Event-Driven Orchestration Bus. This allows the agents to “subscribe” to the data they care about most.

    Article content

    Each agent operates in its own containerized world with its own specialized knowledge. They communicate findings through the secure message bus, sharing evidence and trigger events. This allows us to scale each agent independently based on the workload, if we have an influx of alerts, we simply scale the Triage Agent without affecting the Compliance checks.

    By networking these agents together, we achieved:

    – 90% automation of compliance checks

    – 87% reduction in raw alert noise

    – Audit readiness in just 2 hours instead of 2 weeks

    Building this ecosystem changed everything. We stopped building isolated AI features and started architecting an intelligent workforce.

    Have you ever experienced “Alert Fatigue”? How did your team handle the noise before AI?

    Continue to self learn, reflect on the old models, and refine your approach.

  • Building a Self-Learning AI Chatbot for Database Operations

    How I built a domain-specific chatbot that learns from every user interaction using RAG, PostgreSQL, and a local LLM, with zero training cost :-).

    The Problem

    We manage hundreds of PostgreSQL, MSSQL, Oracle, MongoDB, and MySQL servers across multiple products. Every day, engineers ask the same kinds of questions:

    > “How many critical alerts came in last week?”

    > “Show me the database growth trend for server X.”

    We needed a chatbot that could answer these questions and get smarter over time based on our team’s feedback. But we didn’t have access to GPU clusters for fine-tuning, and we didn’t want to send proprietary data to external APIs.

    The constraint: Build an AI assistant that learns from users, runs entirely on-prem(with very basic server config is more than enough), and costs nothing to train.

    The Architecture

    Here is the complete system at a glance:

    Article content

    How the Chatbot Works -Three Layers

    The chatbot does not just throw everything at an LLM. It uses a layered approach to answer questions as fast and accurately as possible:

    Article content

    Layer 1: Rule Engine (< 50ms): Most operational questions follow patterns. “Show critical alerts for PROJ1” matches a rule, fills a SQL template, and returns data instantly. No LLM needed.

    Layer 2: Knowledge Base: If the rule engine doesn’t match, we check reviewer notes and historical incident data for relevant context.

    Layer 3: LLM with RAG: For everything else, the local LLM takes over. But before it generates an answer, we retrieve similar past Q&A pairs rated positively by users and inject them into the prompt.

    The Learning Loop: How RAG Makes the Chatbot Smarter

    Every response the chatbot gives has thumbs-up/down buttons. When a user interacts with these buttons, a learning cycle begins:

    Article content

    What happens on 👍 (Thumbs Up)

    The question-answer pair is promoted to the Learned Q&A table, a curated knowledge base of verified good answers. If the same question gets multiple thumbs-up, the number quality_score increases, making it rank higher in future retrievals.

    What happens on 👎 + Correction

    The user types what the answer should have been. The correction replaces the bad answer in the knowledge base. Next time a similar question is asked, the LLM sees the corrected answer as context.

    The key insight

    The LLM model weights never change. We don’t fine-tune. Instead, we give the LLM a “cheat sheet” of verified answers before it responds. This is Retrieval-Augmented Generation, the same architecture behind many production AI systems, adapted for a small local model.

    Pure-SQL RAG:- No Vector Database Required

    Most RAG implementations require a vector database (Pinecone, pgvector, Weaviate) and an embedding model. We didn’t have access to either.

    Instead, we built RAG using PostgreSQL’s built-in pg_trgm extension for fuzzy text matching:

    Article content

    How it works:

    1. pg_trgm breaks text into 3-character sequences (trigrams)

    2. It compares the user’s question against every learned Q&A

    3. Matches above 15% similarity and minimum quality score are returned

    4. Top 3 matches are formatted and appended to the LLM’s system prompt

    No embeddings. No vector DB. No GPU. Just PostgreSQL doing what it does best.

    What the LLM Actually Sees

    When a user asks “critical alerts this week”, the LLM receives this prompt:

    Article content

    The LLM reads the verified answer as context and produces a response that’s informed by what worked before. Over time, as more feedback accumulates, the context gets richer and more accurate.

    Key Metrics

    We track four metrics to verify the system is actually learning:

    learned_qa_count – New verified answers added to knowledge base

    use_count – Learned answers being retrieved and used in prompts

    quality_score – Same answer reinforced by multiple users

    corrections – Users actively teaching better answers

    After just a running few learning cycles, we saw:

    – 3 learned Q&A pairs with quality scores up to 1.5 (multi-user reinforcement)

    – Learned answers being retrieved 3+ times for similar queries

    – 52-71% similarity matches catching question variations accurately

    What’s Next

    1. More feedback = better retrieval.

    2. Full-text search with ts_vector.

    3. Fine-tuning.

    Key Takeaway

    With PostgreSQL’s pg_trgm, a local 7B model, and a simple feedback → store → retrieve → inject loop, you can build a domain-specific assistant that gets smarter with every interaction.

    The learning happens not by changing the model, but by changing what the model sees.

    Note: This is something I tried over the last few days. Since access to GPUs and very large models isn’t always feasible (cost + infra constraints), I wanted to see how far we can go with a smaller local model + smart retrieval. Curious to hear what others think, feedback is very welcome 🙂.

  • Architecting a Production Multi-Agent System

    “Alert Fatigue.” When your infrastructure generates 5,000 alerts a month and 95% of them are noise, human engineers get burnt out, and critical outages are missed.

    To solve this, we could not just build a simple AI chatbot. We needed a system capable of reasoning, correlating, and acting autonomously. We needed to orchestrate a team of specialized AI workers.

    I am going to walk you through the exact architecture of the Multi-Agent Alert Intelligence Platform we built, which successfully reduced alert noise by 87%, filtering thousands of raw alerts down to just a few hundred highly actionable incidents.

    Welcome to the depths of Agentic AI. (Grab a coffee and read through).

    The Architecture: How Specialized Agents Collaborate

    The hardest part of building complex AI products is not the machine learning model itself; it is the orchestration. If you ask one AI model to do everything, it will hallucinate and fail.

    Instead, designed an architecture where four specialized, distinct AI agents collaborate in a step-by-step pipeline.

    Here is the visual blueprint of how data flows through the system:

    Article content

    The Intelligent Decision Gate: Orchestrating the Handover

    One of the most complex parts of this architecture is the “Decision Gate” between the Ranking Agent and the Router/Remediator. It is not just a linear path; it is a high-speed logic tree that evaluates risk, history, and impact in milliseconds.

    Article content

    Each agent is a specialized microservice that operates independently but shares a unified vision of the infrastructure state. We use a combination of vector similarity and specialized decision trees to ensure that no action is taken without high confidence.

    Breaking Down the 4 Agents

    1. The Correlation Agent (Finding Patterns in the Noise)

    The first agent receives every metric, log, and warning generated by the infrastructure. Its only job is to figure out what is related.

    If a database CPU spikes to 99%, and a minute later an application starts throwing “timeout errors,” traditional systems treat these as two separate emergencies. Our Correlation Agent uses vector embeddings (to understand the semantic relationships in the data) and temporal analysis (to understand timing) to determine that they are the exact same event.

    It automatically clusters these 50 separate noisy alerts into one unified “Incident Profile.”

    2. The Ranking Agent (Understanding Business Context)

    Not all emergencies are equal. A server failing in a testing environment on a Sunday is not the same as a checkout cart failing on Black Friday.

    The ranking agent analyzes the clustered incident and assigns it a business impact score (1 to 100). It acts entirely based on context. It looks at whether the affected database serves customer-facing traffic, what time of day it is, and whether this issue has historically caused revenue loss. Only issues that truly impact the business wake up the engineering team at 3:30 AM.

    3. The Routing Agent (The Dispatcher)

    Once an incident is verified and ranked as high priority, it needs to be fixed. The Routing Agent acts as an intelligent dispatcher.

    Instead of just paging whichever engineer happens to be “on call,” it looks at a skill matrix. If the incident involves a complex issue with database lock contention, the agent routes the alert directly to the engineer who has the highest historical success rate at solving that specific database problem (SME), while also checking if their current workload allows it. It provides the engineer with the root cause analysis immediately, saving hours of manual investigation.

    4. The Remediation Agent (The Autonomous Fixer)

    This is the most advanced part of the system. Over time, the agents learn how humans fix recurring problems.

    If the correlation agent identifies an incident as a “known issue” (for example, a simple connection pool exhaustion), the remediation agent intercepts it before a human is ever notified. It automatically executes the pre-approved fix (like increasing the connection limit by 20%), verifies that the metrics have stabilized, and logs the entire action in an immutable audit trail.

    If the autofix fails, then it routes it to a human.

    The Developer PM Takeaway

    Building this system completely redefined my understanding of AI architecture. We moved away from thinking “How do we write a better prompt?” to “How do we orchestrate a secure, multi-agent digital workforce?”

    This is how you achieve true operational scale. By having AI agents handle the noise, correlation, and basic remediation, human engineers are freed up to do what they do best: complex problem solving and innovation.

    Tomorrow, we will go one level deeper. We will look specifically at the Correlation Engine and how an AI actually processes massive data streams without crashing.

    Until then, have you ever worked with a multi-agent system, or are you still primarily using single-prompt generative AI tools?