How I built a domain-specific chatbot that learns from every user interaction using RAG, PostgreSQL, and a local LLM, with zero training cost :-).
The Problem
We manage hundreds of PostgreSQL, MSSQL, Oracle, MongoDB, and MySQL servers across multiple products. Every day, engineers ask the same kinds of questions:
> “How many critical alerts came in last week?”
> “Show me the database growth trend for server X.”
We needed a chatbot that could answer these questions and get smarter over time based on our team’s feedback. But we didn’t have access to GPU clusters for fine-tuning, and we didn’t want to send proprietary data to external APIs.
The constraint: Build an AI assistant that learns from users, runs entirely on-prem(with very basic server config is more than enough), and costs nothing to train.
The Architecture
Here is the complete system at a glance:
How the Chatbot Works -Three Layers
The chatbot does not just throw everything at an LLM. It uses a layered approach to answer questions as fast and accurately as possible:
Layer 1: Rule Engine (< 50ms): Most operational questions follow patterns. “Show critical alerts for PROJ1” matches a rule, fills a SQL template, and returns data instantly. No LLM needed.
Layer 2: Knowledge Base: If the rule engine doesn’t match, we check reviewer notes and historical incident data for relevant context.
Layer 3: LLM with RAG: For everything else, the local LLM takes over. But before it generates an answer, we retrieve similar past Q&A pairs rated positively by users and inject them into the prompt.
The Learning Loop: How RAG Makes the Chatbot Smarter
Every response the chatbot gives has thumbs-up/down buttons. When a user interacts with these buttons, a learning cycle begins:
What happens on 👍 (Thumbs Up)
The question-answer pair is promoted to the Learned Q&A table, a curated knowledge base of verified good answers. If the same question gets multiple thumbs-up, the number quality_score increases, making it rank higher in future retrievals.
What happens on 👎 + Correction
The user types what the answer should have been. The correction replaces the bad answer in the knowledge base. Next time a similar question is asked, the LLM sees the corrected answer as context.
The key insight
The LLM model weights never change. We don’t fine-tune. Instead, we give the LLM a “cheat sheet” of verified answers before it responds. This is Retrieval-Augmented Generation, the same architecture behind many production AI systems, adapted for a small local model.
Pure-SQL RAG:- No Vector Database Required
Most RAG implementations require a vector database (Pinecone, pgvector, Weaviate) and an embedding model. We didn’t have access to either.
Instead, we built RAG using PostgreSQL’s built-in pg_trgm extension for fuzzy text matching:
How it works:
1. pg_trgm breaks text into 3-character sequences (trigrams)
2. It compares the user’s question against every learned Q&A
3. Matches above 15% similarity and minimum quality score are returned
4. Top 3 matches are formatted and appended to the LLM’s system prompt
No embeddings. No vector DB. No GPU. Just PostgreSQL doing what it does best.
What the LLM Actually Sees
When a user asks “critical alerts this week”, the LLM receives this prompt:
The LLM reads the verified answer as context and produces a response that’s informed by what worked before. Over time, as more feedback accumulates, the context gets richer and more accurate.
Key Metrics
We track four metrics to verify the system is actually learning:
learned_qa_count – New verified answers added to knowledge base
use_count – Learned answers being retrieved and used in prompts
quality_score – Same answer reinforced by multiple users
corrections – Users actively teaching better answers
After just a running few learning cycles, we saw:
– 3 learned Q&A pairs with quality scores up to 1.5 (multi-user reinforcement)
– Learned answers being retrieved 3+ times for similar queries
– 52-71% similarity matches catching question variations accurately
What’s Next
1. More feedback = better retrieval.
2. Full-text search with ts_vector.
3. Fine-tuning.
Key Takeaway
With PostgreSQL’s pg_trgm, a local 7B model, and a simple feedback → store → retrieve → inject loop, you can build a domain-specific assistant that gets smarter with every interaction.
The learning happens not by changing the model, but by changing what the model sees.
Note: This is something I tried over the last few days. Since access to GPUs and very large models isn’t always feasible (cost + infra constraints), I wanted to see how far we can go with a smaller local model + smart retrieval. Curious to hear what others think, feedback is very welcome 🙂.