The Online Part of RAG

In the offline half I built the index: read documents, chunk them, embed each chunk, store the vectors in pgvector. That half has no language model in it at all. This is the other half — the one that runs per question and finally involves an LLM: retrieve the relevant chunks, wrap them in a prompt, and let the model answer from that context and nothing else.

It's built on the same project, spring-ai-rag-pgvector, with one deliberate twist: the architecture is now hybrid. Embeddings still run locally on Ollama; generation runs in the cloud on DeepSeek. RAG makes that split natural — the privacy-sensitive, run-often part (your documents becoming vectors) stays on the machine, and only the retrieved snippets plus the question travel to the rented model.

Three steps hiding behind one call

Spring AI’s QuestionAnswerAdvisor turns a single fluent call into retrieve, augment, and generate.

The whole online loop is one line of application code:

String answer = chatClient.prompt().user(question).call().content();

That reads like a plain chatbot call, but a QuestionAnswerAdvisor is attached to the ChatClient as a default advisor, and it expands the call into three steps. It runs a similarity search over the vector store, splices the retrieved chunks into the prompt through a template, and only then calls the model. An advisor in Spring AI is an interceptor around the chat call — essentially AOP for prompts — so the retrieval logic lives outside your controller entirely.

Wiring it: two providers, one classpath

The interesting configuration problem is having two model providers present at once. Spring AI resolves it with explicit selection rather than guesswork:

spring:
  ai:
    model:
      chat: deepseek        # generation  -> DeepSeek (cloud)
      embedding: ollama     # embeddings  -> Ollama  (local, 768-d)
    deepseek:
      api-key: ${DEEPSEEK_API_KEY:not-configured}
      chat:
        options:
          model: deepseek-v4-flash
          temperature: 0.3    # low -> factual over creative

Those two model.* lines are what keep the wiring unambiguous: DeepSeek backs the ChatModel, Ollama backs the EmbeddingModel, and no bean collides. One dependency note that cost a compile: QuestionAnswerAdvisor ships in spring-ai-advisors-vector-store, which the starters do not pull in transitively — add it explicitly.

Grounding is a prompt, not magic

The advisor merges context and question through a template, and that template is where a RAG system earns its trust. The two placeholders are required; the rules are mine:

<query>

Context information is below, between the dashed lines.
---------------------
<question_answer_context>
---------------------

Given the context information and no prior knowledge, answer the query.
Rules:
1. If the answer is not in the context, say you don't know - never invent one.
2. Do not write "based on the context" or "the provided information".
3. Be concise and factual.

Rule 1 is the whole game. Without it the model happily falls back on its training data, and you get fluent answers that have nothing to do with your documents. With it, the model is fenced to the retrieved context.

The proof: an honest "I don't know"

The model plainly knows France won in 1998 — but it isn’t in the retrieved context, so the answer is “I don’t know.”

Here is the endpoint answering Who won the world cup in 1998? against a knowledge base that only contains notes about RAG. DeepSeek certainly holds that fact in its weights — the answer is France. But retrieval returned only RAG chunks, scoring around 0.28, and the grounding rule fenced the model to that context. So it answered “I don't know.”

That refusal is the difference between a chatbot and a retrieval-grounded system. When the same endpoint is asked something the documents do cover — “what is chunking and why does it matter?” — it returns an accurate, on-topic answer drawn straight from the stored chunks. Same code, same model; the only variable is whether retrieval found something.

Return the sources, always

An answer you can't audit is a liability, so /rag/ask returns the retrieved chunks alongside the generated text — source name, similarity score, and the passage itself. You can see exactly what the answer stood on, and the low scores on the World Cup question are themselves the tell that retrieval came up empty. It also means the endpoint degrades gracefully: with no chat key configured it still returns the retrieved context, just without the generated summary on top. Retrieval is the durable half; generation rides on it.

Where this leaves the project

Both halves now exist: an offline pipeline that turns documents into a searchable index, and an online endpoint that answers questions from it and refuses when it can't. The code is on GitHub, with a Bruno collection to exercise every endpoint. The obvious next steps are the tuning knobs that decide quality: a similarity threshold so weak retrievals are dropped before they reach the model, streaming for responsiveness, and inline citations that point at the exact chunk behind each claim. But the shape is complete — and the most reassuring thing it does is say “I don't know.”

The official Spring AI doc under RAG section is nicely written, please have a look Spring AI RAG official reference