Short answer: Production RAG evaluation should measure whether the system retrieves the right evidence, answers from that evidence, respects permissions, handles fresh data, and refuses unsupported questions. Compare keyword, vector, SQL, and hybrid retrieval against the same question set. Use retrieval metrics, answer-quality checks, and production failure tests before deciding which path belongs in the application.

RAG, or retrieval-augmented generation, retrieves evidence from an authoritative source and gives that evidence to a language model before it answers. Production RAG evaluation tests both halves of that process: whether retrieval found the right evidence and whether the generated answer used it correctly.

A RAG demo can pass with a few clean documents and one friendly question. Production is where the system starts meeting real users.

They ask for exact IDs. They ask vague questions. They ask about data that changed five minutes ago. They ask across tenants, versions, tables, PDFs, status fields, and long conversations. Sometimes the right answer is not in the corpus at all.

That is why the useful question is not “Should I use vector search or hybrid search?” The useful question is “Which retrieval path gives the application the right evidence, under the constraints this system actually has?”

This article turns that question into a practical evaluation plan for developers building RAG with Oracle AI Database. The companion notebook benchmarks three retrieval methods:

  • keyword retrieval for exact terms, identifiers, and lexical matches
  • vector retrieval for semantic similarity and vocabulary mismatch
  • RRF hybrid retrieval when keyword and vector candidates both add value

SQL or natural-language-to-SQL is treated as a separate route for current structured data. Evaluate it with query-correctness, permission, freshness, and result-limit tests rather than forcing it into document-retrieval metrics.

The goal is not to declare a universal winner. The goal is to build the evidence needed to choose the right retrieval strategy for a production RAG system.

Key takeaways

  • Evaluate retrieval and generated answers separately. A model can retrieve relevant evidence and still produce an unsupported or incorrect answer.
  • Use SQL or NL2SQL for current structured facts, and use keyword, vector, or hybrid retrieval for unstructured content. Route mixed questions across both.
  • Compare keyword, vector, and RRF hybrid retrieval against the same ground-truth question set before choosing a production default.
  • Test freshness, tenant isolation, metadata filters, exact identifiers, citations, and abstention alongside average retrieval metrics.
  • Treat hybrid search as a measured option, not an automatic winner. Keep numerical claims unpublished until they are traceable to the notebook exports.

What does production RAG look like?

Short answer: Production RAG is a measured retrieval and answer system. It needs repeatable ingestion, versioned chunks, access controls, freshness rules, citations, abstention, observability, and regression tests. Features such as BM25, vector search, RRF, reranking, HyDE, and incremental indexing only matter when you can prove they improve the answers users actually need.

A common developer question is: “I already have hybrid search, reranking, citations, and a no-hallucination policy. What else makes RAG production ready?”

The missing piece is usually the evaluation harness. Retrieval features are easy to add. Proving that they still work after a chunking change, embedding model swap, schema change, or reranker update is the harder production problem.

A production RAG system should have:

  1. A ground-truth question set with required evidence and expected answer behaviour.
  2. Retrieval metrics for keyword, vector, SQL, and hybrid paths.
  3. Answer evaluation for groundedness, correctness, citation validity, and abstention.
  4. Versioned ingestion, parsing, chunking, embeddings, prompts, and retrieval configuration.
  5. Metadata filters for tenant, permission, source, status, freshness, and document version.
  6. Observability for empty results, retrieval misses, latency, citation failures, and stale evidence.
  7. A rollback path when a new retrieval change improves the average but breaks an important query class.

Chunking needs its own tests. Short support notes, long PDFs, policy documents, tables, code, and product documentation do not share one ideal chunk size. Treat chunk size, overlap, parsing, parent document links, and table handling as versioned configuration. Then test those choices against the same question set before you publish the change.

This is the practical bar: if you cannot detect a broken retrieval change, the system is not production ready yet.

How do I improve a RAG pipeline over a sparse SQL database?

Short answer: Do not embed every row from a sparse SQL database. Route structured questions to SQL, filter null and empty fields before building searchable text, and embed only fields with useful language. Use keyword retrieval for exact IDs and status values, vector retrieval for descriptive text, and metadata filters for valid rows.

A common developer question is: “My database has many empty tables and null columns. I embedded rows, but the model retrieves poor context. How do I make the RAG pipeline efficient?”

The problem is usually not the model. The problem is that the retrieval corpus contains low-information chunks. If a row has many empty columns, turning the whole row into text gives the retriever noise that still competes for context-window space.

Separate the jobs before adding more retrieval tricks:

User question Best first route Example
Aggregations, counts, dates, filters SQL or NL2SQL “How many open high risks have no mitigation?”
Exact identifiers and controlled values Keyword plus SQL predicates “Show risk RSK-1042 with status OPEN”
Narrative similarity Vector retrieval over meaningful text “Find incidents involving delayed supplier access”
Mixed exact and semantic intent Keyword and vector candidates fused with RRF “OPEN risks similar to the supplier outage”

Create a retrieval view rather than embedding every physical row. The view should include stable IDs, required business metadata, and a deliberate search_text field built from non-empty descriptive columns.

For example, a useful retrieval record might include:

risk_id: RSK-1042
tenant_id: acme
status: OPEN
severity: HIGH
search_text: Supplier access delay caused a missed shipment window. Mitigation owner is reviewing backup routing options.

That is different from serialising twenty columns where half the fields are empty. Preserve nulls as database state for SQL reasoning. Do not turn the word “null” into semantic content unless the absence itself is the thing being searched.

For sparse relational data, evaluate routes separately before combining them:

  • SQL quality: does the generated or selected SQL return the correct rows?
  • keyword quality: do exact IDs, codes, statuses, and controlled terms match reliably?
  • vector quality: do descriptive fields retrieve semantically related incidents or risks?
  • hybrid quality: does RRF improve mixed queries without adding noisy rows?

RRF is useful when both keyword and vector result lists contain useful evidence. It is not a cleanup step for a bad corpus.

How should RAG handle real-time dynamic data?

Short answer: Frequently changing structured data should usually be queried live, not re-embedded on every update. Use SQL or NL2SQL for current rows, and reserve RAG for unstructured text that benefits from semantic retrieval. For changing documents, update affected embeddings, track freshness metadata, and test stale-versus-current answer behaviour.

A common developer question is: “My backend data updates every five minutes. Should I use RAG, SQL, caching, MCP, tool calling, or something else?”

Start by asking what kind of data needs to be fresh.

If the answer lives in current structured rows, query the database at request time. Re-embedding the full dataset every five minutes creates a constant race with the source of truth. The vector copy can become stale before the indexing job finishes.

If the answer lives in unstructured documents, use retrieval. But make freshness explicit. Store source timestamps, version IDs, current-version flags, ingestion times, and deletion state. Then evaluate whether the system chooses the current evidence instead of a stale chunk.

Use a simple routing model:

Need Route
Current account, risk, ticket, inventory, or status data SQL or NL2SQL against governed live tables
Policies, manuals, support notes, or PDFs Keyword, vector, or hybrid retrieval over indexed documents
Current facts plus explanatory documents SQL for current state, retrieval for explanation, answer composition with citations
Cross-session user preferences or agent context Scoped agent memory, not a document index

Caching, MCP, and tool calling are useful, but they solve different problems.

A cache reduces repeated work, but it must be invalidated when source data changes. MCP can expose database tools to an assistant, but it does not make stale data fresh. Tool calling lets a planner choose SQL, retrieval, or another service, but every tool still needs permissions, timeouts, result limits, and traceable outputs.

The production pattern is not “embed everything.” It is “route each question to the freshest authoritative source, then evaluate whether the answer used that source correctly.”

When should I use SQL, vector search, keyword search, or hybrid search?

Short answer: Use SQL for structured facts, keyword search for exact terms, vector search for semantic similarity, and hybrid search when the same workload contains both lexical and semantic queries. The right choice depends on query distribution, freshness needs, permission rules, latency, and measured retrieval quality.

Each retrieval path has a job.

SQL is the right first route when the question is about rows, filters, joins, dates, counts, totals, statuses, permissions, or current business state. If the data already has structure, keep using it.

Keyword search is strong when the user supplies exact terms: error codes, product names, SKUs, ticket IDs, risk IDs, function names, policy clauses, and other tokens where spelling matters.

Vector search is useful when the query and the document use different language for the same concept. It helps with paraphrases, fuzzy intent, natural-language descriptions, and vocabulary mismatch.

Hybrid search is appropriate when the workload has both patterns and both result lists contribute. A common implementation is Reciprocal Rank Fusion, or RRF. It retrieves candidates from keyword and vector search independently, then fuses ranks:

RRF score(document) = sum(1 / (60 + rank_in_result_set))

RRF avoids pretending that keyword scores and vector distances live on the same numeric scale. A document found by both routes receives contributions from both rankings. A document found by only one route can still survive if it ranks well enough.

The mistake is using “hybrid” as a default badge of seriousness. Hybrid retrieval adds query work, tuning, latency, and operational complexity. Add it when the evaluation shows it improves the questions that matter.

What metrics should I use for RAG evaluation?

Short answer: Use retrieval metrics to test whether the system finds the right evidence, then answer metrics to test whether the model uses that evidence correctly. Retrieval metrics include NDCG, MAP, recall, and precision. Answer metrics should cover groundedness, correctness, citation validity, and abstention quality.

RAG evaluation has two layers that should not be collapsed into one score.

First, evaluate retrieval. Retrieval metrics are model-independent and can run without a generation API. That makes them useful for frequent regression checks.

Metric What it tells you Useful question
NDCG@k Whether relevant documents appear high in the ranking Did the retriever put the best evidence near the top?
Recall@k How much known relevant evidence was recovered Did retrieval miss evidence the answer needed?
Precision@k How much of the returned set was relevant How much noise did retrieval add?
MAP@k Ranking quality across multiple queries Is performance consistently useful across the test set?

The notebook records retrieval results for keyword, vector, and RRF hybrid retrieval:

Method NDCG@10 MAP@10 Recall@10 Precision@10
Keyword {{KEYWORD_NDCG_10}} {{KEYWORD_MAP_10}} {{KEYWORD_RECALL_10}} {{KEYWORD_PRECISION_10}}
Vector {{VECTOR_NDCG_10}} {{VECTOR_MAP_10}} {{VECTOR_RECALL_10}} {{VECTOR_PRECISION_10}}
RRF hybrid {{HYBRID_NDCG_10}} {{HYBRID_MAP_10}} {{HYBRID_RECALL_10}} {{HYBRID_PRECISION_10}}

These placeholders must stay placeholders until the notebook has been run cleanly and the exported metrics have been reviewed. Do not turn them into claims manually.

Second, evaluate generated answers. A retrieved document can be relevant while the generated answer is still wrong, unsupported, overconfident, or badly cited.

Use answer-level checks for:

  • groundedness: does the answer stay within the retrieved context?
  • correctness: does it match the reference answer?
  • citation validity: do cited documents support the claims attached to them?
  • abstention quality: does the system refuse when the corpus does not contain enough evidence?

Model-based judging is a review signal, not ground truth. Keep the raw generated answers, retrieved IDs, reference answers, scores, and rationales so a human can inspect surprising results.

How do I test production failure modes?

Short answer: Add a production challenge set next to the clean benchmark. Include exact identifiers, paraphrases, stale and current versions, tenant isolation, metadata filtering, messy questions, multi-hop questions, and unsupported questions. These cases catch the failures that average retrieval scores often hide.

A clean benchmark is useful, but production traffic is not clean. Developers need a small challenge set that reflects the application’s actual risk.

The notebook includes ten production-style cases:

  1. An exact identifier query.
  2. A paraphrase query.
  3. A stale version query that should not use old evidence.
  4. A current version query that should prefer the latest evidence.
  5. A tenant isolation query.
  6. A messy user question with irrelevant wording.
  7. A question requiring two documents.
  8. An unsupported question where the system should abstain.
  9. A lexical entity query.
  10. An embedding-migration sequence question.

Those cases should be adapted to the domain before publication. For a risk-register system, include risk IDs, mitigation statuses, sparse rows, tenant boundaries, and null-heavy records. For a support assistant, include error codes, product versions, stale documentation, and unsupported product claims. For a real-time operational chatbot, include recently changed records and cache-invalidation cases.

The key is to keep the challenge set stable. Run it before changing chunking, embedding models, metadata filters, SQL generation, rerankers, or prompts. If a change improves the average but breaks tenant isolation or stale-data handling, it is not an improvement.

Decision guide: keyword, vector, SQL, or hybrid

Use the query type and failure cost to pick the starting route.

Your workload contains Start with Then test
Current structured facts SQL or NL2SQL Query correctness, permissions, freshness, and safe result limits
Error codes, IDs, SKUs, names, exact clauses Keyword retrieval Whether vector search improves paraphrases without losing exact matches
Natural-language questions and vocabulary mismatch Vector retrieval Exact-identifier failures and metadata constraints
Both lexical and semantic questions RRF hybrid retrieval Candidate depth, latency, and ranking gains over both baselines
Sparse relational tables SQL plus selective semantic fields Null handling, exact IDs, and route-level quality
Data that changes every few minutes Live SQL for structured data, incremental indexing for documents Stale-versus-current answer behaviour
Regulated or multi-tenant data Any method with mandatory metadata filters Isolation, auditability, deletion, and freshness
Unsupported questions Retrieval plus abstention policy False answers and false refusals

The important move is not choosing one retrieval method forever. It is making the retrieval route explicit, measuring it, and changing it only when the evidence says the system gets better.

Frequently asked questions about production RAG evaluation

When should I use hybrid search instead of vector search for RAG?

Short answer: No. Hybrid search helps when a workload contains both exact lexical queries and semantic queries, and when both candidate lists contribute relevant evidence. It can add latency and tuning work. Compare hybrid retrieval against keyword and vector baselines on the same question set before adopting it.

Can I evaluate RAG without an LLM API?

Short answer: Yes. Retrieval evaluation can run without a generation model. Use relevance judgements to calculate NDCG, MAP, recall, and precision for each retrieval method. Add answer-level evaluation later to test groundedness, correctness, citation validity, and abstention.

Should I use RAG or NL2SQL for a SQL database?

Short answer: Use SQL or NL2SQL when the answer depends on structured rows, joins, filters, dates, counts, or current state. Use RAG for unstructured documents and descriptive text. Many production applications need routing: SQL for live facts and retrieval for supporting explanations.

What is Reciprocal Rank Fusion in hybrid search?

Short answer: Reciprocal Rank Fusion, or RRF, combines independently ranked keyword and vector results by rank position. It avoids comparing raw keyword scores with vector distances directly. Documents that rank highly in either list can survive, while documents found by both methods receive contributions from both rankings.

How often should I evaluate a production RAG system?

Short answer: Run retrieval regression tests whenever chunking, parsing, embeddings, indexes, filters, rerankers, prompts, or source schemas change. Also run a scheduled production challenge set to detect corpus drift, stale evidence, permission failures, and changing user-query patterns.

Next steps

For Oracle AI Database RAG, start with the implementation and documentation that matches the route you need: