Short answer: Evaluate production RAG by testing each retrieval route against the same questions. Keyword search should handle exact terms. Vector search should handle semantic matches. SQL should handle current structured facts. Hybrid retrieval should prove it improves mixed workloads. Then evaluate whether the generated answer is grounded, cited, current, permission-safe, and willing to abstain.
A RAG demo can look good with one document, one question, and one happy path. Production is different. Users ask for IDs, stale policies, tenant-specific records, live data, tables, and questions that are not actually answerable from the corpus.
The wrong move is to debate retrieval methods in the abstract. The useful move is to build an evaluation harness that makes each method earn its place.
Key takeaways:
- Evaluate retrieval before changing prompts or models.
- Treat SQL as a first-class route for current structured data.
- Do not call hybrid search a win until it beats keyword and vector baselines on your actual query mix.
What should production RAG evaluation measure?
Short answer: Production RAG evaluation should measure retrieval quality, answer quality, freshness, permission safety, and operational reliability. A high average retrieval score is not enough if the system fails exact identifiers, stale documents, tenant isolation, or unsupported questions.
Use two layers.
| Layer | What it measures | Failure it catches |
|---|---|---|
| Retrieval evaluation | Whether the right evidence appears in top-k | Wrong chunk, missing row, stale document, noisy table |
| Answer evaluation | Whether the model uses evidence correctly | Unsupported answer, bad citation, false confidence |
For retrieval, start with recall@k, precision@k, NDCG@k, and MAP@k. For answers, score groundedness, correctness, citation validity, and abstention quality.
Key takeaways:
- Retrieval metrics are useful because they can run without an LLM call.
- Answer metrics are necessary because relevant evidence can still produce a bad answer.
- Production cases need to sit next to clean benchmark cases.
How do I compare vector-only vs hybrid RAG with keyword and SQL retrieval?
Short answer: Build one question set and run every route against it. Keyword should win exact lexical queries. Vector should win paraphrases. SQL should win current structured facts. Hybrid should improve mixed intent without making exact or governed queries worse.
Use this comparison model:
| Query type | First route to test | What to measure |
|---|---|---|
| Error codes, product names, risk IDs | Keyword retrieval | Exact match, top-k position, false positives |
| Paraphrases and fuzzy intent | Vector retrieval | Semantic recall, noise, ranking quality |
| Counts, filters, current state | SQL or NL2SQL | Query correctness, permissions, freshness |
| Mixed exact and semantic intent | Hybrid retrieval | Gain over both baselines, latency cost |
Oracle AI Database supports vector search and hybrid search patterns. The Oracle hybrid-search documentation describes combining full-text and vector similarity search, including fusion approaches such as RRF.
Key takeaways:
- Compare retrieval routes against the same questions, not separate anecdotes.
- SQL is not a fallback for RAG. It is the correct path for structured current facts.
- Hybrid search should be evaluated as a tradeoff: better recall versus added complexity and latency.
What should the evaluation set include?
Short answer: Include the cases that break real systems: exact identifiers, paraphrases, stale and current versions, tenant boundaries, metadata filters, table lookups, multi-hop questions, and unsupported questions. A clean evaluation set gives clean scores and hides production risk.
A practical first set can be 50 to 100 questions. Each question should include expected evidence, expected answer behaviour, and known failure modes.
Include:
- exact IDs, error codes, SKUs, names, and model numbers
- paraphrases where the corpus uses different words than the user
- stale and current versions of the same document
- tenant and permission boundaries
- questions that require SQL, not semantic retrieval
- table and spreadsheet questions
- unsupported questions where the system should abstain
Key takeaways:
- Keep the evaluation set stable so changes are comparable.
- Add new failed production queries after triage.
- Label required evidence and forbidden evidence, not only final answers.
How do I implement this with Oracle AI Database?
Short answer: Store documents, chunks, embeddings, metadata, and structured records in the database path. Run keyword, vector, SQL, and hybrid retrieval as separate routes. Use metadata filters before generation, and export metrics after every retrieval change.
An Oracle implementation should separate the routes clearly:
- Oracle Text or keyword search for exact terms.
- Oracle AI Vector Search for semantic retrieval.
- Select AI or application-controlled SQL for current structured data.
- Hybrid search for workloads where keyword and vector evidence both matter.
- Mandatory filters for tenant, source, version, status, and permissions.
Useful docs and runnable assets:
- Oracle AI Vector Search User’s Guide
- Understand Hybrid Search
- About Select AI
- Oracle AI Agent Memory getting-started guide
- RAG evaluation notebook: oracle_rag_with_evals.ipynb
Key takeaways:
- Keep retrieval logic visible and testable.
- Apply permission and freshness filters before evidence reaches the model.
- Publish numerical claims only after the notebook or benchmark run has exported traceable results.
What should I do next?
Run the evaluation before choosing the architecture. If the system fails exact terms, improve lexical retrieval. If it fails paraphrases, improve vector retrieval or parsing. If it fails current facts, route to SQL. If it fails mixed intent, test hybrid retrieval. If it fails memory continuity, move to scoped agent memory rather than stuffing more context into the prompt.