Local LLM and RAG - Documents, Tests and Conclusions

The problem: answers must be grounded in the document

I wanted to ask questions about my own documents and receive answers that preserved their content, conditions and exceptions. A source reference alone was not enough: the model could receive an incomplete passage or misuse the information available.

In an access management procedure, omitting a condition can change the meaning of a recommendation. Even a simple question about a control name requires an exact match, rather than a similar entry. I therefore needed to assess retrieval separately from faithfulness to the source.

ISMS documents and procedures are demanding material for RAG: omitting a condition, exception or responsibility can change the meaning of an answer. Running locally can limit the transfer of content outside your own environment and increase control over data. Confidentiality still depends on service configuration, access and data storage.

The trials aimed to locate the source of errors and assess whether additional configuration delivered a benefit worth maintaining.

RAG - brief definitions

The goal was an answer faithful to my own document. RAG supplies selected passages to the model; these need to be assessed alongside the answer.

Solutions tested

I served the model through LM Studio and used Open WebUI for conversations. I tested two ways of supplying documents: a custom Qdrant integration and the RAG functionality built into Open WebUI.

Custom importer and Qdrant

The importer prepared chunks and embeddings, while an API intermediary exposed search. I checked imports, search results and answers to questions about an IAM procedure.

Build stage · Qdrant collections

Improvement stage · custom importer

RAG within Open WebUI

I then simplified document handling by using Open WebUI. This reduced the maintenance of custom integration code. I continued to assess retrieval and answer quality separately.

Open WebUI knowledge bases

Test stage · Open WebUI knowledge bases

Test methodology

The tests used Polish questions and documents. Screenshots preserve the original evidence; English wording in this article is explanatory, not a separate English-language test run.

This is a case study, not a model ranking. The first stage covered integration and an IAM procedure. In later trials, I used a short list of controls with unambiguous reference answers.

• Document: an original list of 93 control names in four groups, using unofficial Polish translations. It is neither the full standard nor an implementation guide.
• Question about an entry: the exact identifier and name of the control concerning the use of cloud services. The expected answer was A.5.23 and the name given in the list.
• Question about structure: the four groups, their counts and whether the material contains implementation instructions. Expected: A.5: 37, A.6: 8, A.7: 14, A.8: 34, and no such instructions.
• Conversation history: I started each trial in a new chat, without previous questions, answers or corrections. I did not compare these results with a conversation containing history, so I cannot assess whether retaining it would improve or worsen the answers.
• Checks: the correct passage supplied directly, a limit change followed by a return to its previous value, and the original list compared with a version using ### headings.
• Evidence: the source text, visible retrieved passages and final answer. The interface's thinking text was an additional trace, not proof of correctness.

Technical details and results

How I assess an answer

• Entry selection: A.5.23 and the exact name in the material. Selecting another entry is a substantive error.
• Identifier precision: “23.” is the list item number. “23. A.5.23” identifies the right entry but fails the exact-identifier requirement.
• Completeness: a question about groups requires all four groups and the correct counts.
• Faithfulness to the source: an unsupported additional claim is an error even if the main answer is correct. Without a full reading, claims about the whole document are not justified.

I describe results against these criteria without calculating an aggregate score.

Environment and settings

How to run a similar small test

1. Define the reference answer. Choose a document and a question with an unambiguous answer. Record the required information and claims the source does not support.
2. Keep a record. Save the question, settings, retrieved passages and answer. Mark anything you could not verify.
3. Separate the causes of error. First check whether the required content was retrieved, then whether the model used it correctly.
4. Change one element. Repeat the same question in a new conversation. If results improve, perform a check such as returning to the previous value. One success does not establish consistent reliability.

Below are a question and a list excerpt for a reading test. A retrieval test requires uploading the full document.

19. A.5.19 Bezpieczeństwo informacji w relacjach z dostawcami
20. A.5.20 Uwzględnianie bezpieczeństwa informacji w umowach z dostawcami
23. A.5.23 Bezpieczeństwo informacji przy korzystaniu z usług chmurowych

Working with documents

Evidence: incorrect and correct answers about groups

Earlier integration trials

Changes and their effects

I do not treat the K change as a confirmed fix. The collected trials also did not establish a need to change the embedding model. Any further change should address a specific, reproduced error.

Lessons from testing

• I check three things separately: whether the request executed, whether the right data was retrieved, and whether the answer preserved its meaning.
• I distinguish “not found” from “does not exist”. The selected passages limit what can be claimed about the entire document.
• I check exceptions and conditions. In the IAM procedure, omitting a condition changes the meaning of the answer even when the retrieved passage is relevant.
• I do not judge success by the number of sources. One document can supply several passages, and a correct reference does not prove a correct conclusion.
• I use small diagnostic tests. Returning to the previous value and checking with the correct passage provided more insight than further arbitrary tuning.

Conclusions from my project

The right passage is a prerequisite for a good answer, not a guarantee. The trials showed both retrieval gaps and omissions of information already present in the context. I therefore check the document, retrieved passages and answer as separate stages.

For my use case, I chose simpler document handling in Open WebUI. This was a decision about integration maintenance costs, not a general assessment of Qdrant or proof that one architecture is superior.

I did not establish that increasing K or adding headings alone improves answers. I did demonstrate specific cases of faithful reading when the necessary content was supplied. For GRC tasks, I still check important conditions directly in the source.

Implications for information security and GRC

• Problem: an answer can sound credible despite incomplete context or an omitted condition present in the source.
• Risk: using such an answer without checking it can lead to an incorrect decision or compliance assessment.
• Requirements: verify conditions and exceptions in the document before deciding. Retain the question, source version, settings, retrieved passages and answer as a verification record, marking missing data. A record of passages alone does not prove answer correctness or a search of the entire document.

My observations and other research

The following publications provide a reference point, not independent confirmation of my environment's effectiveness. I refer to the specified preprint versions.

Conclusion: the observations support the need to check answer faithfulness. They do not confirm the effectiveness of training methods, architectural superiority or the scale of effects reported by others.