AI agent and local model - practical comparison

Why compare two variants?

Working with documentation naturally raises questions about AI automation and support. In practice, at least two approaches appear:

• cloud variant - an agent in an external service, easy to start, document integration, billing tied to the service model and its token-based limits,
• local variant - a model running on my own computer (here: Ollama and Mistral 7B), generation without vendor token limits, but with hardware, time, and energy costs.

I selected the technology only after defining the comparison question and objective. In my laboratory, the critical factors are data privacy, control over the environment, and the ability to work without an external provider. This does not mean the cloud is bad; it simply is not always the best fit for a given context or set of requirements.

Cloud and local variant - quick comparison

This comparison reflects the typical trade-offs of both approaches from the perspective of a practical deployment. It is not a universal statement of technical truth, but rather an illustration of the compromises that arise when privacy, control over the environment, and independence from external services are priorities.

Locality does not automatically mean “safer in every respect.” It means a different trust boundary: less dependence on an external API and more responsibility for your own system. Cloud remains important when the objective is convenience, scalability, or enterprise integration, and it must be governed with proper controls such as ISO/IEC 27001:2022 clauses A.5.23 and A.8.1.

Business use and compliance with ISO 27001 and legal requirements

Key requirements and controls when using cloud AI for business purposes:

• Business / Enterprise / API only: avoid free AI tools when company data is involved.
• Confidentiality clause in the contract: a formal agreement with the AI provider that protects prompts and input data.
• Employee policy: clear rules on what may or may not be pasted into AI, including customer data, source code, or financial information.
• Location control (GDPR): know where data is processed and stored, and whether it is within the EU.
• Secure access: mandatory MFA for AI accounts and clear administrative ownership.
• Exit plan: a process for data recovery, moving to another vendor, and business continuity.

That is not everything, but it is the minimum that should be verified before using AI tools with company data.

Fine-tuning, RAG, and limited context

When adding knowledge to a model, it is easy to confuse three different approaches:

• Fine-tuning - modifying model weights using your own data. Expensive and complex; if processes change significantly, the process must be repeated. I did not do this in this project.
• RAG (Retrieval-Augmented Generation) - the system retrieves the most relevant document fragments for a question and adds them to the prompt. Useful for large documentation sets, but requires indexing and pipeline work. There is no full RAG here.
• Limited context + local generation. In this project, I build a narrow knowledge base from HTML, Python passes it to Ollama together with the prompt, the result is written to JSON, and the page only displays it. This is the approach used here.

A limit of roughly 12,000 characters of context comes from practical experience. As the documentation grows, the natural next step would be RAG, not necessarily fine-tuning immediately.

Architecture

Documentation limited context Ollama Python JSON verification SecureHaveNET

The public site does not connect to the model. Generation is local, and the demonstrator reads a prepared cache. That means I do not expose Ollama to the internet and I do not build a public generation API.

Environment and model

• Hardware - AMD Ryzen 7 7435HS, 16 GB RAM, NVIDIA GeForce RTX 4060 GPU (8 GB)
• Model: mistral:latest in Ollama (typically Mistral 7B; the exact version depends on the tag downloaded at the time)
• Temperature: 0.2 - lower randomness, more predictable responses; this is not a measure of risk or compliance
• Context in the script: num_ctx = 8192, timeout 300 s, atomic JSON write after each response

How does the demonstrator work?

Select a question to see a response generated locally. The page interface only loads a prepared JSON file - without connecting to Ollama in the browser.

What can we conclude from the demonstrator?

Generation times come from measurements in the script.

Note: the shorter the response time for a question, the better - as long as correctness, consistency, and factual alignment are preserved.

Average time for this set: about 12.56 s per question (from about 4.92 s to about 22.38 s). This gives a realistic picture of local model performance in this specific pipeline and on this hardware.

Quality conclusion (Check): the measurements show that response time depends on query complexity and prompt structure. However, a fast answer does not automatically mean substantive correctness. Questions about technical parameters, such as the role of temperature 0.2, require critical review because the model can be too cautious or imprecise. The verification phase must therefore cover not only hardware parameters but also business and audit validation of the content.

Risks that local execution does not eliminate

• access to the computer and account running Ollama,
• quality and timeliness of the knowledge base,
• inaccurate or overconfident model responses,
• the temptation to treat responses as approved procedures,
• logs, cache, and JSON files containing sensitive content,
• future connection to the internet or internal systems without a renewed risk analysis.

Human-in-the-Loop

LLM proposal verification human decision

The model can speed up organization of knowledge and help identify risks. It does not bear responsibility for procedure compliance, quality decisions, or security. The answer about temperature in the demonstrator shows that even with a simple setup the model can be imprecise - which is why verification is not decorative, but essential.

How I built the solution

Define the problem risk architecture aligned with the objective implementation testing correction my decision

• Plan - compare the local variant and the cloud variant in terms of generation limits, privacy, and costs.
• Do - Ollama, Mistral 7B, Python, limited HTML context, JSON, and a static page.
• Check - solution operation, generation times, the quality of the demonstrator responses, and their qualitative verification.
• Act - including a 300 s timeout, context limit, atomic cache writes after each response, and a review procedure for generated content before it is accepted as binding guidance.

Comparison: Ollama (Mistral 7B), Gemini, and Grok

The same seven questions, similar system profile (SecureHaveNET, GRC, local inference). Local measurements come from the script and Ollama on the project hardware. Cloud results (Gemini, Grok) are separate runs used as a benchmark.

* Cloud time is not a perf_counter measurement on an Ollama workstation, so these are not identical laboratory conditions. The local measurements are hard technical readouts produced on my hardware, while cloud times are based on external vendor responses. This is an intentional comparison of extreme environments, not a processor benchmark under the same conditions.

Conclusion: the cloud variant wins on latency and often produces shorter, more condensed answers. The local variant wins on control and privacy of document and query processing. “Faster” does not automatically mean “better,” and a longer local answer does not automatically mean higher precision.

Conclusions

1. It is possible to build a local assistant using your own content without a public model API.
2. The local variant and the cloud variant solve a similar need, but in my case the local solution has the advantage in privacy, full control, cost predictability, and independence from subscriptions.
3. No vendor token limit does not mean free generation: the hardware, computation time, energy, and responsibility for the execution environment remain.
4. In this project, I did not fine-tune the model and did not build full RAG - I used a limited context. This was a conscious trade-off between simplicity and control.
5. Model responses can be helpful but also imprecise. That is why Human-in-the-Loop remains necessary, whether the model is cloud-based or local.
6. This is a demo project, not a complete system for all organizational documentation. However, in my lab, the local variant remains a practical and sensible choice.
7. Deploying a local agent is not only a technical decision (GPU, Ollama), but also a Governance, Risk and Compliance (GRC) decision - transferring responsibility for log security, continuity of operation, and result verification to the internal team.

Summary

Each approach has its strengths and limitations. In cloud conditions, the agent typically wins on ease of use, integration scalability, and a lower entry barrier. However, in my usage model - lab environment, privacy, full control over the environment, independence from subscriptions, and the ability to influence each part of the process - the local path is more practical and better aligned with the requirements.

• control over data and queries - generation remains on my own hardware,
• predictability of generation limits - instead of provider tokens, the machine resources and time apply,
• ability to run an end-to-end mechanism - from prompt and context through timing to a deliberate separation of generation from the public layer,
• simpler exposure boundary - the public page only reads JSON without exposing the model to the network,
• flexibility for experimentation - control over model, temperature, context, and the generation start time.

The limitations of the local variant are also clear: slower response time, environment maintenance, lack of ready-made cloud integrations, and the responsibility for keeping knowledge up to date and information secure.

The key lesson remains common to both paths and aligned with the GRC mindset:

Sources and reference documents

• Ollama - local runtime environment for large language models.
• mistral:latest - the Mistral 7B model in the Ollama library.
• Regulation (EU) 2024/1689 (AI Act) - EU legal framework for AI systems, including transparency and risk management.
• Large language model - encyclopedia overview of language models.
• NIST AI Risk Management Framework (AI RMF 1.0) - AI risk management framework from NIST.
• ISO/IEC 27001:2022 - Information security, cybersecurity and privacy protection - Information security management systems - Requirements
• ISO/IEC 27002:2022 - Information security, cybersecurity and privacy protection - Information security controls.

The responses in the demonstrator were generated automatically by a local language model. In line with transparency principles, AI-generated content requires human verification and does not constitute an independent legal, quality, or decision-making recommendation. The reference to the AI Act concerns EU regulatory frameworks; this demonstration project is not an assessment of compliance for any specific system under the regulation.