LLMs and Long-Term Memory - Benefits and Limitations

Can an assistant remember agreements between conversations?

The same requirements come up repeatedly when working on a project. I do not want to repeat the same rules every time. I tested whether an assistant could save an agreement, find it in a new conversation and apply it to the next task.

I used Hindsight as external memory: a separate information store shared by different assistants. The model is not retrained. It receives the retrieved information before preparing an answer.

How I tested memory

I connected Hermes Agent with Solar Mini4 and Open WebUI with Qwen3-8B running on my computer in LM Studio to the same memory store. Codex also helped save agreements and independently inspect memory. Some components used external services, so this was not an entirely local setup.

The main test concerned a simple rule:

Images for published articles and projects belong in the shared directory dist/assets/images. If the assistant is unsure which subfolder to choose, it should ask the user.

I requested a memory lookup, asked about an earlier decision and then assigned an ordinary task without mentioning memory. I checked whether the assistant gave the correct directory and asked for the missing decision instead of inventing one.

What worked and what caused problems

Hermes with Solar Mini4 consulted earlier agreements on its own more often than the tested Open WebUI and Qwen3-8B setup. Neither consistently applied the entire rule. Clearer instructions helped in some trials but did not resolve every problem.

View example answers and an error

Benefits and limitations

The main benefit is keeping agreements available between conversations. The trials also showed that one assistant could retrieve information saved by another. This provides a basis for reducing repeated instructions.

The main limitation is uncertainty about whether the assistant will use memory correctly. It may skip the lookup, omit an important condition or treat an earlier suggestion as an approved decision. Automatically saving entire conversations also preserved incorrect answers.

Verification was sometimes harder because an answer did not identify a specific source document. Saying “I found it in memory” is not enough to establish where the information came from or whether it still applies.

Conclusions from my project

I therefore adopted a simple workflow: the assistant checks earlier agreements, proposes saving a specific decision and waits for my approval. After saving, it reads the decision back and compares it with the approved wording. I have not yet verified the complete workflow in every application I use.

Settings need to be checked in practice

Hermes reported that automatic saving was disabled, although the file setting remained enabled. Even the conversation requesting that it be disabled was saved to memory. Documents also appeared after a delay. An assistant’s statement or the time shown in the interface is therefore not enough to establish whether a change has taken effect.

A similar issue involved LM Studio: the model was loaded, but the service making it available to other applications was stopped. After starting the service, repeated save attempts succeeded. However, I did not establish that this was the only cause of the earlier errors.

Shared memory and resource use

Once an agreement has been saved and processed, different assistants can reuse it. If it replaces a long conversation history, it may reduce the amount of text needed for a task. Preparing memory also requires processing: in our trial, Hindsight used a local model to extract the agreement. The assistant using memory must then process the retrieved result, which also includes information describing the entries. We confirmed that agreements could be shared, but did not measure token savings for the complete task. More information from memory does not automatically mean a better answer or lower resource use.

Did the advanced GPT-6 Astra model consistently use external memory?

While working with GPT-6 Astra in Codex, I noticed that this assistant did not always check Hindsight either. I had agreed with it that memory should be consulted for every new task. Yet when updating this article, it read the files, changed the text and ran checks without retrieving memory. Earlier successful calls had confirmed that access worked.

The necessary information was already in the conversation, so the assistant could complete the task without another search. However, that did not mean it followed the agreed procedure. When I pointed this out, it acknowledged skipping the lookup. We did not establish a single cause or how often this happened. This describes a specific case, not the model’s reliability across all uses.

For me, the key lesson is simple: a correct answer does not prove that external memory was used. Actual tool calls need checking, rather than relying on the assistant’s assurances. If retrieval is required, it is worth considering having the application perform it before the answer is prepared, instead of relying solely on a model instruction. This is a possible improvement that I have not yet verified in this project. Whether the retrieved information was applied correctly still needs checking.

Memory helps maintain continuity, but important decisions still need human oversight. It is worth checking not only whether information exists, but also whether it was used correctly.

What this means for information security

Incorrect or outdated information can return in later answers. A saved entry should make clear who made the decision, what it covers and whether it was approved. It must also be possible to correct or remove it.

You also need to know where the data goes. Hosting memory on your own server does not ensure complete privacy if its contents are passed to a model operated by an external provider.