← All posts
AI agentsAgent MemoryHarnessesHuggingFaceFunes

Stop making every agent start from scratch

One shared memory for multiple agents

Remember at the beginning of ChatGPT where for every conversation on the same topic you had to keep repeating the same information and make sure you mastered prompt engineering correctly so the model had context ?

Yeah me too, and that was annoying.

Then around April 2025, ChatGPT solved that partially with the Chat History reference. But the problem remained with Codex CLI and most other agents.

That’s one of the reasons that got me to use the Hermes Agent early on. It didn’t feel particularly special at the beginning, but after one week of using it I could already see the value of having an agent improving itself over time from my usage and memorizing things.

It quickly became my favourite agent and I ended up having it installed on my different machines and Sprites. Problem: what Hermes memorizes is bound per profile, plus a local SQLite database for session search, none of which is synced across hosts by default. Hermes’ docs recommend an external memory provider for shared memory, one of which is Honcho (see below).

In parallel, I was already using Pi, mostly for managing my local machine and VPS. I also use Ampcode, which comes with its own agent - Amp. And, finally, Claude Code, because until recently using Anthropic models with a subscription was only possible from that agent.

So I was back to square one, having to repeat myself across my fleet of agents, no way to grep which agent I had which conversation with, or, worse, having to manage skill markdown files myself. And handing off from one agent to the other is costly in tokens.

A solution ?

Then at the beginning of September, I came across this blog post from HuggingFace - Give Your Coding Agents a Memory You Own, where the author, David Corvoysier, described a situation very similar to mine and a solution: HuggingFace funes - a durable local memory from which any agent can recall:

One coding agent with FunesA coding agent sends every turn to Funes, which indexes it in a local memory. The agent asks Funes to recall and gets back verbatim passages from past sessions.agentmemoryClaude Codeyour sessionfuneslocal memoryvector + BM25 searchindex every turnrecall verbatim passages
One agent, one memory. Every turn is indexed locally as it happens; the agent recalls verbatim passages from earlier sessions instead of starting from zero.
Several coding agents sharing one Funes memoryClaude Code, Codex, pi and Hermes each index their turns into a single Funes memory and each recall passages from it, including passages written by the other agents.agentsone memoryfunesevery agent's historyin one searchable indexClaude CodeCodexpiHermesindex turnsrecall passages
Many agents, one memory. Claude Code, Codex, pi and Hermes all write to the same index and recall from each other's history, so a decision made in one is available to the rest.

Under the hood it’s a Lance dataset with a local model for embeddings, while recall fuses vector and BM25 search:

One agent turn using FunesA sequence of ten steps. The user sends a prompt. The agent decides the question touches a past decision and calls Funes recall. Funes ranks passages locally and returns three verbatim passages. The agent calls get to open the best hit with surrounding turns, then uses code tools to check the current code, and finally answers the user with a response grounded in both the past session and the code.Youthe promptCoding agentany harnessfunesrecall · get · sketchCode toolsripwire · grep · read1"why did we add a watermark?"new prompt2skill: earlier work matters → recall3recall("nightly sync incremental watermarkdecision orb replacement re-export")MCP tool call4vector + BM25 → rerank → recency53 passages, verbatimeach names agent, session, turn6get(session, turn)open the best hit in context7full turn + neighboursthe reasoning, not a summary8grep / read / ripwirecheck the code as it is today9call sites, current code10grounded answercites the session and the code
One turn, from the agent's side. A skill tells the agent when earlier work matters; it then calls recall on Funes, opens the best hit with get, and checks the code as it is today before answering. Without that nudge an agent that merely has the tool rarely reaches for it. Illustrative example: the query is modelled on the long, keyword-dense ones agents really write, and the ripwire step is whatever code tooling your agent has.

The simplicity of the setup is appealing. One binary (okay, two if you want to publish to a shared dataset, as it relies on TruffleHog to scan for secrets before anything leaves your machine) and no infrastructure to operate yourself. Your dataset can be published on a personal HF account, and it’s private by default.

Results

I have been using this memory for about two months now. Despite some initial quirks (see below), it made hopping between agents less of a broken record.

Funes is now set up on all my machines and devboxes. Some as read-only and some as read/write.

Agents on several machines sharing one memory through a Hugging Face datasetA desktop running Claude Code, Hermes, OMP and pi, a laptop running Codex, Hermes and OMP, and a VPS running pi each keep a local Funes index and publish to a shared private Hugging Face dataset at session boundaries, after a TruffleHog secrets scan. A fresh Amp orb publishes nightly. A sprite running Hermes has a read-only token and only recalls. Recall reads the dataset directly; its files are cached locally.shared memoryHF datasetprivate by defaultdesktopread / writeClaude CodeHermesOMPpifuneslocal indexlaptopread / writeCodexHermesOMPfuneslocal indexvpsread / writepifuneslocal indexspriteread-onlyHermesfunesrecall onlyamp orbsr/w · nightlyAmpfunesnightly exportpush at session start / end (Amp: nightly)recall reads the dataset (cached)secrets scan before every push
Many machines, one memory. Every host indexes locally and publishes to a private Hugging Face dataset at session start and end, after a secrets scan; Amp publishes nightly from a fresh cloud orb. Recall reads the dataset directly, with files cached on disk rather than mirrored into the local index. The sprite's token is read-only: it can recall from the whole fleet but never writes to it.

One cool thing with Funes is that funes add <agent> converts the supported agent’s existing local sessions into its spool and indexes them incrementally over time (starting with a 60s pass on the first setup, then backfilling in bounded steps on later turns), so you can recall work prior to the setup. If you are curious about where the conversations live, deja-vu has a pretty exhaustive list of where each agent stores its conversation history.

There are some CPU/GPU spikes when funes embeds new turns and when it reranks recall candidates, but that’s to be expected.

Gotchas

As with most agentic tools, the setup is a one-line install. But how do you make sure they are actually being used and being useful?

So I had one agent chart recall calls per agent. Some had never made one: their sessions went into the dataset, but nothing ever came back out.

The mistake I made was trusting the process, thinking it would magically add value to my exchanges. And to a certain extent, it did. However, I started noticing that some of my agents never reached for that memory.

Two (skill) issues:

  • My Hermes agent had a working setup through MCP tools and lifecycle hooks, but nothing to instruct the agent to use funes when earlier work matters. The Funes integration for Hermes didn’t set up a SKILL.md that the agent would pick up, so recall hits were low until I noticed. The Claude Code bundle does ship a funes skill, however.
  • One of my agent integrations (OMP), with no official Funes integration, wasn’t set up properly and wasn’t bound to the shared dataset hosted on HuggingFace, so it was indexing locally and not sharing the recalls from other agents. Without funes bind <org>/<repo> or passing a dataset, Funes defaults to local-only indexing.

Alternatives ?

Yep, there are some but each with different trade-offs:

  • claude-mem is probably the first one you’d come across. It has a different approach in that it uses an LLM to compress sessions into summaries, observations, or conclusions, unlike Funes, which stores verbatim chunks. This makes claude-mem more token-efficient, but at the cost of losing the original passage. Another notable difference is that it also tends to push the context to the session rather than it being only pulled.
  • Honcho is a memory service going the same direction as claude-mem: it stores your messages, then runs models in the background that turn them into conclusions and peer cards, so the interpretation is pre-computed instead of worked out at read time.
  • deja-vu is the closest to Funes but built differently: plain BM25 search with no model (embeddings are optional), reading the transcripts of many supported agents straight from disk, and, besides an MCP tool the agent can call, pushing recall into the session through hooks.
How four agent-memory tools differA comparison of four pipelines from source to what the agent gets back. claude-mem captures tool use through hooks, has an observer LLM compress it into observations stored in SQLite and Chroma, and serves an index the agent drills into. Honcho stores messages, then reasoning models derive conclusions and peer cards in the background, and serves curated context. deja-vu reads the transcripts agents already write, builds a BM25 index with no model, and pushes recalled passages in through hooks. Funes converts each turn into verbatim records, embeds them locally without an LLM into a Lance dataset that can live on Hugging Face, and returns passages when the agent calls recall.reads fromat write timestored asagent gets backdistill firstan LLM decides what matters while the session is writtenclaude-memplugin + workerHookstool calls and prompts,captured as you workObserver LLMcompresses each sessioninto observationsLLMSQLite + Chromaobservations, summarieskeyword + vector searchIndex firstsearch, timeline, thenfetch full detailHonchomemory serviceAPI / pluginsmessages stored first,reasoned over laterReasoning modelsDeriver + Dreamerconclude in backgroundLLMPeer representationconclusions, peer cards,summaries, messagesCurated contextreasoning plus history,or ask it directlykeep verbatimnothing is interpreted until a search asks for itdeja-vulocal, no modelTranscripts on diskwhat 43 agents alreadywrote, even pre-installIndex onlyBM25, secrets stripped,embeddings optionalno LLMLocal search indexbuilt from those files,no worker, no modelPassages, pushedhooks inject recall atstart, prompt, failureFuneslocal + HF datasetConverted sessionshooks turn every turninto verbatim recordsChunk + embedlocal models, secretsredacted, no LLMno LLMLance datasetlocal, optionally aprivate HF datasetPassages, on requestagent calls recall orget, then reasons
Where the interpretation happens. claude-mem and Honcho spend an LLM at write time to decide what matters, so the agent reads a distilled version: compact, but a summary can drop the detail you later need. deja-vu and Funes keep the original text and rank passages when asked: deja-vu with plain BM25 and recall pushed in by hooks, Funes with local embeddings and a tool call the agent has to make.
Drawn from each project's public docs at the time of writing.

note

A recent paper, Memory Tax: Dissecting the Net Value of Agent Memory, backs up the verbatim camp. Benchmarked against naive RAG over raw transcripts, most LLM-distilled memory systems score lower — the authors call this the memory tax: summarizing at write time drops details (faithfulness), rephrasing buries what’s left (accessibility), and tying the index to the model that wrote it makes recall brittle when you switch models (robustness). Their fix? Keep a direct path back to the raw turns.

Now what ?

Having your agents persist their memory can be not only useful to them, but also to you. And I’ll cover an example of how I use this dataset to generate a personal work journal next.

tip

Using Ampcode ?

If you are not, you are missing out. If you do, your orbs can benefit from Funes as well. Ask Puck - the meta agent - to set it up for you.

Mine includes a nightly Amp schedule that starts a fresh orb in a dedicated repository, where a single Node script exports every non-archived thread on the account, and Funes converts the sessions into chunk-stable turns. TruffleHog scans the push before it reaches the private Hugging Face dataset. This way, any of my projects, be it in Ampcode or not, can later recall what past sessions decided.