Honcho - Persistent Memory for Local LLMs

Every conversation with a local model starts from zero. You explain your setup, your preferences, the thing you were working on last week — and then the context window closes and it’s all gone. Honcho fixes that, and it runs entirely on your own hardware.

This is the full build: what Honcho is, how it works internally, and every step to get it running against local models. It also documents the things that went wrong, because most of them fail silently and none of them are in the docs.

What Honcho actually is

Honcho is a memory library for stateful agents, from Plastic Labs. The one-line version: it turns conversation into durable knowledge about the people in it.

The important thing to understand up front — Honcho is not a model. It’s a service that calls models. You give it messages, it extracts observations, stores them as vectors, and answers natural-language questions about a person later. The model doing that work is whatever you point it at.

That distinction matters because it’s what makes memory portable. Your accumulated profile lives in Postgres keyed by peer, not by model. Swap your chat model tomorrow and the new one inherits everything the old one learned.

The peer paradigm

Honcho makes no structural distinction between humans and AI agents — both are peers. Vector storage is keyed by (observer, observed) pairs, which means the same mechanism handles self-representation (observer == observed) and cross-peer modelling. Your agent builds a model of you; you build a model of it; a second agent builds its own separate model of you.

Sessions group conversations. Workspaces isolate everything.

How it works internally

Six things in Honcho call a model. Understanding which does what is essential, because they have wildly different cost profiles and only one of them sits in your latency path.

The deriver

Reads batches of messages and produces observations like:

flaviu writes technical playbooks for his homelab
flaviu publishes his playbooks on his own site
flaviu prioritises reproducible configurations over benchmark numbers

Note the shape: atomic (one fact each) and self-contained (each stands alone without context). That’s the target format, and it’s what a purpose-built deriver model is trained to produce.

The deriver is also the only component that requests structured output. Remember that — it becomes relevant when a provider rejects json_schema.

The dialectic

The recall engine. Five reasoning levels, differing in how many tool iterations and output tokens each may spend:

This is the component that will make your agent feel slow, because every one of those iterations is a sequential LLM call happening while you wait for a reply.

Dream

The most interesting component, and the one that justifies calling this a memory system rather than a log.

The deriver only captures what was said. Give it enough conversation and you accumulate a pile of atomic facts:

flaviu writes technical playbooks for his homelab
flaviu publishes his playbooks on his own site
flaviu prioritises reproducible configurations over benchmark numbers
flaviu color grades his FPV drone footage

Individually true, individually shallow. Nothing there tells an agent how to work with you.

Dream runs two reasoning passes over the accumulated conclusions during idle periods:

• Deduction — what necessarily follows from facts already stored. If you documented three separate builds and published all three, it can conclude you document systematically rather than occasionally.
• Induction — generalising patterns across many observations. The step from “prefers reproducible configs” + “writes exact commands” + “asks for the diagnostic before the theory” to a working model of how you actually think.

The output is a different kind of memory: things you never said, inferred from the shape of everything you did say. That’s what makes recall feel like understanding rather than search. Without dream, ask Honcho about yourself and it replays facts. With it, it can characterise you.

What it costs: the heaviest workload in the system — up to 20 tool iterations against a 16K history budget per run. But it’s self-gating on roughly 50 documents minimum, 60 minutes idle, and 8 hours between runs, so it costs literally nothing until you have material and then runs only when the machine is quiet.

Enable it from day one. Because of the document threshold there’s no downside — it simply won’t fire until there’s something worth consolidating. The one thing to remember is that it’s a separate model config (DREAM_DEDUCTION_MODEL_CONFIG__* and DREAM_INDUCTION_MODEL_CONFIG__*), so flipping DREAM_ENABLED=true without setting those means the dream specialists fall back to Honcho’s built-in defaults — which point at a cloud provider you probably don’t have a key for.

The build

Hardware

• Spark — NVIDIA DGX Spark, GB10, 128 GB unified memory, arm64, DGX OS
• Nostromo — Mac Studio M3 Ultra, 256 GB (added later — optional)
• Yutani — Mac running Hermes Agent

You don’t need three machines. Everything below works on one box; the split is an optimisation covered at the end.

What you need running before you start

Honcho needs two model endpoints:

1. A chat model with reliable tool calling. Community consensus is 32B+; models under 14B miss tool calls and malform output. Every Honcho agent requires OpenAI-format tool calling.
2. An embedding model. Separate from the chat model, with its own config block.

The second one surprises people. The most-referenced community guide states that local embeddings aren’t feasible and tells you to bring a cloud API key. That’s out of date — this guide runs them locally.

Part 1: Serving the models

The chat model

I’m running nvidia/Qwen3.6-35B-A3B-NVFP4 under vLLM. The critical flags:

docker run -d --name qwen36-35b --ipc=host --restart unless-stopped \
  --gpus all -p 8000:8000 \
  -e HF_HOME=/models -v ~/models:/models \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly \
  nvidia/Qwen3.6-35B-A3B-NVFP4 \
    --host 0.0.0.0 --port 8000 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --enable-prefix-caching \
    --moe-backend marlin \
    --max-model-len 65536 \
    --gpu-memory-utilization 0.4 \
    --max-num-seqs 4

Verify tool calling before you touch anything else:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"nvidia/Qwen3.6-35B-A3B-NVFP4",
  "messages":[{"role":"user","content":"weather in Timisoara?"}],
  "tools":[{"type":"function","function":{"name":"get_weather",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
  "tool_choice":"auto"}' | python3 -m json.tool | grep -c tool_calls

Non-zero means you’re good. Zero means stop and fix the parser.

Choosing an embedding model

This decision has a hard constraint most people discover too late.

pgvector caps HNSW and IVFFlat indexes at 2000 dimensions on the vector type, because every index tuple has to fit in PostgreSQL’s 8 KB page. Exceed it and you get:

ERROR: column cannot have more than 2000 dimensions for hnsw index

Honcho indexes its conclusion vectors. So anything above 2000 dims is out unless you patch the schema to use halfvec.

That rules out Qwen3-Embedding-4B (2560 dims) and most large embedders. What’s left:

I went with Jina v5 — it’s the strongest sub-1B embedder available and costs 1.4 GB. The licence is non-commercial, which is fine for a homelab and not fine for a product. Qwen3-Embedding-0.6B is the permissive alternative at about one MTEB point behind.

The 1536-dim option exists purely to avoid a schema migration. Don’t take it — the migration is one command and gte-Qwen2 is two generations old.

export HF_HOME=~/models
hf download jinaai/jina-embeddings-v5-text-small-retrieval \
  --exclude "*.gguf" "onnx/*" \
  --local-dir ~/models/jina-v5-small

Serve it:

docker run -d --name jina-embed --ipc=host --restart unless-stopped \
  --gpus all -p 8001:8000 \
  -v ~/models/jina-v5-small:/model:ro \
  vllm/vllm-openai:nightly \
  --model /model --served-model-name jina-embed \
  --host 0.0.0.0 --port 8000 \
  --runner pooling \
  --gpu-memory-utilization 0.06 \
  --max-model-len 8192

Older vLLM builds want --task embed instead of --runner pooling.

Verify the embeddings actually work

This is the step people skip, and it’s the one that fails silently. Jina v5 uses last-token pooling. If vLLM doesn’t pick that up from the model config, you get vectors that look perfectly normal and retrieve badly. No error, no warning — just a memory system that feels vague six months later.

Test against the model card’s own reference values:

curl -s localhost:8001/v1/embeddings -H 'Content-Type: application/json' -d '{
  "model":"jina-embed",
  "input":["Query: Which planet is known as the Red Planet?",
           "Document: Mars, known for its reddish appearance, is often referred to as the Red Planet.",
           "Document: Saturn, famous for its rings, is sometimes mistaken for the Red Planet."]}' \
| python3 -c "
import sys,json,math
e=[d['embedding'] for d in json.load(sys.stdin)['data']]
cos=lambda a,b: sum(x*y for x,y in zip(a,b))/(math.sqrt(sum(x*x for x in a))*math.sqrt(sum(y*y for y in b)))
print('dims  ', len(e[0]))
print('mars  ', round(cos(e[0],e[1]),4))
print('saturn', round(cos(e[0],e[2]),4))"

Expected: 1024, ~0.76, ~0.62. Mine came back 0.7599 and 0.6197.

If Mars doesn’t clearly beat Saturn, pooling is wrong. Add --override-pooler-config '{"seq_pooling_type":"LAST","normalize":true}' and retest before writing any real data.

Part 2: Honcho

Clone and configure

cd ~
git clone https://github.com/plastic-labs/honcho.git
cd honcho
cp docker-compose.yml.example docker-compose.yml

Two edits to docker-compose.yml. First, remap the API port — Honcho defaults to 8000, which is your inference server:

  api:
    ports:
      - "8100:8000"

Second, both api and deriver need to reach the host. Add to each service block:

    extra_hosts:
      - "host.docker.internal:host-gateway"

The environment file

Honcho’s config precedence is env > .env > config.toml > defaults. Every model feature is configured independently — there’s no “use this model everywhere” switch.

LLM_OPENAI_API_KEY=not-needed
AUTH_USE_AUTH=false
LOG_LEVEL=INFO

# Deriver — memory formation
DERIVER_MODEL_CONFIG__TRANSPORT=openai
DERIVER_MODEL_CONFIG__MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1
DERIVER_WORKERS=1
DERIVER_FLUSH_ENABLED=true

# Summarizer
SUMMARY_MODEL_CONFIG__TRANSPORT=openai
SUMMARY_MODEL_CONFIG__MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
SUMMARY_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1

# Dialectic — all five levels need configuring individually
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__TRANSPORT=openai
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1
DIALECTIC_LEVELS__low__MODEL_CONFIG__TRANSPORT=openai
DIALECTIC_LEVELS__low__MODEL_CONFIG__MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
DIALECTIC_LEVELS__low__MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1
# ... repeat for medium, high, max

# Honcho defaults these to tool_choice="any", which isn't in the OpenAI spec
DIALECTIC_LEVELS__minimal__TOOL_CHOICE=required
DIALECTIC_LEVELS__low__TOOL_CHOICE=required

# Dream
DREAM_ENABLED=true
DREAM_DEDUCTION_MODEL_CONFIG__TRANSPORT=openai
DREAM_DEDUCTION_MODEL_CONFIG__MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
DREAM_DEDUCTION_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1
DREAM_INDUCTION_MODEL_CONFIG__TRANSPORT=openai
DREAM_INDUCTION_MODEL_CONFIG__MODEL=nvidia/Qwen3.6-35B-A3B-NVFP4
DREAM_INDUCTION_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1

# Embeddings — LOCKED VALUES, see below
EMBEDDING_MODEL_CONFIG__TRANSPORT=openai
EMBEDDING_MODEL_CONFIG__MODEL=jina-embed
EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8001/v1
EMBEDDING_VECTOR_DIMENSIONS=1024
EMBEDDING_MODEL_CONFIG__DIMENSIONS_MODE=never
EMBEDDING_MAX_INPUT_TOKENS=8192

# Latency tuning — not context limits
DERIVER_MAX_INPUT_TOKENS=20000
DIALECTIC_MAX_INPUT_TOKENS=28000
DERIVER_REPRESENTATION_BATCH_WORK_UNIT_TARGET_TOKENS=128
DERIVER_REPRESENTATION_BATCH_TARGET_INPUT_TOKENS=128

LLM_DEFAULT_MAX_TOKENS=8000

DIMENSIONS_MODE=never stops Honcho forwarding a dimensions= parameter that vLLM may reject.

The vector dimension bootstrap

docker compose build
docker compose up -d database redis

docker compose run --rm --entrypoint /app/.venv/bin/python api -m alembic upgrade head

docker compose run --rm --entrypoint /app/.venv/bin/python api scripts/configure_embeddings.py --dry-run

Read the dry run carefully. It should say:

target dim: 1024
current:    public.documents.embedding=1536, public.message_embeddings.embedding=1536
planned operations (single transaction):
  - LOCK TABLE public.documents IN ACCESS EXCLUSIVE MODE
  - refuse if any non-null embeddings exist
  - DROP existing HNSW indices on the embedding columns
  - ALTER COLUMN embedding TYPE vector(1024) USING NULL on both tables
  - CREATE HNSW indices from snapshotted definitions

It drops and recreates the HNSW indices from snapshotted definitions, which is why 1024 works and 2560 wouldn’t. Single transaction, so a failure rolls back cleanly.

docker compose run --rm --entrypoint /app/.venv/bin/python api scripts/configure_embeddings.py --yes
docker compose up -d
curl http://localhost:8100/health

The API staying up is itself meaningful — a startup validator crashes the process if the schema and your config disagree on dimensions.

Verify reachability from inside

The image may not have curl, so use its bundled Python:

docker compose exec api /app/.venv/bin/python -c \
  "import urllib.request;print(urllib.request.urlopen('http://host.docker.internal:8000/v1/models',timeout=5).read()[:200])"
docker compose exec api /app/.venv/bin/python -c \
  "import urllib.request;print(urllib.request.urlopen('http://host.docker.internal:8001/v1/models',timeout=5).read()[:200])"

End-to-end test

curl -s -X POST http://localhost:8100/v3/workspaces \
  -H "Content-Type: application/json" -d '{"name":"test"}'

curl -s -X POST http://localhost:8100/v3/workspaces/test/sessions/chat1/messages \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"peer_id":"flaviu","content":"I run a DGX Spark at home and fly FPV drones on weekends."}]}'

docker compose logs -f deriver

Then ask it what it learned:

curl -s -X POST http://localhost:8100/v3/workspaces/test/peers/flaviu/chat \
  -H "Content-Type: application/json" \
  -d '{"query":"What hobbies does this person have?"}' | python3 -m json.tool

An answer about drones means the whole chain works: storage, extraction, embedding, search, synthesis.

Part 3: Hermes Agent

Honcho is a first-class memory provider plugin in Hermes, so this is one command:

hermes memory setup honcho

Use port 8100, not 8000. Every Honcho doc says 8000 because that’s Honcho’s default — yours is remapped.

Config lands in ~/.hermes/honcho.json:

{
  "baseUrl": "http://192.168.10.5:8100",
  "hosts": {
    "hermes": {
      "enabled": true,
      "aiPeer": "hermes",
      "peerName": "flaviu",
      "workspace": "hermes"
    }
  }
}

The wizard answers that matter

Lessons learned

This is the part that isn’t documented anywhere else.

1. Reasoning models break the deriver

Symptom:

src.utils.json_parser - ERROR - ❌ Repair failed: Expecting value: line 1 column 1 (char 0)
src.deriver.deriver - WARNING - Deriver generated zero observations for messages 8:13!
PERFORMANCE minimal_deriver_13 | llm_call_duration=26205ms | observation_count=0

That error points at JSON parsing. The actual cause is nowhere near it.

char 0 means the parser received an empty string. With --reasoning-parser qwen3, if the token budget is exhausted before the model closes its thinking block, everything stays in the reasoning channel and content comes back empty. Honcho’s default LLM_DEFAULT_MAX_TOKENS is 2500. A deriver extraction with thinking enabled blows straight through it.

The fix — config.toml:

[deriver.model_config.overrides.provider_params.extra_body.chat_template_kwargs]
enable_thinking = false

[summary.model_config.overrides.provider_params.extra_body.chat_template_kwargs]
enable_thinking = false

[dream.deduction_model_config.overrides.provider_params.extra_body.chat_template_kwargs]
enable_thinking = false

[dream.induction_model_config.overrides.provider_params.extra_body.chat_template_kwargs]
enable_thinking = false

Mount it into both containers:

    volumes:
      - ./config.toml:/app/config.toml:ro

The result:

A 20x improvement, and the failures stopped entirely.

Leave thinking on for the dialectic — that’s the path where reasoning earns its cost.

2. Thinking-off can produce better output

A side finding worth its own note. Same prompt, same model, temperature 0:

thinking=true   completion_tokens: 2799   content_chars: 2469
thinking=false  completion_tokens: 1385   content_chars: 4077

Thinking-on: under one character per token — impossible for prose, meaning ~2,200 tokens went into an invisible reasoning channel. Thinking-off produced a longer, better-structured answer from half the tokens.

Also note reasoning_content came back empty in both cases despite --reasoning-parser qwen3. On this build the reasoning lands in neither field, so token-based cost estimates look inflated and any tooling reading that field gets nothing.

3. The deriver queue silently stalls

Messages land, sessions exist, and nothing ever gets derived. The deriver logs its startup banner and then nothing at all.

Honcho batches representation work until a token threshold is reached — sensible for production, useless for a personal agent where messages trickle in. Work sits pending indefinitely, up to DERIVER_REPRESENTATION_BATCH_MAX_AGE_SECONDS (default 1800).

DERIVER_FLUSH_ENABLED=true

Defaults to false. Two independent field reports flag this as the thing that blocked them for hours. Set it on day one.

4. docker compose restart doesn’t reload .env

Use docker compose up -d --force-recreate after any environment change. restart may reuse the existing container with stale environment, and you’ll spend an hour debugging a setting that never applied.

5. Two settings can never be changed

The embedding model and its dimension. Everything else in Honcho is a restart away from being different. These two require down -v and re-embedding everything from scratch, because vectors from different models aren’t comparable even at identical dimensions.

Write it down somewhere you’ll find it:

cat > ~/honcho/LOCAL-NOTES.md << 'EOF'
LOCKED — changing either requires down -v + full re-embed:
  EMBEDDING_MODEL_CONFIG__MODEL
  EMBEDDING_VECTOR_DIMENSIONS=1024

Schema was ALTERed 1536 -> 1024 at bootstrap via
scripts/configure_embeddings.py. Must be re-run on any fresh
database BEFORE first write, or the API crash-loops on the
dimension validator.
EOF

6. Dense models are slower than sparse ones, even when smaller

The most counterintuitive result of the build.

Decode is memory-bandwidth-bound, so cost tracks active parameters and precision — not total size:

A 35B MoE with 3B active parameters is roughly 8x cheaper per token than a dense 8B. On a bandwidth-bound box, “use a smaller model for the background work” is exactly backwards.

7. I benchmarked a purpose-built deriver model. It lost.

dman1011/recall-honcho-8b is a Qwen3-8B fine-tune specialised for Honcho’s explicit-derivation step — an independent re-creation of what Plastic Labs’ proprietary Neuromancer XR does in their managed service. Apache 2.0, trained on 7,160 synthetic examples distilled from Claude Opus, 93.8% eval token accuracy.

Same input, same session, back to back:

The observations were the same six facts, differing only in phrasing:

Qwen:   flaviu engages in FPV (First Person View) drone activities on weekends
Recall: flaviu does FPV drone flying on weekends

The fine-tune is arguably tighter. It’s also 5.7x slower on this hardware, and needed three config workarounds Qwen didn’t: json_object, a lower output budget, and a larger context window.

Conclusion: a general model that’s already good at instruction-following matches a purpose-built specialist on straightforward extraction. The gap Neuromancer closes is a training data gap, not a model-selection gap.

Worth revisiting for messy multi-party conversations with ambiguous attribution — the case the fine-tune specifically claims. My test had one speaker and no dates.

8. Transcript exhaust

The quality problem nobody warns you about. My first real extraction produced eleven observations — all about the agent, none about me:

hermes is unable to access LinkedIn profiles due to restrictions in its tools
hermes can extract public fragments such as location and job title
hermes offers to store context regarding the user's personal connections

Every one is a correctly-extracted explicit fact. Every one is useless. Agents generate enormous amounts of text that is explicit but not worth remembering — troubleshooting steps, command output, status updates — and a deriver told to extract “all explicit facts” faithfully preserves all of it.

The useful test: would knowing this materially help in a fresh conversation a month from now? Good observations change future behaviour. Bad ones summarise what happened.

Config-level fixes first — set observation mode to unified so the AI peer stops observing itself. Source patches to the deriver prompt exist in the wild but are a fork you’d maintain across every update. Don’t reach for them until you have a few hundred observations and can judge the actual ratio.

9. The memory feedback loop

If your client injects recalled memory into the prompt and writes the full prompt back to Honcho, the deriver re-ingests old facts as if you just said them. A memory system deriving memories from its own memories degrades quietly.

Check whether it’s happening:

curl -s -X POST "http://localhost:8100/v3/workspaces/hermes/sessions/$SID/messages/list" \
  -H "Content-Type: application/json" -d '{}' \
  | grep -ciE 'memory-context|CONTEXT COMPACTION'

Zero is what you want.

Community forks and patches

Honcho is AGPL, and the self-hosting community has patched around three distinct problems. Worth knowing what exists before you decide to solve any of them yourself — and worth understanding the maintenance cost, because a fork is a commitment.

WZXsea/honcho-cn — provider quirks baked in

A fork aimed at Chinese providers (DeepSeek, MiniMax, Mimo), published as ghcr.io/wzxsea/honcho:shared.

What it does:

• Auto-applies json_object + thinking disabled for DeepSeek and Mimo
• Gives MiniMax a reasoning configuration suited to it
• Removes the hardcoded 1536 embedding dimension in favour of config-driven initialization
• Per-model configuration across api, deriver, summary, dialectic and dream

Why it matters even if you don’t use those providers: this is independent confirmation of two of the hardest findings in this article. Someone else hit the empty-content-from-thinking problem and the 1536 dimension assumption, decided both were structural rather than per-deployment, and patched them into a distribution. If you were wondering whether the config.toml thinking fix is a Qwen quirk — it isn’t.

Benefit: if your provider is on their list, you skip the entire debugging path in this article.

Caveat: you’re now on someone else’s image, tracking their update cadence rather than upstream’s. For a provider they support that’s a good trade. For a provider they don’t, you get nothing and lose upstream velocity. Read their auto-detection logic rather than adopting the fork wholesale — the same behaviour is achievable in your own config.toml.

brav0charlie’s field guide — observation quality

Not a fork. Three targeted source patches aimed squarely at the transcript-exhaust problem:

What it’s for: stopping your memory store from filling with a summary of its own transcript. If your observations read like “flaviu asked about X” and “hermes updated a file” rather than facts that would help a future conversation, this is the fix.

Benefits: the deterministic filter in particular is the right architecture — prompt instructions are advisory, code is not. And the prompt rewrite targets a real design gap: Honcho’s default deriver prompt optimises for completeness, which is correct for a product serving many users and wrong for a personal agent where signal-to-noise matters more than recall.

Caveats, and they’re substantial:

• These are patches to files that change upstream. Every git pull risks a conflict, and a silently-merged conflict in the deriver prompt degrades memory quality with no error.
• The filter is a judgement call encoded in code. Tune it too aggressively and you lose observations you wanted.
• Don’t reach for this early. I had roughly forty observations when I found the guide — nowhere near enough to judge what fraction were junk. Apply the config-level fix first (observation mode unified, so the AI peer stops observing itself), accumulate a few hundred observations, then read them and decide.

The honest test before forking: dump your conclusions and apply the 30-day question to each one.

curl -s -X POST http://localhost:8100/v3/workspaces/hermes/conclusions/list \
  -H "Content-Type: application/json" -d '{}' | python3 -m json.tool | less

Would knowing this materially help in a fresh conversation a month from now? If most pass, you don’t need the patches.

oangelo’s boot-time sed — now obsolete

Worth documenting because it shows how recently this got easier. A fully-local setup running Qwen3.6-27B on llama.cpp with Ollama embeddings, which patched three files on every boot:

• src/models.py — the hardcoded 1536 dimension
• migrations/ — the same value in the schema
• src/embedding_client.py — a hardcoded OpenAI base URL

Because at the time, Honcho assumed text-embedding-3-small and didn’t expose the embedding client’s base URL as config.

Current Honcho needs none of this. EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL, EMBEDDING_VECTOR_DIMENSIONS, and scripts/configure_embeddings.py cover all three. If you find a guide telling you to patch source for local embeddings, it predates those features — check the version before you follow it.

Running the model on another host

Honcho doesn’t care where the model lives. Every feature takes a __OVERRIDES__BASE_URL, so pointing it at another machine is a URL change plus some backend-specific care.

Networking

Drop host.docker.internal — that’s for reaching the Docker host, not a different machine. Use the LAN IP directly:

cd ~/honcho
cp .env .env.before-remote

# rewrite every model base URL except embeddings
sed -i '/EMBEDDING/!s|__BASE_URL=.*|__BASE_URL=http://192.168.10.2:8080/v1|' .env

grep 'BASE_URL' .env

That /EMBEDDING/! guard matters — it leaves the embedding endpoint pointed wherever it already is. Keeping embeddings local is usually right: a 677M model isn’t worth a network round-trip per write.

Verify from inside the container, not from the host shell:

docker compose exec api /app/.venv/bin/python -c \
  "import urllib.request;print(urllib.request.urlopen('http://192.168.10.2:8080/v1/models',timeout=5).read()[:200])"

No extra_hosts needed — a plain LAN address routes through the normal bridge.

llama.cpp

llama-server is OpenAI-compatible and the closest behavioural match to vLLM.

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 32768 \
  --jinja

Ollama

Convenient, but it has the sharpest edge of the three.

On the Ollama host — Linux:

sudo systemctl edit ollama

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_KEEP_ALIVE=-1"

macOS has no systemd, so it goes through launchctl — then quit and relaunch the app, since setenv only affects processes started afterward:

launchctl setenv OLLAMA_HOST "0.0.0.0:11434"
launchctl setenv OLLAMA_CONTEXT_LENGTH "32768"
launchctl setenv OLLAMA_KEEP_ALIVE "-1"

Verify the context actually took:

curl -s http://192.168.10.2:11434/api/show -d '{"model":"glm-4.7-flash:latest"}' \
  | python3 -m json.tool | grep -i "num_ctx\|context_length"

If the env var isn’t taking, bake it into a derived model:

printf 'FROM glm-4.7-flash:latest\nPARAMETER num_ctx 32768\n' > /tmp/Modelfile
ollama create glm-honcho -f /tmp/Modelfile

OLLAMA_KEEP_ALIVE=-1 matters more than it looks. Ollama unloads after ~5 minutes idle; Honcho’s deriver goes quiet for hours. Without it, every extraction after a lull pays a full model reload.

Also expect to need DERIVER_MODEL_CONFIG__STRUCTURED_OUTPUT_MODE=json_object — Ollama’s format parameter is loose JSON mode, not OpenAI Structured Outputs.

The checklist for any remote backend

Run these three before trusting it:

# 1. Exact model id — Ollama uses tags, llama.cpp often reports a filename
curl -s http://HOST:PORT/v1/models | python3 -m json.tool

# 2. Tool calling — dialectic and dream hard-require it
curl -s http://HOST:PORT/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"MODEL","messages":[{"role":"user","content":"weather in Timisoara?"}],
  "tools":[{"type":"function","function":{"name":"get_weather",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
  "tool_choice":"auto"}' | python3 -m json.tool | grep -c tool_calls

# 3. Non-empty content on a real prompt — the thinking trap again
curl -s http://HOST:PORT/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"MODEL","max_tokens":2000,
  "messages":[{"role":"user","content":"Extract facts as JSON: I fly FPV drones on weekends."}]}' \
| python3 -c "
import sys,json
m=json.load(sys.stdin)['choices'][0]['message']
print('content len  :', len(m.get('content') or ''))
print('reasoning len:', len(m.get('reasoning') or m.get('reasoning_content') or ''))"

Empty content with populated reasoning on step 3 means you need the thinking disable — and note that config.toml’s chat_template_kwargs may not survive every backend’s OpenAI compatibility layer. Ollama in particular passes through a limited parameter set. If the disable doesn’t land, the fallbacks are reasoning_effort: none, a PARAMETER think false Modelfile, or simply raising LLM_DEFAULT_MAX_TOKENS so thinking can finish and still leave room for output.

Sizing after a move

Reset the token limits to the new backend’s window, not the old one:

sed -i 's|^DERIVER_MAX_INPUT_TOKENS=.*|DERIVER_MAX_INPUT_TOKENS=12000|' .env
sed -i 's|^DIALECTIC_MAX_INPUT_TOKENS=.*|DIALECTIC_MAX_INPUT_TOKENS=16000|' .env
sed -i 's|^LLM_DEFAULT_MAX_TOKENS=.*|LLM_DEFAULT_MAX_TOKENS=4000|' .env
docker compose up -d --force-recreate api deriver

Remember that input plus output must fit the window. A 32K context with LLM_DEFAULT_MAX_TOKENS=8000 leaves 24K for the prompt, and Honcho counts the output budget against the total — I hit a 400 that read requested 8000 output tokens and your prompt contains at least 193 input tokens, for a total of at least 8193 tokens on an 8192 window. Over by one.

Backing up your memories

Everything Honcho knows lives in Postgres. The vectors are derived data — regenerable from messages — but the observations, conclusions, peer cards, and dream output are not. Lose the volume and you start from zero.

What actually needs backing up

The config files matter more than people expect. EMBEDDING_VECTOR_DIMENSIONS in particular: restore a database without knowing its dimension and the API crash-loops with no obvious cause.

One-off backup

mkdir -p ~/honcho-backups
cd ~/honcho

docker compose exec -T database pg_dump -U postgres -Fc postgres \
  > ~/honcho-backups/honcho-$(date +%Y%m%d-%H%M).dump

cp .env config.toml docker-compose.yml ~/honcho-backups/

-Fc is custom format — compressed, and pg_restore can be selective with it. Use plain SQL (pg_dump without -Fc) if you want a diffable text file instead.

exec -T disables TTY allocation. Without it the dump works interactively and fails under cron.

Scheduled

crontab -e

0 3 * * * cd /home/flaviu/honcho && docker compose exec -T database pg_dump -U postgres -Fc postgres > /home/flaviu/honcho-backups/honcho-$(date +\%Y\%m\%d).dump 2>>/home/flaviu/honcho-backups/backup.log && cp /home/flaviu/honcho/.env /home/flaviu/honcho/config.toml /home/flaviu/honcho-backups/ && find /home/flaviu/honcho-backups -name 'honcho-*.dump' -mtime +30 -delete

Two things that bite:

• % must be escaped as \% in crontab, or the line silently truncates at the first one and your dump filename becomes garbage.
• Redirect stderr to a log. A backup that fails quietly every night is worse than no backup, because you’ll believe you have one.

Verify the next morning:

ls -lh ~/honcho-backups/

A dump of a few hundred KB is normal early on. A dump of zero bytes means the command failed — check backup.log.

Restoring

cd ~/honcho
docker compose down
docker compose up -d database redis
sleep 10

docker compose exec -T database psql -U postgres -c \
  "DROP DATABASE IF EXISTS postgres_old; ALTER DATABASE postgres RENAME TO postgres_old;" postgres || true

cat ~/honcho-backups/honcho-20260729.dump \
  | docker compose exec -T database pg_restore -U postgres -d postgres --clean --if-exists

cp ~/honcho-backups/.env ~/honcho-backups/config.toml ~/honcho/
docker compose up -d
curl http://localhost:8100/health

Test the restore

An untested backup is a hypothesis. Prove it once:

# Count what you have now
curl -s -X POST http://localhost:8100/v3/workspaces/hermes/conclusions/list \
  -H "Content-Type: application/json" -d '{}' \
  | python3 -c "import sys,json;print(json.load(sys.stdin).get('total','?'), 'conclusions')"

Note the number, run a restore into a scratch environment, and confirm it matches.

Exporting in a portable form

If you want the memories in something readable rather than a Postgres dump — for inspection, migration, or peace of mind:

curl -s -X POST http://localhost:8100/v3/workspaces/hermes/conclusions/list \
  -H "Content-Type: application/json" -d '{}' \
  > ~/honcho-backups/conclusions-$(date +%Y%m%d).json

curl -s -X POST http://localhost:8100/v3/workspaces/hermes/sessions/list \
  -H "Content-Type: application/json" -d '{}' \
  > ~/honcho-backups/sessions-$(date +%Y%m%d).json

This won’t restore into Honcho directly — no import endpoint — but it’s human-readable, greppable, and survives a schema change that would break an old dump. It’s also the file to read when you do the 30-day-test audit on your observation quality.

Worth doing quarterly alongside the nightly dumps.

Updating Honcho

cd ~/honcho
docker compose exec -T database pg_dump -U postgres -Fc postgres \
  > ~/honcho-backups/pre-update-$(date +%Y%m%d).dump

docker compose down
git pull
docker compose up -d --build
docker compose ps

Take the dump first, every time. Migrations run automatically on startup, and a release can change the schema. Alembic migrations are generally forward-only — if an update breaks something, rolling back the code doesn’t roll back the database. The dump is your only exit.

Your .env and config.toml are gitignored, so they survive git pull untouched. That’s convenient and also a trap: a new release may add settings your config doesn’t have, or change a default your config was silently relying on. Skim the changelog rather than assuming your file is still complete.

After the update, confirm the two things most likely to have shifted:

curl http://localhost:8100/health
docker compose logs api --tail 30
docker compose exec deriver cat /app/config.toml

A crash-looping api after an update usually means either a migration failed or the embedding dimension validator is unhappy — check the logs before assuming the release is broken. And verify config.toml is still mounted; if the upstream docker-compose.yml.example changed and you regenerate your compose file from it, your volume mount and port remap go with it.

The build step takes several minutes on arm64 since it compiles from source.

Debugging playbook

Where to look, in order

# 1. Is anything queued or stuck?
curl -s http://localhost:8100/v3/workspaces/hermes/queue/status | python3 -m json.tool

# 2. What is the deriver actually doing?
cd ~/honcho && docker compose logs -f deriver

# 3. Turn on the detail
printf '%s\n' 'LOG_LEVEL=DEBUG' 'DERIVER_LOG_OBSERVATIONS=true' >> .env
docker compose up -d --force-recreate deriver

# 4. Can the containers reach the models?
docker compose exec api /app/.venv/bin/python -c \
  "import urllib.request;print(urllib.request.urlopen('http://host.docker.internal:8000/v1/models',timeout=5).read()[:200])"

The line that tells you everything:

PERFORMANCE minimal_deriver_56 | llm_call_duration=1730ms | observation_count=6

Symptom table

Reading the numbers

At ~6,200 tok/s prefill, time-to-first-token is prompt_tokens ÷ 6200. So DIALECTIC_MAX_INPUT_TOKENS=28000 costs ~4.5 s before a single token comes back. Those settings are latency choices, not context limits — my model’s window is 262,144 tokens.

Also: the first call after any container recreate hits a cold prefix cache. Ignore it. The second call is your real number.

The three-host split

Once it worked, the remaining problem was contention. Chat and memory sharing one GPU means Honcho’s background work competes with the reply you’re waiting for.

The fix isn’t a second container — same bandwidth, same memory pool, and you replace one scheduler that interleaves work with two that can’t see each other. The fix is a second machine:

• Spark — chat model on :8000, Honcho containers on :8100, embeddings on :8001
• Nostromo (M3 Ultra 256 GB) — llama.cpp serving Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf for all five Honcho text features
• Yutani — Hermes Agent

Same model family on both ends, so extraction quality is unchanged. The M3 Ultra also has roughly 3x the memory bandwidth of a GB10, so it may well extract faster than the Spark did.

Redirecting is a one-liner:

sed -i '/EMBEDDING/!s|__BASE_URL=.*|__BASE_URL=http://nostromo:8080/v1|' .env
docker compose up -d --force-recreate api deriver

Embeddings stay local — a 677M model isn’t worth a network hop.

Things still worth doing

Back it up. See Backing up your memories above — nightly pg_dump, config files alongside it, and one tested restore so you know the hypothesis holds.

Authenticate it. Publishing on 0.0.0.0:8100 with AUTH_USE_AUTH=false means anything on your network can read every memory and write whatever it likes:

docker compose exec api /app/.venv/bin/python scripts/generate_jwt_secret.py

Set it as AUTH_JWT_SECRET, flip AUTH_USE_AUTH=true, reissue tokens to clients. Or keep the bind on loopback and reach it over an SSH tunnel.

Think about what you’re storing. Honcho remembers everything, including details about other people who come up in conversation. It doesn’t distinguish between “prefers concise answers” and a profile of someone who never consented to being profiled. Worth deciding deliberately rather than by default — deleting a session before extraction is cheap; unwinding it afterward means hunting individual conclusions.

Reference

• Honcho documentation — official docs
• plastic-labs/honcho — source, AGPL
• elkimek/honcho-self-hosted — community quick-start
• WZXsea/honcho-cn — fork with thinking/JSON fixes baked in
• brav0charlie’s field guide — observation-quality patches and the durable-memory prompt
• dman1011/recall-honcho-8b — open deriver fine-tune
• jinaai/jina-embeddings-v5-text-small-retrieval — the embedder

Closing

The infrastructure took an afternoon. The debugging took considerably longer, and almost all of it came down to three things: a reasoning model silently eating its own output budget, a batching default tuned for production volume, and an embedding dimension that can’t be changed after first write.

None of those produce a useful error message. All three are one line of config.

What you get for it is a memory layer that belongs to you — running on hardware you own, storing observations no third party ever sees, and outliving whatever model you happen to be running this month.