Flaviu Vlaicu← whois

Cybersecurity | DevOps | HomeLab | HomeAutomation

Hermes Agent Operator's Manual

The Operator’s Manual for Hermes Agent Building an AI assistant that can act, remember, and improve Operator’s Manual · Edition 3.2 · Verified against official Nous Research documentation About This Manual This manual explains how to deploy and operate Hermes Agent as a persistent “operator” — an AI system that runs continuously, uses tools, remembers context across sessions, and improves over time — rather than as a single-session chatbot. It covers architecture, installation, the core mental model, day-to-day workflows, the operator loop, common failure modes, advanced configuration (including offline skill optimization with GEPA), and a distilled set of operational lessons. ...

May 24, 2026 ·  47 min

Honcho - Persistent Memory for Local LLMs

Every conversation with a local model starts from zero. You explain your setup, your preferences, the thing you were working on last week — and then the context window closes and it’s all gone. Honcho fixes that, and it runs entirely on your own hardware. This is the full build: what Honcho is, how it works internally, and every step to get it running against local models. It also documents the things that went wrong, because most of them fail silently and none of them are in the docs. ...

August 4, 2026 ·  31 min · 
TL;DR
  • Honcho gives local agents persistent memory — conversations become durable observations about the people in them, stored in Postgres keyed by peer, not by model. Swap your chat model tomorrow and the new one inherits everything the old one learned. This is the end-to-end build against fully local models, with nothing leaving the network.
  • Six components call a model, only one is in your latency path — deriver (extraction), summarizer, dream (consolidation) and peer cards all run in the background; the dialectic runs while you wait. Tune that one first, and note its iteration ceilings aren’t monotonic — low allows 5 round-trips, medium allows 2.
  • Two settings block everything, both silent — reasoning models exhaust their token budget mid-thought and return empty content, which surfaces as Repair failed: Expecting value: line 1 column 1 and points you at JSON parsing instead of the actual cause. And DERIVER_FLUSH_ENABLED defaults to false, so small personal-agent batches never reach the threshold and sit queued for 30 minutes. Fixing the first took extraction from 37,275 ms to 1,730 ms.
  • Embeddings are the one irreversible decision — the model and its dimension are locked after first write; changing either means down -v and re-embedding everything. pgvector caps HNSW indexes at 2000 dimensions, which rules out most large embedders. scripts/configure_embeddings.py must run before any data exists, and last-token pooling fails silently if you don’t verify it.
  • Dense is slower than sparse, even when smaller — decode is bandwidth-bound, so a 35B-A3B MoE at NVFP4 reads ~2 GB/token against ~16 GB for a dense 8B at BF16. “Use a small model for background work” is exactly backwards on this hardware.
  • A purpose-built deriver fine-tune lost — recall-honcho-8b (Claude-Opus distilled, Apache 2.0) produced the same six observations as general Qwen3.6, 5.7× slower, and needed three config workarounds Qwen didn’t. Tested on straightforward single-speaker extraction; the messy multi-party case is still open.
  • Copy-paste ready — the model split across three hosts, remote-backend recipes for vLLM, llama.cpp and Ollama (including Ollama’s silent truncation), a full symptom→cause→fix table, and a backup procedure that saves your config alongside the dump, because restoring one without the other crash-loops the API.

DGX Spark + vLLM Playbook

A practical, end-to-end guide for serving LLMs with vLLM on a DGX Spark (GB10 Grace Blackwell), assembled from NVIDIA’s official playbook, the vLLM team’s deep-dive, model cards, and battle-tested community setups. ⚠️Warning Flag choices on Spark are model- and image-specific, not hardware-wide defaults. The recipes below are starting points that worked for their authors against a specific container tag. Validate against the exact image you run, and pin a known-good tag/digest for anything you depend on. Copying a flag from one model’s recipe to another can silently regress throughput or output quality. ...

June 20, 2026 ·  65 min · 
TL;DR
  • Serving LLMs on a DGX Spark (GB10) is its own discipline — 128 GB of unified memory, an sm_121 GPU that breaks half the default FP4 kernels, and flags that help one model and wreck the next. This is the end-to-end playbook
  • Right engine, right job — vLLM for concurrent serving, with the honest boundary on when llama.cpp, SGLang, or TensorRT-LLM beats it.
  • The hardware drives everything — decode is bandwidth-bound (small –max-num-seqs), NVFP4 MoE with a few-billion active params is the sweet spot, and on sm_121 FP4 MoE must run Marlin or you get !!!!!.
  • Per-model recipes that actually boot — Qwen3.6, Qwen3-Coder-Next, Qwen3.5-122B, Gemma-4 (31B/26B/coder), and Nemotron-3-Nano, each with the right container tag, quant flag, and the parser trio agents need.
  • Spark surprises you under load — single-stream looks modest, but aggregate throughput jumps from tens to hundreds of tok/s at concurrency; benchmark where you’ll actually run.
  • 2–3× single-stream for free — n-gram, MTP, and DFlash speculative decoding, easiest-first, with the sm_121 gotchas (Triton attention, BF16 KV, the images that make DFlash work).
  • Copy-paste ready — a measured prefill-vs-decode bench script (single-stream and aggregate), a full troubleshooting table, and the memory math behind “why is my RAM maxed?”

The Modern Ubuntu Bash Terminal Setup

This is a long-form, opinionated guide to setting up a terminal that’s both pretty (syntax-highlighted, themed, autosuggesting) and productive (fuzzy everything, smart history, per-project Python envs, modern replacements for the classic Unix tools). It’s everything I wish I’d known before assembling the stack — including the half-dozen subtle ordering and key-binding issues that ate a couple of evenings of my life. Target: Ubuntu 22.04 or newer, with bash as your shell. No zsh, no fish — bash all the way. The reason: it’s the default, it’s everywhere, and with ble.sh it gets ~95% of zsh’s quality-of-life features. ...

June 19, 2026 ·  21 min · 
TL;DR
  • Turn Ubuntu’s default bash into a modern terminal: ble.sh for syntax highlighting + autosuggestions, Oh My Posh for the prompt, fzf everywhere.
  • Catppuccin Frappé across the whole stack — terminal, prompt, ble.sh syntax colors, fzf — with a single .blerc + Oh My Posh JSON.
  • Modern replacements for the classics: bat, eza, fd, rg, zoxide, dust, duf, procs, glow, doggo, trip (as mtr).
  • Atuin replaces ~/.bash_history with a context-rich SQLite store on Ctrl-R; up-arrow stays normal.
  • TUIs for the rest of life: yazi (files, with y to cd on quit), lazygit, lazydocker, ncdu, btop.
  • Auto-activating Python venvs with uv + direnv and a uvenv one-shot alias.
  • Layer order matters: ble.sh source first, atuin before Oh My Posh, direnv before ble-attach, and ble-attach absolutely last.
  • Owns vs steals: don’t load tv init bash — it hijacks both Ctrl-T from fzf and Ctrl-R from atuin.
  • Includes the full .bashrc, .blerc, and ten gotchas (the prompt-disappears-after-reboot one, the silent direnv one, the errexit vs exit naming trap).

LLM Quantization

Quantization is the single most important technique for running large language models outside a datacenter. It is what turns a model that needs eight enterprise GPUs into one that runs on a gaming card, a laptop, or a Mac mini. But the moment you go to download a model, you are confronted with an intimidating wall of cryptic names — Q4_K_M, IQ3_XXS, UD-Q5_K_XL, GPTQ-Int4, AWQ, NF4, EXL3, NVFP4 — with little explanation of what they mean or which one you should pick. ...

June 18, 2026 ·  27 min · 
TL;DR
  • Bits per weight is the master variable: it sets file size and is the main predictor of quality. Every method just spends a fixed bit budget well.
  • Quality has a knee around 4–5 bits — it collapses below and barely moves above. A good 4-bit quant is the default; 8-bit+ is usually wasted memory.
  • Which method you use (GGUF, GPTQ, AWQ, NF4, EXL3, FP8/FP4) is dictated by your runtime and hardware, not a universal best. Match the format to what your stack accelerates.
  • At a fixed memory budget, a bigger model quantized harder beats a smaller one quantized lightly. Go below 4-bit only with the methods built for it.
  • The KV cache is a separate knob, and for long contexts it can outweigh the weights. Quantize it too.
  • Perplexity hides the damage: quantization hits reasoning and code hardest. Judge a quant on tasks like your real workload, not one number.

DGX Spark + LlamaCPP Playbook

Complete Setup & Operations Guide Everything needed to build, run, update, and operate local LLMs on an NVIDIA DGX Spark (GB10 / sm_121) with llama.cpp and the llm helper command. 1. How the pieces fit The Spark (GB10). Blackwell GPU at compute capability 12.1 (sm_121), 128 GB unified LPDDR5x shared between CPU and GPU, ~273 GB/s memory bandwidth. Bandwidth is the bottleneck for token generation, so Mixture-of-Experts (MoE) models with few active parameters run far faster than dense models of the same total size. Prefer MoE. ...

June 17, 2026 ·  28 min · 
TL;DR
  • Build llama.cpp for the GB10 (sm_121) with LLAMA_OPENSSL=ON and the 121a native-FP4 target.
  • Serve any GGUF model over an OpenAI-compatible API with one command: llm run <model> [port].
  • All the Spark tuning is baked in — --no-mmap, flash-attention, q8_0 KV cache, batch 2048, 20 threads.
  • 121a adds native FP4 (MXFP4/NVFP4) speedups; it’s neutral on standard quants like Q8_0 and Q4_K_M.
  • Prefer MoE models: the Spark is memory-bandwidth-bound, so low active-parameter models run fastest.
  • Manage everything with the llm helper: run, stop, ps, ls, wait, test, speed, log, update.
  • Wire Hermes or Open WebUI to http://:/v1; runnable = GGUF + supported arch + ≤ ~200B.
  • Includes the full llm script, a cheatsheet, and a troubleshooting table.

Minisforum A2

I bought a Minisforum MS-A2, lived with it for months, modified most of it, pushed it harder than most people will, and then sold it. This review is the long answer to why, and it isn’t a clean recommendation either way. The MS-A2 is one of the most impressive small machines you can buy. It’s also one I’d never put on my desk or in my living room. I’ll explain how both of those are true. ...

June 12, 2026 ·  23 min

Q-feeds

Q-Feeds delivers curated indicators of compromise (IPs and domains) on a schedule. The OPNsense plugin is purpose-built to consume the IP feeds, and the official documentation assumes you’ll feed the domain side into Unbound. If you’re running AdGuard Home as your primary DNS resolver instead of Unbound — as I am — that integration path doesn’t apply directly, and you have to wire the domain feeds in manually. A two-layer threat intelligence setup is only as good as the DNS path that feeds it. This post walks through wiring Q-Feeds into OPNsense (IP layer) and AdGuard Home (DNS layer), and then — the part that turned out to matter most — actually forcing every device on the network to use that DNS path, instead of just offering it. ...

May 7, 2026 ·  25 min