<div id="readability-page-1" class="page"><div id="post-content"> Running an AI agent locally means picking the right model for your hardware. I run <a href="https://github.com/NousResearch/hermes-agent">Hermes Agent</a> on an RTX 4090 (24 GB VRAM) alongside an embedding model for RAG. That shared VRAM budget turns model selection into a constraint satisfaction problem: quality, speed, and memory all compete.
I built <a href="https://github.com/jaigouk/gpumod">gpumod</a> to manage this. It tracks VRAM allocations, switches between service configurations, and simulates resource requirements before deployment. Here’s how I used it to pick the right model.
<h2 id="the-candidates"><a aria-label="Link to this section" href="#the-candidates">The Candidates</a></h2>
I started with three architectures:





























<table><thead><tr><th>Model</th><th>Architecture</th><th>Quant</th><th>VRAM</th></tr></thead><tbody><tr><td>Qwen3.6-27B</td><td>Dense (27B active)</td><td>Q4_K_M</td><td>~18 GB</td></tr><tr><td>Qwen3.6-35B-A3B</td><td>MoE (3B active)</td><td>UD-Q4_K_S</td><td>~22 GB</td></tr><tr><td>Gemma 4 E4B</td><td>Dense (~4B)</td><td>BF16</td><td>~16 GB</td></tr></tbody></table>
The MoE architecture is interesting here. Qwen3.6-35B-A3B has 35 billion parameters total but only activates 3 billion per token. That means the knowledge capacity of a much larger model with the inference speed of a small one.
<h2 id="benchmarking"><a aria-label="Link to this section" href="#benchmarking">Benchmarking</a></h2>
Each model ran through 15 iterations of a 5-level coding benchmark: basic queue implementation, retry logic, priority scheduling, concurrency bug fixing, and multi-file refactoring. Real pytest tests validated every response.
<pre tabindex="0" data-language="plaintext"><code>gpumod start qwen36-35b-a3b-q4
python scripts/run_qwen36_benchmark.py --model qwen36-35b-a3b</code></pre>
Results:

































<table><thead><tr><th>Model</th><th>Quant</th><th>Mean Score</th><th>Std Dev</th><th>TPS</th></tr></thead><tbody><tr><td>Qwen3.6-35B-A3B</td><td>Q4_K_S</td><td>90.0</td><td>0.0</td><td>173.7</td></tr><tr><td>Gemma 4 E4B</td><td>BF16</td><td>88.3</td><td>6.5</td><td>82.9</td></tr><tr><td>Qwen3.6-27B</td><td>Q4_K_M</td><td>80.3</td><td>6.9</td><td>46.9</td></tr></tbody></table>
The 35B-A3B MoE scored 90/100 on every single iteration — zero variance. It was also 2x faster than Gemma 4 and 3.7x faster than the 27B dense model. MoE wins on every metric.
<h2 id="the-vram-problem"><a aria-label="Link to this section" href="#the-vram-problem">The VRAM Problem</a></h2>
The Q4_K_S quant uses ~22 GB. My embedding model (Qwen3-Embedding-0.6B) needs ~2.5 GB. Total: 24.5 GB on a 24 GB card. That’s 64 MB of headroom — not enough once you account for CUDA context overhead, KV cache growth, and runtime allocations.
gpumod’s simulate mode caught this before I tried it live:
<pre tabindex="0" data-language="plaintext"><code>gpumod simulate hermes-agent \
 --remove gemma4-e4b-bf16 \
 --add qwen36-35b-a3b-q4

# Result: fits=true, headroom=64 MB ← technically fits, practically unusable</code></pre>
<h2 id="iq4_xs-the-smaller-quant"><a aria-label="Link to this section" href="#iq4_xs-the-smaller-quant">IQ4_XS: The Smaller Quant</a></h2>
I ran the same benchmark on the IQ4_XS quantization, which saves about 1 GB of VRAM:


























<table><thead><tr><th>Quant</th><th>Mean</th><th>Std Dev</th><th>TPS</th><th>VRAM</th></tr></thead><tbody><tr><td>Q4_K_S</td><td>90.0</td><td>0.0</td><td>173.7</td><td>~22 GB</td></tr><tr><td>IQ4_XS</td><td>87.3</td><td>10.3</td><td>174.5</td><td>~21 GB</td></tr></tbody></table>
IQ4_XS had one outlier iteration (50/100 where the basic queue and concurrency bug-fix levels both failed), but 14 of 15 iterations scored 90. TPS was identical. With the embedding model, total VRAM lands at ~23.5 GB — workable headroom.
The means are within one standard deviation of each other compared to Gemma 4 (87.3 vs 88.3), so quality is comparable, but at 2x the speed.
<h2 id="setting-up-hermes-agent"><a aria-label="Link to this section" href="#setting-up-hermes-agent">Setting Up Hermes Agent</a></h2>
With the model chosen, configuration was straightforward.
Hermes Agent’s system prompt with ~120 MCP tools plus ~36 built-in tools consumes roughly 26K tokens for tool definitions alone. I initially ran at 65K context, but multi-tool tasks (web searches + doc reads) overflowed within a single turn. The fix was twofold: 128K context with Q8_0 KV cache quantization, and per-platform tool filtering for Telegram.
The hybrid DeltaNet architecture is key here — only 10 of 40 layers maintain a KV cache, so 128K context with Q8_0 uses the same VRAM as 65K with FP16. Just 58 MB more in practice:
<pre tabindex="0" data-language="yaml"><code># systemd unit
ExecStart=llama-server \
 --model Qwen3.6-35B-A3B-UD-IQ4_XS.gguf \
 --port 7099 \
 --ctx-size 131072 \
 --n-gpu-layers -1 \
 --cache-type-k q8_0 \
 --cache-type-v q8_0 \
 --jinja</code></pre>
Hermes config points to the local endpoint:
<pre tabindex="0" data-language="yaml"><code># ~/.hermes/config.yaml
model:
 default: Qwen3.6-35B-A3B-UD-IQ4_XS.gguf
 provider: custom
 base_url: http://localhost:7099/v1
 context_length: 131072</code></pre>
For Telegram, I also configured <code>platform_toolsets</code> to load only essential MCP servers (gpumod, outline, tavily), dropping from 157 to 87 tools. This brought first-turn token usage from 49.6% down to 20% of context:
gpumod manages the mode so both services start together:
<pre tabindex="0" data-language="yaml"><code># gpumod modes/hermes-agent.yaml
id: hermes-agent
name: Hermes Agent Mode
services:
 - qwen36-35b-a3b-iq4xs
 - vllm-embedding-code</code></pre>
One command to switch: <code>gpumod switch hermes-agent</code>.
<h2 id="what-i-learned"><a aria-label="Link to this section" href="#what-i-learned">What I Learned</a></h2>
MoE architecture is the sweet spot for local inference. 35B total parameters give you the knowledge base; 3B active parameters give you the speed. The Q4_K_S quant was perfect — zero variance — but didn’t fit alongside other services. IQ4_XS is the practical choice when VRAM is shared.
Benchmark before you commit. Gemma 4 looked like the safe choice at 16 GB, but the MoE model beats it on quality and speed while only needing 5 GB more. Without benchmarking, I would have picked the wrong model.
Tool count matters for context. 157 tools in a system prompt consumed 49.6% of 65K context before the user said anything. The fix was two-pronged: double the context to 128K with Q8_0 KV cache (VRAM-neutral on the hybrid architecture), and filter tools per platform so Telegram only loads what it needs. Plan your context budget around your tool surface area, not conversation length.
The full benchmark results and methodology are on the <a href="https://jaigouk.com/gpumod/benchmarks/20260423_qwen36_gemma4_comparison/">benchmark results page</a>, with raw data and scripts in the <a href="https://github.com/jaigouk/gpumod/tree/main/docs/benchmarks/20260423_qwen36_gemma4_comparison">GitHub repository</a>. </div></div>

Jaigouk.Kim

How I benchmarked Qwen3.6, Gemma 4, and multiple quantizations on an RTX 4090 to find the best local model for Hermes Agent — and how gpumod made the whole process manageable.

Picking a Local LLM for Hermes Agent with gpumod

댓글