Production default
Continuous batching
One multi-sequence context per model, iteration-level admission, sticky-prefix reuse, adaptive prompt-lookup speculation, and bounded streaming.
Apache 2.0Rust + C11post-v0.6 native path
Hermon Reference Engine
Local inference with a scheduler at its core.
A standalone runtime—not the Hermon Backend and not Ollama. One binary speaks OpenAI, Ollama, and Anthropic protocols. A production continuous-batching runtime serves today while Hermon takes measured ownership of paged memory, deterministic attention, and hardware-specific execution.
This repository
hermonai/hermon
Inference runtime, scheduler, memory, kernels, and protocol adapters.
Not the backend
hermonai/hermon-server
Accounts, conversations, agents, policy, and runtime routing.
Not Ollama
Independent runtime option
Hermon Backend may route to Ollama without using Hermon Engine.
Explicit preview
Hermon-owned paged KV
Real packed-GGUF Llama inference on CPU with radix prefix sharing, copy-on-write tails, Hermon attention, and greedy decoding.
Native library
Plan / execute / combine
A stable C11 ABI, native arena and block pool, deterministic attention tasks, scalar reference code, and Arm NEON dispatch.
The execution line
A clean boundary from client intent to device work.
Protocol code does not know page layouts. Kernels do not know queues. The scheduler owns global policy, and providers report cost and execute explicit plans.
01
Protocol edge
OpenAI · Ollama · Anthropic
02
Runtime policy
admit · batch · cache · cancel
03
Execution plan
model · memory · kernel tasks
04
Provider
CPU · Metal · CUDA · ROCm · SYCL
Numbers with provenance
Measured components. No imaginary multiplier.
The published Apple M3 Pro measurements use Metal, Llama-3-3B-Q4_K_M, and four concurrent requests. The often-cited 5.1× figure is the product of three component gains on a favorable workload—not a fourth end-to-end measurement.
Inspect methodology and raw tables →2.24×
continuous-batch throughput vs Ollama
41.4 vs 18.5 tok/s
1.71×
shared-prefix workload improvement
4.89 s to 2.86 s
1.33×
prompt-lookup gain on RAG-quote
51.3 to 68.1 tok/s
Inner loop
Fewer boundaries per transformer layer.
The v0.6 tensor bridge evaluates Q/K/V in one GGML graph and gate/up in another. Packed weights stay packed. Seven projection graph submissions become four, with an exact-output differential test and no unmeasured speedup claim.
RMSNorm
one normalized activation
Q · K · V
one bundled packed graph
RoPE + paged attention
GQA-aware cache traversal
Output projection
residual connection
Gate · Up
one bundled packed graph
SiLU × Up → Down
MLP residual
Engineering constitution
Performance work that remains understandable.
01
Own policy
Hermon owns admission, scheduling, cache policy, lifecycle, streaming, and observability. Kernels never decide request priority.
02
Borrow strength
A pinned llama.cpp provides production model and hardware coverage while native components earn their place behind measured gates.
03
Bulk the boundary
Projection groups, KV tiles, and attention tasks cross language and device boundaries—not individual elements in hot loops.
04
Specify determinism
Shape-derived task grids, indexed partials, ordered combine, F32 accumulation, and explicit model-equivalence corpora.
Build in public
Read the contracts. Reproduce the numbers. Challenge the design.
The repository carries the source of truth: current-vs-preview status, internal logic diagrams, kernel ABI, acceptance gates, benchmarks, and the running innovations log.