CORTEXA
← Browse
arxivcs.CLcs.AIcs.LGcs.PF2026-07-15

Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel

Sietse Schelpe

We report a way to make a frozen small language model both more capable and dramatically cheaper at once, without changing any weights. Verified knowledge is deposited once as a byte-exact key-value (KV) state artifact and later restored, by graft, into a fresh inference context. The restore is bit-exact: under a pinned deterministic configuration, the grafted logits are byte-for-byte identical to a fresh computation (SHA-256 equality), with zero KL divergence and 100% argmax agreement over fifty samples. We show that own-position graft is the unique numerically exact operating point on a model with floating-point rotary encoding, and we verify byte-exactness on two model scales (12B, 31B) and two GPU targets, one through a pre-registered replay. On AIME 2025, a frozen Gemma-4-12B moves from 80.0% to 93.3% once a verified solution library is grafted, above its own 77.5% and its 31B sibling's 89.2% published anchors. On the recurring case, eight problems the base model never solves within a 401,026-token budget are answered from cached verified solutions in 61 total decode tokens, a factor of 6,574 fewer tokens and about 8,700x less energy; the capability claim proper rests on held-out transfer (7 of 7 at 31B). The same byte-exact store widens usable context from 32,768 to 2,854,766 tokens at zero extra accelerator memory, and moves byte-identical between machines of the same architecture. We describe the system at the behavior level; the engine is proprietary, and every reported number is backed by committed input and output hashes so the scoring can be re-checked without it.

View free PDFSource page

Related papers

arxivcs.LGcs.AIcs.ARcs.CLcs.PF2026-07-05

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, et al.

We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16. To our knowledge, HiFA4 is the first Ascend-HIF4-targeted design…

View free PDFSource page
arxivcs.ARcs.AIcs.CLcs.DCcs.LGcs.PF2026-07-21

BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

Fabian Waschkowski, Prabod Rathnayaka, Lukas Wesemann

Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal~4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silico…

View free PDFSource page
arxivcs.LGcs.AIcs.CL2026-07-09Cited by 2

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

Jiantong Jiang, Peiyu Yang, Rui Zhang, Feng Liu

Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In…

View free PDFSource page
arxivcs.CLcs.AIcs.LG2026-07-14

Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

Soumil Mandal

Attention-based KV cache eviction (H2O and its descendants) compresses the memory-constrained state of a long-context model by ranking tokens on accumulated attention mass, treated here as signal energy, and keeping the heaviest. On schema-dense input streams such as nested JSON,…

View free PDFSource page
arxivcs.CLcs.AIcs.HCcs.LG2026-07-10

A small language model detects behavioural faithfulness gaps that frontier judges and human raters miss

Kwan Soo Shin, In Seok Kang, Yunkyung Min, Munho Lee

Whether a language model behaves as it claims is a judgement on which independent human raters cannot agree (Fleiss kappa = 0.074). We show that a small, purpose-built instrument does better. A linear read-out of the frozen representation of a from-scratch 146-million-parameter a…

View free PDFSource page