Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

arXiv:2611.05611v1 Announce Type: new Abstract: Speculative decoding (SD) speeds up large language model (LLM) serving, but its gain depends on the workload's \emph{shape}, such as batch size and context length, and on its \emph{content}. A model-based drafter proposes a short block on any content,

arXiv cs.AIDonnerstag 06:00 · 05.11.1 Min. Lesezeit