https://haoran-qiu.com/publication/icml-2026/
Beyond Prediction: Tail-Aware Scheduling for LLM Inference | Haoran Qiu | Microsoft AzRS
May 1, 2026 - LLM serving exhibits extreme length variability, making size-based scheduling difficult in practice. Recent LLM schedulers approximate SJF/SRPT using predicted...
for llm