Research Article
Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud
Himel Ghosh*
Issue:
Volume 14, Issue 3, September 2026
Pages:
56-67
Received:
21 December 2025
Accepted:
4 January 2026
Published:
12 October 2026
DOI:
10.11648/j.iotcc.20261403.11
Downloads:
Views:
Abstract: This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making real-time applications such as conversational agents and translation services difficult to deploy cost-efficiently. The report first summarizes the cold-start problem and common mitigation strategies in today's Function-as-a-Service (FaaS) platforms, including container warming, host-side checkpoint caching, and cluster-level storage servers. It then reviews the design of ServerlessLLM, which exploits underutilized multi-tier storage in GPU servers, a loading-optimized checkpoint format, live inference migration based on token-level state transfer, and a startup-time-aware model scheduler. Additional methods from recent literature, such as RainbowCake's layered container caching, and the cold-start handling mechanisms of major cloud providers are also discussed. Drawing on the evaluation presented in the original work, the report highlights that multi-tier checkpoint loading and live migration reduce model startup time by up to 6-8× compared to conventional formats, and deliver up to 28× lower startup latency than Ray Serve for large models in realistic traces. The review concludes by outlining open challenges and future directions, including integrating model compression, multi-level inference pipelines, and advanced memory-offloading techniques to further improve the scalability and efficiency of serverless LLM inference.
Abstract: This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making...
Show More
