Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud

Himel Ghosh

doi:doi:10.11648/j.iotcc.20261403.11

Research Article |

| Peer-Reviewed

Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud

Himel Ghosh^*

Published in Internet of Things and Cloud Computing (Volume 14, Issue 3)

Received: 21 December 2025 Accepted: 4 January 2026 Published: 12 October 2026

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making real-time applications such as conversational agents and translation services difficult to deploy cost-efficiently. The report first summarizes the cold-start problem and common mitigation strategies in today's Function-as-a-Service (FaaS) platforms, including container warming, host-side checkpoint caching, and cluster-level storage servers. It then reviews the design of ServerlessLLM, which exploits underutilized multi-tier storage in GPU servers, a loading-optimized checkpoint format, live inference migration based on token-level state transfer, and a startup-time-aware model scheduler. Additional methods from recent literature, such as RainbowCake's layered container caching, and the cold-start handling mechanisms of major cloud providers are also discussed. Drawing on the evaluation presented in the original work, the report highlights that multi-tier checkpoint loading and live migration reduce model startup time by up to 6-8× compared to conventional formats, and deliver up to 28× lower startup latency than Ray Serve for large models in realistic traces. The review concludes by outlining open challenges and future directions, including integrating model compression, multi-level inference pipelines, and advanced memory-offloading techniques to further improve the scalability and efficiency of serverless LLM inference.

Published in	Internet of Things and Cloud Computing (Volume 14, Issue 3)
DOI	10.11648/j.iotcc.20261403.11
Page(s)	56-67
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2026. Published by Science Publishing Group

Keywords

Serverless Inference, Cold-start Latency, Large Language Models (LLMs), Multi-tier Storage, Checkpoint Optimization, GPU Scheduling

References

[1]	I. E. Akkus et al., "SAND: Towards High-Performance Serverless Computing," in USENIX Annual Technical Conference, pp. 923–935, 2018. Available: https://www.usenix.org/conference/atc18/presentation/akkus Accessed: Jan. 1, 2025.
[2]	Anyscale, "Loading LLaMA-2 70B 20× Faster With Anyscale Endpoints," 2025. Available: https://www.anyscale.com/blog/loading-llama-2-70b-20x-faster-with-anyscale-endpoints Accessed: Jan. 1, 2025.
[3]	AWS, "Amazon CloudWatch," 2025. Available: https://aws.amazon.com/cloudwatch/ Accessed: Jan. 1, 2025.
[4]	Microsoft Azure, "Azure Tips and Tricks: Tip 260," 2025. Available: https://microsoft.github.io/AzureTipsAndTricks/blog/tip260.html Accessed: Jan. 1, 2025.
[5]	Microsoft Azure, "Improving Azure Functions Cold Starts," 2025. Available: https://techcommunity.microsoft.com/t5/apps-on-azure-blog/our-latest-work-to-improve-azure-functions-cold-starts/ba-p/4164500 Accessed: Jan. 1, 2025.
[6]	Google Cloud, "Configuring Warmup Requests," 2025. Available: https://cloud.google.com/appengine/docs/legacy/standard/go111/configuring-warmup-requests Accessed: Jan. 1, 2025.
[7]	Google Cloud, "Instance Autoscaling," 2025. Available: https://cloud.google.com/run/docs/about-instance-autoscaling Accessed: Jan. 1, 2025.
[8]	Google Cloud, "Mitigating Cold Starts on Cloud Run," 2025. Available: https://medium.com/google-cloud/3-solutions-to-mitigate-the-cold-starts-on-cloud-run-8c60f0ae7894 Accessed: Jan. 1, 2025.
[9]	J. Daly, "Lambda Warmer," 2025. Available: https://github.com/jeremydaly/lambda-warmer Accessed: Jan. 1, 2025.
[10]	Dashbird, "Dashbird Serverless Monitoring," 2025. Available: https://dashbird.io/ Accessed: Jan. 1, 2025.
[11]	D. Du et al., "Catalyzer: Sub-millisecond Startup for Serverless Computing with Initialization-less Booting," in Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '20), pp. 467–481, 2020, https://doi.org/10.1145/3373376.3378512
[12]	X. Liu et al., "FaaSLight: General Application-level Cold-start Latency Optimization for Function-as-a-Service in Serverless Computing," ACM Transactions on Software Engineering and Methodology, 2023, https://doi.org/10.1145/3585007
[13]	Y. Fu et al., "ServerlessLLM: Low-Latency Serverless Inference for Large Language Models," arXiv:2401.14351, 2024. Available: https://arxiv.org/abs/2401.14351 Accessed: Jan. 1, 2025.
[14]	M. Ghorbian and M. Ghobaei-Arani, "A Survey on the Cold Start Latency Approaches in Serverless Computing: an Optimization-Based Perspective," Computing, 2024, https://doi.org/10.1007/s00607-024-01335-5
[15]	M. Golec, G. K. Walia, M. Kumar, F. Cuadrado, S. S. Gill, and S. Uhlig, "Cold Start Latency in Serverless Computing: A Systematic Review, Taxonomy, and Future Directions," ACM Computing Surveys, vol. 57, no. 3, pp. 1–36, 2024, https://doi.org/10.1145/3700875
[16]	IBM, "What Is Serverless Computing?" 2025. Available: https://www.ibm.com/topics/serverless Accessed: Jan. 1, 2025.
[17]	K. Kaffes et al., "Centralized Core-granular Scheduling for Serverless Functions," in Proceedings of the ACM Symposium on Cloud Computing, pp. 158–164, 2019, https://doi.org/10.1145/3357223.3362709
[18]	P.-M. Lin and A. Glikson, "Mitigating Cold Starts in Serverless Platforms: A Pool-Based Approach," arXiv:1903.12221, 2019. Available: https://arxiv.org/abs/1903.12221 Accessed: Jan. 1, 2025.
[19]	G. McGrath and P. R. Brenner, "Serverless Computing: Design, Implementation, and Performance," in 2017 IEEE 37th International Conference on Distributed Computing Systems Workshops (ICDCSW), pp. 405–410, 2017, https://doi.org/10.1109/ICDCSW.2017.36
[20]	A. Mohan et al., "Agile Cold Starts for Scalable Serverless," in 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19), 2019. Available: https://www.usenix.org/conference/hotcloud19/presentation/mohan Accessed: Jan. 1, 2025.
[21]	E. Oakes et al., "Rapid Task Provisioning with Serverless-Optimized Containers," in USENIX Annual Technical Conference (ATC 18), pp. 57–70, 2018. Available: https://www.usenix.org/conference/atc18/presentation/oakes Accessed: Jan. 1, 2025.
[22]	G. Persson and W. Branth Sj"oberg, "Mitigating Serverless Cold Starts Through Predicting Computational Resource Demand: Predicting Function Invocations Based on Real-Time User Navigation," bachelor's thesis, J"onk"oping University, 2023.
[23]	S. Rajbhandari et al., "ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning," in SC21: International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, https://doi.org/10.1145/3458817.3476205
[24]	Sedai, "AWS Autonomous Concurrency," 2025. Available: https://www.sedai.io/blog/the-end-of-cold-starts-autonomous-concurrency-for-aws-lambda Accessed: Jan. 1, 2025.
[25]	Y. Sheng et al., "FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU," arXiv:2303.06865, 2023. Available: https://arxiv.org/abs/2303.06865 Accessed: Jan. 1, 2025.
[26]	S. Shillaker and P. R. Pietzuch, "A Provider-Friendly Serverless Framework for Latency-Critical Applications," 2018. Available: https://api.semanticscholar.org/CorpusID:51997872
[27]	SlideShare, "Become a Serverless Black Belt: Optimizing Your Serverless Applications," 2025. Available: https://www.slideshare.net/slideshow/become-a-serverless-black-belt-optimizing-your-serverless-applications-srv401-reinvent-2017/840310167 Accessed: Jan. 1, 2025.
[28]	Snowflake, "LLM Inference Guide," 2025. Available: https://www.snowflake.com/guides/llm-inference/ Accessed: Jan. 1, 2025.
[29]	P. Vahidinia et al., "Cold Start in Serverless Computing: Current Trends and Mitigation Strategies," in 2020 International Conference on Omni-layer Intelligent Systems (COINS), pp. 1–7, 2020, https://doi.org/10.1109/COINS49042.2020.9191377
[30]	Y. Wang et al., "Tabi: An Efficient Multi-Level Inference System for Large Language Models," in Proceedings of the 18th European Conference on Computer Systems (EuroSys '23), pp. 233–248, 2023, https://doi.org/10.1145/3552326.3587438
[31]	J. Wen, Z. Chen, X. Jin, and X. Liu, "Rise of the Planet of Serverless Computing: A Systematic Review," ACM Transactions on Software Engineering and Methodology, vol. 32, no. 5, pp. 1–61, 2023, https://doi.org/10.1145/3579643
[32]	Wikipedia, "Serverless Computing," 2025. Available: https://en.wikipedia.org/wiki/Serverless\_computing Accessed: Jan. 1, 2025.
[33]	B. Wu et al., "Fast Distributed Inference Serving for Large Language Models," arXiv:2305.05920, 2023. Available: https://arxiv.org/abs/2305.05920 Accessed: Jan. 1, 2025.
[34]	H. Yu, R. Basu Roy, et al., "RainbowCake: Mitigating Cold-starts in Serverless with Layer-wise Container Caching and Sharing," in ASPLOS '24, 2024, https://doi.org/10.1145/3617232.3624871
[35]	K. Zhao, Z. Zhou, L. Jiao, S. Cai, F. Xu, and X. Chen, "Taming Serverless Cold Start of Cloud Model Inference With Edge Computing," IEEE Transactions on Mobile Computing, vol. 23, no. 8, pp. 8111–8128, 2024, https://doi.org/10.1109/TMC.2023.3348165

Cite This Article

Plain Text BibTeX RIS

APA Style

Ghosh, H. (2026). Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud. Internet of Things and Cloud Computing, 14(3), 56-67. https://doi.org/10.11648/j.iotcc.20261403.11

Copy | Download

ACS Style

Ghosh, H. Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud. Internet Things Cloud Comput. 2026, 14(3), 56-67. doi: 10.11648/j.iotcc.20261403.11

Copy | Download

AMA Style

Ghosh H. Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud. Internet Things Cloud Comput. 2026;14(3):56-67. doi: 10.11648/j.iotcc.20261403.11

Copy | Download

@article{10.11648/j.iotcc.20261403.11,
  author = {Himel Ghosh},
  title = {Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud},
  journal = {Internet of Things and Cloud Computing},
  volume = {14},
  number = {3},
  pages = {56-67},
  doi = {10.11648/j.iotcc.20261403.11},
  url = {https://doi.org/10.11648/j.iotcc.20261403.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.iotcc.20261403.11},
  abstract = {This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making real-time applications such as conversational agents and translation services difficult to deploy cost-efficiently. The report first summarizes the cold-start problem and common mitigation strategies in today's Function-as-a-Service (FaaS) platforms, including container warming, host-side checkpoint caching, and cluster-level storage servers. It then reviews the design of ServerlessLLM, which exploits underutilized multi-tier storage in GPU servers, a loading-optimized checkpoint format, live inference migration based on token-level state transfer, and a startup-time-aware model scheduler. Additional methods from recent literature, such as RainbowCake's layered container caching, and the cold-start handling mechanisms of major cloud providers are also discussed. Drawing on the evaluation presented in the original work, the report highlights that multi-tier checkpoint loading and live migration reduce model startup time by up to 6-8× compared to conventional formats, and deliver up to 28× lower startup latency than Ray Serve for large models in realistic traces. The review concludes by outlining open challenges and future directions, including integrating model compression, multi-level inference pipelines, and advanced memory-offloading techniques to further improve the scalability and efficiency of serverless LLM inference.},
 year = {2026}
}

Copy | Download

TY  - JOUR
T1  - Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud
AU  - Himel Ghosh
Y1  - 2026/10/12
PY  - 2026
N1  - https://doi.org/10.11648/j.iotcc.20261403.11
DO  - 10.11648/j.iotcc.20261403.11
T2  - Internet of Things and Cloud Computing
JF  - Internet of Things and Cloud Computing
JO  - Internet of Things and Cloud Computing
SP  - 56
EP  - 67
PB  - Science Publishing Group
SN  - 2376-7731
UR  - https://doi.org/10.11648/j.iotcc.20261403.11
AB  - This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making real-time applications such as conversational agents and translation services difficult to deploy cost-efficiently. The report first summarizes the cold-start problem and common mitigation strategies in today's Function-as-a-Service (FaaS) platforms, including container warming, host-side checkpoint caching, and cluster-level storage servers. It then reviews the design of ServerlessLLM, which exploits underutilized multi-tier storage in GPU servers, a loading-optimized checkpoint format, live inference migration based on token-level state transfer, and a startup-time-aware model scheduler. Additional methods from recent literature, such as RainbowCake's layered container caching, and the cold-start handling mechanisms of major cloud providers are also discussed. Drawing on the evaluation presented in the original work, the report highlights that multi-tier checkpoint loading and live migration reduce model startup time by up to 6-8× compared to conventional formats, and deliver up to 28× lower startup latency than Ray Serve for large models in realistic traces. The review concludes by outlining open challenges and future directions, including integrating model compression, multi-level inference pipelines, and advanced memory-offloading techniques to further improve the scalability and efficiency of serverless LLM inference.
VL  - 14
IS  - 3
ER  -

Copy | Download

Author Information

Himel Ghosh

School of Computation, Information and Technology, Technical University of Munich, Munich, Germany

Contact Email

http://orcid.org/0009-0008-0492-6106

Download PDF

Submit an Article

Sections

Plain Text BibTeX RIS

APA Style

Ghosh, H. (2026). Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud. Internet of Things and Cloud Computing, 14(3), 56-67. https://doi.org/10.11648/j.iotcc.20261403.11

Copy | Download

ACS Style

Ghosh, H. Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud. Internet Things Cloud Comput. 2026, 14(3), 56-67. doi: 10.11648/j.iotcc.20261403.11

Copy | Download

AMA Style

Ghosh H. Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud. Internet Things Cloud Comput. 2026;14(3):56-67. doi: 10.11648/j.iotcc.20261403.11

Copy | Download

@article{10.11648/j.iotcc.20261403.11,
  author = {Himel Ghosh},
  title = {Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud},
  journal = {Internet of Things and Cloud Computing},
  volume = {14},
  number = {3},
  pages = {56-67},
  doi = {10.11648/j.iotcc.20261403.11},
  url = {https://doi.org/10.11648/j.iotcc.20261403.11},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.iotcc.20261403.11},
  abstract = {This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making real-time applications such as conversational agents and translation services difficult to deploy cost-efficiently. The report first summarizes the cold-start problem and common mitigation strategies in today's Function-as-a-Service (FaaS) platforms, including container warming, host-side checkpoint caching, and cluster-level storage servers. It then reviews the design of ServerlessLLM, which exploits underutilized multi-tier storage in GPU servers, a loading-optimized checkpoint format, live inference migration based on token-level state transfer, and a startup-time-aware model scheduler. Additional methods from recent literature, such as RainbowCake's layered container caching, and the cold-start handling mechanisms of major cloud providers are also discussed. Drawing on the evaluation presented in the original work, the report highlights that multi-tier checkpoint loading and live migration reduce model startup time by up to 6-8× compared to conventional formats, and deliver up to 28× lower startup latency than Ray Serve for large models in realistic traces. The review concludes by outlining open challenges and future directions, including integrating model compression, multi-level inference pipelines, and advanced memory-offloading techniques to further improve the scalability and efficiency of serverless LLM inference.},
 year = {2026}
}

Copy | Download

TY  - JOUR
T1  - Enabling Efficient Serverless Inference Serving for LLM (Large Language Model) in the Cloud
AU  - Himel Ghosh
Y1  - 2026/10/12
PY  - 2026
N1  - https://doi.org/10.11648/j.iotcc.20261403.11
DO  - 10.11648/j.iotcc.20261403.11
T2  - Internet of Things and Cloud Computing
JF  - Internet of Things and Cloud Computing
JO  - Internet of Things and Cloud Computing
SP  - 56
EP  - 67
PB  - Science Publishing Group
SN  - 2376-7731
UR  - https://doi.org/10.11648/j.iotcc.20261403.11
AB  - This review report examines the problem of cold-start latency in serverless inference for large language models (LLMs) and surveys recent systems that address it, with a particular focus on ServerlessLLM. In serverless environments, loading multi-hundred-gigabyte LLM checkpoints and initializing GPU resources can dominate end-to-end latency, making real-time applications such as conversational agents and translation services difficult to deploy cost-efficiently. The report first summarizes the cold-start problem and common mitigation strategies in today's Function-as-a-Service (FaaS) platforms, including container warming, host-side checkpoint caching, and cluster-level storage servers. It then reviews the design of ServerlessLLM, which exploits underutilized multi-tier storage in GPU servers, a loading-optimized checkpoint format, live inference migration based on token-level state transfer, and a startup-time-aware model scheduler. Additional methods from recent literature, such as RainbowCake's layered container caching, and the cold-start handling mechanisms of major cloud providers are also discussed. Drawing on the evaluation presented in the original work, the report highlights that multi-tier checkpoint loading and live migration reduce model startup time by up to 6-8× compared to conventional formats, and deliver up to 28× lower startup latency than Ray Serve for large models in realistic traces. The review concludes by outlining open challenges and future directions, including integrating model compression, multi-level inference pipelines, and advanced memory-offloading techniques to further improve the scalability and efficiency of serverless LLM inference.
VL  - 14
IS  - 3
ER  -

Copy | Download