2026
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
Zili Zhang, Chengxu Yang, Shenglong Zhang, Chenyu Wang, Yufan Zhang, Tuo Dai, Zhouyang Li, Yuhong Ge, Chao Jin, Xin Jin, Yuliang Liu
In preprint 2026
This work presents BigMac, a new training pipeline for multimodal LLMs that breaks the Pareto frontier between compute and memory efficiency. The core idea of BigMac is to elegantly nest the encoder and generator computation into the original LLM pipeline, forming a dependency-safe nested pipeline structure that reduces activation memory complexity while preserving compute efficiency.
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
Zili Zhang, Chengxu Yang, Shenglong Zhang, Chenyu Wang, Yufan Zhang, Tuo Dai, Zhouyang Li, Yuhong Ge, Chao Jin, Xin Jin, Yuliang Liu
In preprint 2026
This work presents BigMac, a new training pipeline for multimodal LLMs that breaks the Pareto frontier between compute and memory efficiency. The core idea of BigMac is to elegantly nest the encoder and generator computation into the original LLM pipeline, forming a dependency-safe nested pipeline structure that reduces activation memory complexity while preserving compute efficiency.
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong, Shan Yu, Chengxu Yang, Bingyang Wu, Zili Zhang, Jing Mai, Qianchao Zhu, Zhouyang Li, Yuliang Liu, Guojie Luo
In preprint 2026
This work presents UltraEP, the first exact-load, real-time balancer for large-EP MoE training and serving prefill on rack-scale nodes (RSNs). Leveraging the extended scale-up connectivity among dozens of GPUs within RSNs, UltraEP rebalances every microbatch and layer on critical paths, which requires nontrivial co-design of plan solving and expert replication communication to minimize exposed overhead. To this end, UltraEP eagerly reacts to post-gating load with an efficient quota-driven planner, and executes the resulting irregular expert-state transfers with RSN-native persistent tile streaming and relay-based fan-out mitigation. We evaluate UltraEP in a multi-RSN deployment of up to 256 GPUs, using cutting-edge MoE models from 106B to 671B parameters.
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong, Shan Yu, Chengxu Yang, Bingyang Wu, Zili Zhang, Jing Mai, Qianchao Zhu, Zhouyang Li, Yuliang Liu, Guojie Luo
In preprint 2026
This work presents UltraEP, the first exact-load, real-time balancer for large-EP MoE training and serving prefill on rack-scale nodes (RSNs). Leveraging the extended scale-up connectivity among dozens of GPUs within RSNs, UltraEP rebalances every microbatch and layer on critical paths, which requires nontrivial co-design of plan solving and expert replication communication to minimize exposed overhead. To this end, UltraEP eagerly reacts to post-gating load with an efficient quota-driven planner, and executes the resulting irregular expert-state transfers with RSN-native persistent tile streaming and relay-based fan-out mitigation. We evaluate UltraEP in a multi-RSN deployment of up to 256 GPUs, using cutting-edge MoE models from 106B to 671B parameters.
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
Chao Jin, Xinming Wei, Yinmin Zhong, Chengxu Yang, Bingyang Wu, Ruidong Zhu, Zili Zhang, Yuliang Liu, Xin Jin
In preprint 2026
This work proposes ReLibra, an MoE RL training system that exploits a unique opportunity in RL's rollout-training workflow, routing replay, to enable fine-grained load balancing at micro-batch granularity. Because rollout and training process the same tokens with the same MoE parameters, the token-to-expert routing decisions are known before training starts. Leveraging this information, ReLibra places two MoE load-balancing mechanisms at inter- and intra-batch timescales, matching their communication patterns to hierarchical network bandwidths. At the inter-batch timescale, ReLibra performs expert reordering to redistribute experts for batch-level cross-node balancing; at the intra-batch timescale, it dynamically performs expert replication within a node to absorb micro-batch-level load fluctuations.
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
Chao Jin, Xinming Wei, Yinmin Zhong, Chengxu Yang, Bingyang Wu, Ruidong Zhu, Zili Zhang, Yuliang Liu, Xin Jin
In preprint 2026
This work proposes ReLibra, an MoE RL training system that exploits a unique opportunity in RL's rollout-training workflow, routing replay, to enable fine-grained load balancing at micro-batch granularity. Because rollout and training process the same tokens with the same MoE parameters, the token-to-expert routing decisions are known before training starts. Leveraging this information, ReLibra places two MoE load-balancing mechanisms at inter- and intra-batch timescales, matching their communication patterns to hierarchical network bandwidths. At the inter-batch timescale, ReLibra performs expert reordering to redistribute experts for batch-level cross-node balancing; at the intra-batch timescale, it dynamically performs expert replication within a node to absorb micro-batch-level load fluctuations.
FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
Bingyang Wu*, Yinmin Zhong*, Zili Zhang*, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin (* equal contribution)
Networking Systems Design and Implementation (NSDI) 2026
This paper presents FastServe, a distributed inference serving system for LLMs. FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. FastServe uses preemptive scheduling to minimize JCT with a novel skip-join Multi-Level Feedback Queue scheduler.
FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
Bingyang Wu*, Yinmin Zhong*, Zili Zhang*, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin (* equal contribution)
Networking Systems Design and Implementation (NSDI) 2026
This paper presents FastServe, a distributed inference serving system for LLMs. FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. FastServe uses preemptive scheduling to minimize JCT with a novel skip-join Multi-Level Feedback Queue scheduler.
Heddle: A Distributed Orchestration System for Agentic RL Rollout
Zili Zhang
In preprint 2026
Heddle is a distributed orchestration system for agentic RL rollout. It coordinates the multi-step, tool-augmented generation of agentic workflows across distributed workers, addressing the scheduling and efficiency challenges of RL rollout with long, branching, and heterogeneous execution paths.
Heddle: A Distributed Orchestration System for Agentic RL Rollout
Zili Zhang
In preprint 2026
Heddle is a distributed orchestration system for agentic RL rollout. It coordinates the multi-step, tool-augmented generation of agentic workflows across distributed workers, addressing the scheduling and efficiency challenges of RL rollout with long, branching, and heterogeneous execution paths.
2025
DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, Xin Jin
ACM Special Interest Group on Data Communication (SIGCOMM) 2025
This work presents DistTrain, an efficient and adaptive framework to reform the training of multimodal large language models on large-scale clusters. The core of DistTrain is the disaggregated training technique that exploits the characteristics of multimodal LLM training to achieve high efficiency and scalability. Specifically, it leverages disaggregated model orchestration and disaggregated data reordering to address model and data heterogeneity respectively.
DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, Xin Jin
ACM Special Interest Group on Data Communication (SIGCOMM) 2025
This work presents DistTrain, an efficient and adaptive framework to reform the training of multimodal large language models on large-scale clusters. The core of DistTrain is the disaggregated training technique that exploits the characteristics of multimodal LLM training to achieve high efficiency and scalability. Specifically, it leverages disaggregated model orchestration and disaggregated data reordering to address model and data heterogeneity respectively.
TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
Bingyang Wu, Zili Zhang, Yinmin Zhong, Guanzhe Huang, Yibo Zhu, Xuanzhe Liu, Xin Jin
In preprint 2025
This work presents TokenLake, which uses a declarative cache interface to expose requests' query tensors, prefix caches, and cache-aware operations for efficient pooling. Powered by this abstraction, TokenLake can manage prefix cache at the segment level with a heavy-hitter-aware load balancing algorithm to achieve better cache load balance, deduplication, and defragmentation. TokenLake also transparently minimizes the communication volume of query tensors and new caches. Based on TokenLake, the scheduler can schedule requests elastically by using existing techniques without considering prefix cache management.
TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
Bingyang Wu, Zili Zhang, Yinmin Zhong, Guanzhe Huang, Yibo Zhu, Xuanzhe Liu, Xin Jin
In preprint 2025
This work presents TokenLake, which uses a declarative cache interface to expose requests' query tensors, prefix caches, and cache-aware operations for efficient pooling. Powered by this abstraction, TokenLake can manage prefix cache at the segment level with a heavy-hitter-aware load balancing algorithm to achieve better cache load balance, deduplication, and defragmentation. TokenLake also transparently minimizes the communication volume of query tensors and new caches. Based on TokenLake, the scheduler can schedule requests elastically by using existing techniques without considering prefix cache management.
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, Xin Jin
ACM Transactions on Computer Systems (TOCS) 2025
This work proposes RAGCache, a novel multilevel dynamic caching system tailored for Retrieval-Augmented Generation (RAG). RAGCache caches the intermediate states of injected external knowledge and shares them across multiple queries, mitigating the high computation and memory costs caused by long-sequence generation in RAG.
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, Xin Jin
ACM Transactions on Computer Systems (TOCS) 2025
This work proposes RAGCache, a novel multilevel dynamic caching system tailored for Retrieval-Augmented Generation (RAG). RAGCache caches the intermediate states of injected external knowledge and shares them across multiple queries, mitigating the high computation and memory costs caused by long-sequence generation in RAG.
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang
In preprint 2025
StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching.
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang
In preprint 2025
StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching.
Optimizing RLHF Training for Large Language Models with Stage Fusion
Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2025
This work presents RLHFuse, an efficient RLHF training system which views the RLHF workflow from a finer-grained subtask-level perspective and opens up opportunities for efficient inter- and intra-stage fused execution, mitigating data skewness and pipeline bubbles in existing systems.
Optimizing RLHF Training for Large Language Models with Stage Fusion
Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2025
This work presents RLHFuse, an efficient RLHF training system which views the RLHF workflow from a finer-grained subtask-level perspective and opens up opportunities for efficient inter- and intra-stage fused execution, mitigating data skewness and pipeline bubbles in existing systems.
2024
dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin
Operating Systems Design and Implementation (OSDI) 2024
This paper introduces dLoRA, an inference serving system for LoRA models. dLoRA achieves high serving efficiency by dynamically orchestrating requests and LoRA adapters in two aspects: dynamically merging and unmerging adapters with the base model, and dynamically migrating requests and adapters between different worker replicas.
dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin
Operating Systems Design and Implementation (OSDI) 2024
This paper introduces dLoRA, an inference serving system for LoRA models. dLoRA achieves high serving efficiency by dynamically orchestrating requests and LoRA adapters in two aspects: dynamically merging and unmerging adapters with the base model, and dynamically migrating requests and adapters between different worker replicas.
Jolteon: Unleashing the Promise of Serverless for Serverless Workflows
Zili Zhang, Chao Jin, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This paper proposes Jolteon, an orchestrator to unleash the promise of automatic resource provisioning for serverless workflows. At the core of Jolteon is a stochastic performance model that combines the benefits of whitebox modeling to capture the execution characteristics of serverless computing and blackbox modeling to accommodate the inherent performance variability.
Jolteon: Unleashing the Promise of Serverless for Serverless Workflows
Zili Zhang, Chao Jin, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This paper proposes Jolteon, an orchestrator to unleash the promise of automatic resource provisioning for serverless workflows. At the core of Jolteon is a stochastic performance model that combines the benefits of whitebox modeling to capture the execution characteristics of serverless computing and blackbox modeling to accommodate the inherent performance variability.
Fast Vector Query Processing for Large Datasets Beyond GPU Memory with Reordered Pipelining
Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This work presents RUMMY, the first GPU-accelerated vector query processing system that achieves high performance and supports large vector datasets beyond GPU memory. The core of RUMMY is a novel reordered pipelining technique that exploits the characteristics of vector query processing to efficiently pipeline data transmission from host memory to GPU memory with query processing in GPU.
Fast Vector Query Processing for Large Datasets Beyond GPU Memory with Reordered Pipelining
Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This work presents RUMMY, the first GPU-accelerated vector query processing system that achieves high performance and supports large vector datasets beyond GPU memory. The core of RUMMY is a novel reordered pipelining technique that exploits the characteristics of vector query processing to efficiently pipeline data transmission from host memory to GPU memory with query processing in GPU.
2023
Ditto: Efficient Serverless Analytics with Elastic Parallelism
Chao Jin, Zili Zhang, Xingyu Xiang, Songyun Zou, Gang Huang, Xuanzhe Liu, Xin Jin
ACM SIGCOMM 2023
Ditto is a serverless analytics system that leverages elastic parallelism to optimize job completion time and cost. Ditto uses a fine-grained resource abstraction and a scheduling mechanism that adaptively adjusts the parallelism of each stage to fully exploit the elasticity of serverless computing.
Ditto: Efficient Serverless Analytics with Elastic Parallelism
Chao Jin, Zili Zhang, Xingyu Xiang, Songyun Zou, Gang Huang, Xuanzhe Liu, Xin Jin
ACM SIGCOMM 2023
Ditto is a serverless analytics system that leverages elastic parallelism to optimize job completion time and cost. Ditto uses a fine-grained resource abstraction and a scheduling mechanism that adaptively adjusts the parallelism of each stage to fully exploit the elasticity of serverless computing.
Fast, Approximate Vector Queries on Very Large Unstructured Datasets
Zili Zhang, Chao Jin, Linpeng Tang, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2023
This paper presents Auncel, a vector query engine for very large unstructured datasets that provides bounded, controllable accuracy while accelerating approximate vector search. Auncel exploits the geometric properties of embedding spaces to prune unnecessary computation and deliver fast, error-bounded query results.
Fast, Approximate Vector Queries on Very Large Unstructured Datasets
Zili Zhang, Chao Jin, Linpeng Tang, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2023
This paper presents Auncel, a vector query engine for very large unstructured datasets that provides bounded, controllable accuracy while accelerating approximate vector search. Auncel exploits the geometric properties of embedding spaces to prune unnecessary computation and deliver fast, error-bounded query results.
Transparent GPU Sharing in Container Clouds for Deep Learning Training
Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2023
This paper presents TGS (Transparent GPU Sharing), a system that provides transparent GPU sharing to deep learning training jobs in container clouds. TGS operates at the OS layer below containers, requiring no modification to user code, and uses adaptive rate control and transparent unified memory to achieve high utilization while avoiding interference.
Transparent GPU Sharing in Container Clouds for Deep Learning Training
Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2023
This paper presents TGS (Transparent GPU Sharing), a system that provides transparent GPU sharing to deep learning training jobs in container clouds. TGS operates at the OS layer below containers, requiring no modification to user code, and uses adaptive rate control and transparent unified memory to achieve high utilization while avoiding interference.
Rise of Distributed Deep Learning Training in the Big Model Era: From A Software Engineering Perspective
Xuanzhe Liu, Diandian Gu, Zhenpeng Chen, Jinfeng Wen, Zili Zhang, Yun Ma, Haoyu Wang, Xin Jin
ACM Transactions on Software Engineering and Methodology (TOSEM) 2023
This survey reviews distributed deep learning training in the big model era from a software engineering perspective, systematically summarizing the techniques, systems, and challenges of training large-scale deep learning models across distributed infrastructure.
Rise of Distributed Deep Learning Training in the Big Model Era: From A Software Engineering Perspective
Xuanzhe Liu, Diandian Gu, Zhenpeng Chen, Jinfeng Wen, Zili Zhang, Yun Ma, Haoyu Wang, Xin Jin
ACM Transactions on Software Engineering and Methodology (TOSEM) 2023
This survey reviews distributed deep learning training in the big model era from a software engineering perspective, systematically summarizing the techniques, systems, and challenges of training large-scale deep learning models across distributed infrastructure.
2022
Optimizing Half Precision Winograd Convolution on ARM Many-Core Processors
Dedong Xie, Zili Zhang, Yun Ma, Xuanzhe Liu
ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) 2022
This work optimizes half-precision Winograd convolution on ARM many-core processors, exploiting the architectural characteristics of ARM CPUs to accelerate convolution while preserving numerical accuracy.
Optimizing Half Precision Winograd Convolution on ARM Many-Core Processors
Dedong Xie, Zili Zhang, Yun Ma, Xuanzhe Liu
ACM SIGOPS Asia-Pacific Workshop on Systems (APSys) 2022
This work optimizes half-precision Winograd convolution on ARM many-core processors, exploiting the architectural characteristics of ARM CPUs to accelerate convolution while preserving numerical accuracy.