PhD student @ Peking UniversityI am a fourth-year Ph.D. candidate majoring in Computer Science at the Computer Systems Research Group, Peking University (2023 - present), advised by Prof. Xin Jin. My research and engineering work focus on systems for Multimodal LLM (MLLM) pretraining and reinforcement learning.
I received my B.E. from the School of Electronics Engineering and Computer Science (EECS), Peking University (2019 - 2023). I used to be a research assistant at the Software Engineering Institute advised by Prof. Xin Jin (2020 - 2023).
Outside of research, I am an amateur runner — here are my ITRA profile and Running Page. I love outdoor traveling, hiking, skiing, and camping — I believe that only by immersing myself in nature can I find inner peace. I also enjoy video games for their immersive storytelling.
Warning
Problem: The current name of your GitHub Pages repository ("Solution: Please consider renaming the repository to "
http://".
However, if the current repository name is intended, you can ignore this message by removing "{% include widgets/debug_repo_name.html %}" in index.html.
Action required
Problem: The current root path of this site is "baseurl ("_config.yml.
Solution: Please set the
baseurl in _config.yml to "Education
-
Peking UniversitySchool of Computer Science
Ph.D. StudentSep. 2023 - present -
Peking UniversityB.E. in Computer Science (School of EECS)Sep. 2019 - Jul. 2023
Experience
-
RedNote Infra TeamRedStar Intern, Infra for Multimodal LLM + RLSep. 2025 - Jun. 2026 -
ByteDance Seed Multimodal TreamTopSeed Intern, Data and Algorithm for MLLM + RLApr. 2025 - Sep. 2025 -
StepFun Infra TreamResearch Intern, Infra for LLM Training + RLHFApr. 2024 - Apr. 2025
Honors & Awards
-
NSFC Basic Research Program for Young Students (Principal Investigator)2025
-
Top 10 Academic Stars, School of Computer Science, PKU (10/800)2025
-
Merit Student (1/56)2024
-
National Scholarship (Top 0.4% nationally)2024
-
Presidential Scholarship of Peking University2024
-
Top 10 Bachelor Thesis, School of EECS, PKU (10/408)2023
-
Outstanding Graduation Thesis, Peking University (33/4239)2023
-
Outstanding Graduation Thesis, Beijing City2023
-
Excellent Graduate, Peking University (613/4239)2023
-
Representor of Excellent Graduates, Peking University (14/4239)2023
-
Exceptional Award for Academic Innovation, PKU (5/408)2022
-
Award for Scientific Research, PKU2022
-
Lee Wai Wing Scholarship, PKU2022
-
Award for Scientific Research, PKU2021
-
Award for Academic Excellents, PKU2020
-
The Third Prize of Peking University Scholarship2020
Teaching
-
Teaching Assistant, Operating System (Honor Track), PKU2024 Spring
-
Teaching Assistant, Introduction to Computer System (Honor Track), PKU2022 Fall
-
Teaching Assistant, Introduction to Computer System, PKU2021 Fall
Academic Service
-
IEEE TMC, Invited Reviewer2026
-
EuroSys, Shadow PC2025
Selected Publications (view all )
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
Zili Zhang, Chengxu Yang, Shenglong Zhang, Chenyu Wang, Yufan Zhang, Tuo Dai, Zhouyang Li, Yuhong Ge, Chao Jin, Xin Jin, Yuliang Liu
In preprint 2026
This work presents BigMac, a new training pipeline for multimodal LLMs that breaks the Pareto frontier between compute and memory efficiency. The core idea of BigMac is to elegantly nest the encoder and generator computation into the original LLM pipeline, forming a dependency-safe nested pipeline structure that reduces activation memory complexity while preserving compute efficiency.
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
Zili Zhang, Chengxu Yang, Shenglong Zhang, Chenyu Wang, Yufan Zhang, Tuo Dai, Zhouyang Li, Yuhong Ge, Chao Jin, Xin Jin, Yuliang Liu
In preprint 2026
This work presents BigMac, a new training pipeline for multimodal LLMs that breaks the Pareto frontier between compute and memory efficiency. The core idea of BigMac is to elegantly nest the encoder and generator computation into the original LLM pipeline, forming a dependency-safe nested pipeline structure that reduces activation memory complexity while preserving compute efficiency.
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong, Shan Yu, Chengxu Yang, Bingyang Wu, Zili Zhang, Jing Mai, Qianchao Zhu, Zhouyang Li, Yuliang Liu, Guojie Luo
In preprint 2026
This work presents UltraEP, the first exact-load, real-time balancer for large-EP MoE training and serving prefill on rack-scale nodes (RSNs). Leveraging the extended scale-up connectivity among dozens of GPUs within RSNs, UltraEP rebalances every microbatch and layer on critical paths, which requires nontrivial co-design of plan solving and expert replication communication to minimize exposed overhead. To this end, UltraEP eagerly reacts to post-gating load with an efficient quota-driven planner, and executes the resulting irregular expert-state transfers with RSN-native persistent tile streaming and relay-based fan-out mitigation. We evaluate UltraEP in a multi-RSN deployment of up to 256 GPUs, using cutting-edge MoE models from 106B to 671B parameters.
UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing
Xinming Wei, Chao Jin, Tuo Dai, Yinmin Zhong, Shan Yu, Chengxu Yang, Bingyang Wu, Zili Zhang, Jing Mai, Qianchao Zhu, Zhouyang Li, Yuliang Liu, Guojie Luo
In preprint 2026
This work presents UltraEP, the first exact-load, real-time balancer for large-EP MoE training and serving prefill on rack-scale nodes (RSNs). Leveraging the extended scale-up connectivity among dozens of GPUs within RSNs, UltraEP rebalances every microbatch and layer on critical paths, which requires nontrivial co-design of plan solving and expert replication communication to minimize exposed overhead. To this end, UltraEP eagerly reacts to post-gating load with an efficient quota-driven planner, and executes the resulting irregular expert-state transfers with RSN-native persistent tile streaming and relay-based fan-out mitigation. We evaluate UltraEP in a multi-RSN deployment of up to 256 GPUs, using cutting-edge MoE models from 106B to 671B parameters.
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
Chao Jin, Xinming Wei, Yinmin Zhong, Chengxu Yang, Bingyang Wu, Ruidong Zhu, Zili Zhang, Yuliang Liu, Xin Jin
In preprint 2026
This work proposes ReLibra, an MoE RL training system that exploits a unique opportunity in RL's rollout-training workflow, routing replay, to enable fine-grained load balancing at micro-batch granularity. Because rollout and training process the same tokens with the same MoE parameters, the token-to-expert routing decisions are known before training starts. Leveraging this information, ReLibra places two MoE load-balancing mechanisms at inter- and intra-batch timescales, matching their communication patterns to hierarchical network bandwidths. At the inter-batch timescale, ReLibra performs expert reordering to redistribute experts for batch-level cross-node balancing; at the intra-batch timescale, it dynamically performs expert replication within a node to absorb micro-batch-level load fluctuations.
ReLibra: Routing-Replay-Guided Load Balancing for MoE Training in Reinforcement Learning
Chao Jin, Xinming Wei, Yinmin Zhong, Chengxu Yang, Bingyang Wu, Ruidong Zhu, Zili Zhang, Yuliang Liu, Xin Jin
In preprint 2026
This work proposes ReLibra, an MoE RL training system that exploits a unique opportunity in RL's rollout-training workflow, routing replay, to enable fine-grained load balancing at micro-batch granularity. Because rollout and training process the same tokens with the same MoE parameters, the token-to-expert routing decisions are known before training starts. Leveraging this information, ReLibra places two MoE load-balancing mechanisms at inter- and intra-batch timescales, matching their communication patterns to hierarchical network bandwidths. At the inter-batch timescale, ReLibra performs expert reordering to redistribute experts for batch-level cross-node balancing; at the intra-batch timescale, it dynamically performs expert replication within a node to absorb micro-batch-level load fluctuations.
FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
Bingyang Wu*, Yinmin Zhong*, Zili Zhang*, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin (* equal contribution)
Networking Systems Design and Implementation (NSDI) 2026
This paper presents FastServe, a distributed inference serving system for LLMs. FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. FastServe uses preemptive scheduling to minimize JCT with a novel skip-join Multi-Level Feedback Queue scheduler.
FastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
Bingyang Wu*, Yinmin Zhong*, Zili Zhang*, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin (* equal contribution)
Networking Systems Design and Implementation (NSDI) 2026
This paper presents FastServe, a distributed inference serving system for LLMs. FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. FastServe uses preemptive scheduling to minimize JCT with a novel skip-join Multi-Level Feedback Queue scheduler.
DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, Xin Jin
ACM Special Interest Group on Data Communication (SIGCOMM) 2025
This work presents DistTrain, an efficient and adaptive framework to reform the training of multimodal large language models on large-scale clusters. The core of DistTrain is the disaggregated training technique that exploits the characteristics of multimodal LLM training to achieve high efficiency and scalability. Specifically, it leverages disaggregated model orchestration and disaggregated data reordering to address model and data heterogeneity respectively.
DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, Xin Jin
ACM Special Interest Group on Data Communication (SIGCOMM) 2025
This work presents DistTrain, an efficient and adaptive framework to reform the training of multimodal large language models on large-scale clusters. The core of DistTrain is the disaggregated training technique that exploits the characteristics of multimodal LLM training to achieve high efficiency and scalability. Specifically, it leverages disaggregated model orchestration and disaggregated data reordering to address model and data heterogeneity respectively.
TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
Bingyang Wu, Zili Zhang, Yinmin Zhong, Guanzhe Huang, Yibo Zhu, Xuanzhe Liu, Xin Jin
In preprint 2025
This work presents TokenLake, which uses a declarative cache interface to expose requests' query tensors, prefix caches, and cache-aware operations for efficient pooling. Powered by this abstraction, TokenLake can manage prefix cache at the segment level with a heavy-hitter-aware load balancing algorithm to achieve better cache load balance, deduplication, and defragmentation. TokenLake also transparently minimizes the communication volume of query tensors and new caches. Based on TokenLake, the scheduler can schedule requests elastically by using existing techniques without considering prefix cache management.
TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
Bingyang Wu, Zili Zhang, Yinmin Zhong, Guanzhe Huang, Yibo Zhu, Xuanzhe Liu, Xin Jin
In preprint 2025
This work presents TokenLake, which uses a declarative cache interface to expose requests' query tensors, prefix caches, and cache-aware operations for efficient pooling. Powered by this abstraction, TokenLake can manage prefix cache at the segment level with a heavy-hitter-aware load balancing algorithm to achieve better cache load balance, deduplication, and defragmentation. TokenLake also transparently minimizes the communication volume of query tensors and new caches. Based on TokenLake, the scheduler can schedule requests elastically by using existing techniques without considering prefix cache management.
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, Xin Jin
ACM Transactions on Computer Systems (TOCS) 2025
This work proposes RAGCache, a novel multilevel dynamic caching system tailored for Retrieval-Augmented Generation (RAG). RAGCache caches the intermediate states of injected external knowledge and shares them across multiple queries, mitigating the high computation and memory costs caused by long-sequence generation in RAG.
RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, Xin Jin
ACM Transactions on Computer Systems (TOCS) 2025
This work proposes RAGCache, a novel multilevel dynamic caching system tailored for Retrieval-Augmented Generation (RAG). RAGCache caches the intermediate states of injected external knowledge and shares them across multiple queries, mitigating the high computation and memory costs caused by long-sequence generation in RAG.
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang
In preprint 2025
StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching.
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang
In preprint 2025
StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching.
Optimizing RLHF Training for Large Language Models with Stage Fusion
Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2025
This work presents RLHFuse, an efficient RLHF training system which views the RLHF workflow from a finer-grained subtask-level perspective and opens up opportunities for efficient inter- and intra-stage fused execution, mitigating data skewness and pipeline bubbles in existing systems.
Optimizing RLHF Training for Large Language Models with Stage Fusion
Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2025
This work presents RLHFuse, an efficient RLHF training system which views the RLHF workflow from a finer-grained subtask-level perspective and opens up opportunities for efficient inter- and intra-stage fused execution, mitigating data skewness and pipeline bubbles in existing systems.
dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin
Operating Systems Design and Implementation (OSDI) 2024
This paper introduces dLoRA, an inference serving system for LoRA models. dLoRA achieves high serving efficiency by dynamically orchestrating requests and LoRA adapters in two aspects: dynamically merging and unmerging adapters with the base model, and dynamically migrating requests and adapters between different worker replicas.
dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin
Operating Systems Design and Implementation (OSDI) 2024
This paper introduces dLoRA, an inference serving system for LoRA models. dLoRA achieves high serving efficiency by dynamically orchestrating requests and LoRA adapters in two aspects: dynamically merging and unmerging adapters with the base model, and dynamically migrating requests and adapters between different worker replicas.
Jolteon: Unleashing the Promise of Serverless for Serverless Workflows
Zili Zhang, Chao Jin, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This paper proposes Jolteon, an orchestrator to unleash the promise of automatic resource provisioning for serverless workflows. At the core of Jolteon is a stochastic performance model that combines the benefits of whitebox modeling to capture the execution characteristics of serverless computing and blackbox modeling to accommodate the inherent performance variability.
Jolteon: Unleashing the Promise of Serverless for Serverless Workflows
Zili Zhang, Chao Jin, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This paper proposes Jolteon, an orchestrator to unleash the promise of automatic resource provisioning for serverless workflows. At the core of Jolteon is a stochastic performance model that combines the benefits of whitebox modeling to capture the execution characteristics of serverless computing and blackbox modeling to accommodate the inherent performance variability.
Fast Vector Query Processing for Large Datasets Beyond GPU Memory with Reordered Pipelining
Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This work presents RUMMY, the first GPU-accelerated vector query processing system that achieves high performance and supports large vector datasets beyond GPU memory. The core of RUMMY is a novel reordered pipelining technique that exploits the characteristics of vector query processing to efficiently pipeline data transmission from host memory to GPU memory with query processing in GPU.
Fast Vector Query Processing for Large Datasets Beyond GPU Memory with Reordered Pipelining
Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, Xin Jin
Networking Systems Design and Implementation (NSDI) 2024
This work presents RUMMY, the first GPU-accelerated vector query processing system that achieves high performance and supports large vector datasets beyond GPU memory. The core of RUMMY is a novel reordered pipelining technique that exploits the characteristics of vector query processing to efficiently pipeline data transmission from host memory to GPU memory with query processing in GPU.