From Tokens to Topology: System-Level LLM Optimization for Latency, Throughput, and Cost in Production
As large language models (LLMs) are increasingly deployed in real-world applications, their practical performance depends not only on model capability but also on the efficiency of the supporting systems. This talk presents a system-level view of LLM optimization, focusing on training-free, plug-and-play strategies to address challenges such as long-context processing, KV-cache traffic, redundant computation in multi-agent pipelines, and workflow-level latency. Through examples spanning long-context inference, multi-agent collaboration, and non-autoregressive generation, it will show how coordinating computation, memory, and execution can improve time-to-first-token, latency, throughput, and GPU utilization without sacrificing quality, offering practical insights into scaling LLM systems for production workloads.
Yiran Chen is the John Cocke Distinguished Professor of Electrical and Computer Engineering at Duke University. He serves as Principal Investigator and Director of the NSF AI Institute for Edge Computing and Systems (Athena), Director of the Institute for AI Engineering (IAIE), and Co-Director of the Duke Center for Computational Evolutionary Intelligence (DCEI). His research spans emerging memory and storage technologies, machine learning systems, neuromorphic computing, and edge AI. He has authored more than 800 publications and holds 96 U.S. patents. His research has been recognized with three Test-of-Time Awards and 15 Best Paper and Best Poster Awards. He is a recipient of the IEEE Circuits and Systems Society Charles A. Desoer Technical Achievement Award, the IEEE Computer Society Edward J. McCluskey Technical Achievement Award, and the IEEE Council on Electronic Design Automation A. Richard Newton Technical Impact Award. Chen is the founding Editor-in-Chief of the IEEE Transactions on Circuits and Systems for Artificial Intelligence (TCASAI) and the founding Chair of the IEEE Circuits and Systems Society’s Machine Learning Circuits and Systems (MLCAS) Technical Committee. He is a Fellow of the AAAS, ACM, IEEE, and NAI, and a member of the European Academy of Sciences and Arts.