직함: [seminar] Best Practices for Generative AI with LLMs on a Supercomputer
Generative AI with LLMs refers to the use of large language models like GPT-3 for generating human-like content, spanning text, images and even code. LLMs are trained on a vast amount of data and code, and usually carefully prompt-engineered or fine-tuned to suit specific downstream tasks such as Chatbots, Translation, Question Answering and Summarization. The contents and python codes of this seminar are mainly originated from the 16-hour “Generative AI with LLMs” course offered by the DeepLearning.AI. This talk will cover the key concepts and practices of a typical LLM-powered Generative AI lifecycle, from data gathering and model selection, to instruction fine-tuning and RLHF-based alignment to human preference, to performance evaluation and deployment. I will show a short demo on how users can create a conda virtual environment on the KISTI Neuron cluster with 260 GPUs, launch a Jupyter server on a compute node and have access it to from his/her own PC or Labtop for Genera tive AI practices on a supercomputer. The demo will illustrate how to conduct LLM practices including prompting and prompt engineering, and instruction fine-tuning and parameter-efficient fine-tuning (PEFT) with LoRA, and evaluation and benchmark on LLMs.
Soonwook Hwang is a Principal Researcher and formal Director General at the National Supercomputing Center of the Korea Institute of Science and Technology Information (KISTI). He is also a Professor at the KISTI school of Data & High Performance Computing Science in the University of Science and Technology (UST). Dr. Hwang serves as a columnist for Joongdo Daily News, where he has been writing science columns primarily focused on Supercomputing and Artificial Intelligence since 2019.
As the Director General of KISTI’s National Supercomputing Center, he was in charge of the development of the KISTI-5 supercomputer, Nurion, in partership with Cray and Intel, playing a pivotal role in Nurion debuting the No. 11 spot on the Top500 list in June, 2018.
Dr. Hwang’s recent research interests center around exploring and developing large-scale distributed deep learning practices including pre-training/fine-tuning LLMs on supercomputers, especially on National Supercomputing Facilities like KISTI in Korea and LBNL/NERSC in the U.S.
He joined KISTI in 2006 as a Chief Architect for Korea’s national e-Science project. He led the development of the AMGA Metadata catalog software which used to be a de facto grid metadata service. AMGA is still being exploited as a metadata service for data analytics in the Belle2 experiment, an international collaboration in High Energy Physics hosted by KEK in Japan. He also led the design and development of the HTCaaS (High-Throughput Computing as a Service), a platform designed to support the execution of High-Throughput Computing (HTC) tasks at scale by efficiently harnessing available computing resources, such as supercomputers, grids, and clouds.
Before his tenure at KISTI, Dr. Hwang worked for the Japanese National Research Grid Initiative (NAREGI) from 2004 to 2006. NAREGI aims to develop grid middleware for a next-generation CyberScience infrastructure in Japan.
Dr. Hwang received his Ph.D. in Computer Science from the University of Southern California (USC) in 2003 under the supervision of Prof. Carl Kesselman, a pioneer in grid computing and the creator of the Globus open-source toolkit. He also earned his B.S. and M.S. degrees in Mathematics and Computer Science from Seoul National University, Korea, in 1990 and 1995, respectively.