UROP Project

Large Language Models, Efficiency

Research Mentor: Dr. Shangqian Gao,
Department, College, Affiliation: Computer Science, Arts and Sciences
Contact Email: sg24bi@fsu.edu
Research Assistant Supervisor (if different from mentor):
Research Assistant Supervisor Email:
Faculty Collaborators:
Faculty Collaborators Email:

Looking for Research Assistants: Yes
Number of Research Assistants: 2
Relevant Majors: Computer Science, Electrical Engineering
Project Location: On FSU Main Campus
Research Assistant Transportation Required:
Remote or In-person: Partially Remote
Approximate Weekly Hours: 10,
Roundtable Times and Zoom Link: Not participating in the Roundtable

Project Description

Large Language Models (LLMs) have gained significant popularity recently. However, their model size is often too large to be deployed on commercial-grade hardware. The objective of this research project is to explore cutting-edge techniques for reducing the size of LLMs, such as weight pruning, structural pruning, and other similar methods. The project begins with the implementation of existing techniques on various LLMs, including OPT, Phi, LLama, and others. With a thorough understanding of the limitations of current methods, novel approaches can be proposed to address these limitations.

Research Tasks: Research Tasks:
a. Literature Review on Large Language Models and Model Compression
b. Implement previous model compression methods for Large Language Models
c. Improve the previous model compression algorithms based on the understanding of the Implementation.

Skills that research assistant(s) may need: Programming skills in Python are required.
Knowledge of Linear Algebra and Probability are required.
Experience with Pytorch is highly recommended.