LoRA

Sun, 07 Jul 2024 00:00:00 +0000

LoRA (Low-Rank Adaptation)#

LoRA is one of the PEFT (Parameter-Efficient Fine-Tuning) techniques.
This technique efficiently fine-tunes large pre-trained models for specific tasks.

(The following content is referenced from the paper “LoRA: Low-Rank Adaptation of Large Language Models.”)

Background/Problem#

Models like LLM (Large Language Models) have an extremely large number of parameters.
For example, the llama3 model, released in April 2024, has about 70 billion parameters and a file size of over 40GB, with many models being even larger.
Full fine-tuning of such large models requires high-performance GPUs and considerable training time.
Additionally, fully fine-tuning the base model may potentially degrade the fundamental performance learned during pretraining.

Fine tuning on Semonan Book

LoRA

LoRA (Low-Rank Adaptation)#

Background/Problem#