The unprecedented scale of foundation models (foundation language models and vision transformers) enables extraordinary capabilities in natural language processing, computer vision, and multimodal reasoning tasks. However, such large-scale pre-trained models are primarily designed to operate on the cloud with high-end graphics processing units (GPUs) and extensive memory. Moving such models to resource-limited hardware on the edge involves significant engineering challenges. Edge devices such as smartphones, autonomous cars, IoT hardware, and industrial sensors have limited power budget, thermal dissipation budget, onboard memory, and memory bandwidth. Therefore, running such large-scale foundation models is nontrivial on the edge due to the increased computational costs and long inference time. Computer scientists use model optimization techniques, including quantization, pruning, and low-rank adaptation, to make such large-scale machine learning models efficient on edge devices.
Source: https://www.projectpro.io/article/llm-quantization/1086
Quantization is a model optimization technique that reduces a model’s memory footprint and computational requirements. Foundation models are typically trained with weights represented with 32-bit floating-point numbers (FP32). These large models require higher memory bandwidth for moving activations and weights between the memory and the processor for computations during the inference step. Quantization reduces memory bandwidth consumption by converting FP32 weights to lower precision (e.g., 8-bit or 4-bit integers). Post-training quantization involves quantizing already trained large-scale neural networks. On the other hand, Quantization-aware training involves training the neural networks with noisy quantized weights. The converted weights enable the neural network to be portable during the inference stage while maintaining the model’s accuracy. Therefore, quantization is critical in reducing the energy consumption during inference and improving the portability of foundation models on edge devices because it reduces memory footprint and computational requirements. Pruning is another model optimization technique that reduces the memory and computational requirements of neural networks by removing weights, channels, or parameters that contribute minimally to a model’s performance during the inference step. Pruning involves training a neural network to have sparse weights, where some weights are zero and do not contribute to the computation. Pruning algorithms are mainly of two types: unstructured and structured pruning. Unstructured pruning algorithms randomly set some weights in a neural network to zero during training. The zero weights are typically removed during the inference step, resulting in a smaller network with fewer parameters. However, unstructured pruning algorithms usually result in sparse weights, which are not amenable to efficient matrix multiplications. Therefore, unstructured pruning algorithms typically require specialized accelerators during the inference step. Structured pruning algorithms, on the other hand, involve permanently removing parameters, channels, or tensors during training. Structured pruning results in dense weights, and the inference procedure only needs standard linear algebra operations. Pruning algorithms help reduce the peak memory consumption during the inference step while also reducing the number of operations (FLOPs) needed during the computation. The reduction in FLOPs translates to improved battery life and reduced power consumption on edge devices since they consume less processing power. Fine-tuning already trained foundation models, such as the Llama series, is crucial in most applications but is usually memory- and computationally expensive. Low-Rank Adaptation is a model optimization technique that reduces the memory consumption and computational requirements of fine-tuning. Low-Rank Adaptation makes the fine-tuning process memory-efficient by first freezing the weights of the foundation model and introducing adaptation layers that are trainable. These adaptation layers typically have much lower rank (number of parameters) and only modify specific parts of the model. Therefore, Low-Rank Adaptation reduces the memory consumption and computational requirements of fine-tuning a large-scale foundation model. Recent research in Low-Rank Adaptation involves combining model optimization techniques, such as using lower-precision (quantization) weights in the base model and introducing adaptation layers with low-rank parameters, to further reduce the memory consumption of fine-tuning processes in resource-constrained edge devices. Researchers can combine different model optimization techniques to achieve better results in reducing the memory consumption and computational requirements of large-scale foundation models (e.g., CLIP or Flamingo). Combining pruning algorithms with quantization techniques can significantly reduce the memory footprint and computational requirements of a given foundation model. The pruned and quantized model can then be fine-tuned using Low-Rank Adaptation to introduce some model-specific adaptations while reducing the memory consumption and computational requirements during the fine-tuning process. Advanced model optimization techniques such as pruning, quantization, and Low-Rank Adaptation will be increasingly important in enabling the large-scale adoption and deployment of artificial intelligence on both the cloud and the edge.
Sources
https://www.mdpi.com/2073-431X/15/10/660 https://www.researchgate.net/publication/391165113_Efficient_LLMs_for_Edge_Devices_Prunin g_Quantization_and_Distillation