What are the most common techniques for retraining machine learning models?

Incremental learning or Online learning is a very efficient way to learn upon dynamic data sources. This should be complemented with model monitoring system. Where one continuously evaluates model via drift detection mechanisms. There could be 2 key things that could be happening at a high level: * Data drift * Model drift / Concept drift Data drift happens when the underlying feature distribution of data is changing. The model's accuracy will be effected as it was trained upon a different data distribution. For such cases Incremental learning is ideal as it adapts to the new feature distribution. Concept drift or model drift happens when the relation between target variable and data changes, in such case it better to retrain the model.

Batch learning is a widely utilized concept, particularly in academic settings. However, in dynamic environments where data updates are essential, such as responding to user feedback or handling continuous data streams, batch learning may not be the most suitable approach. In these scenarios, continual learning emerges as the preferred method, offering the flexibility needed to adapt to evolving datasets and incorporate real-time information.

Transfer learning is a popular technique to capitalise on the learnings of large models either in image or text domain. With the advent of LLMs (Large Language Models) this technique has gotten high prominence again as it is practically not possible to re-train such massive models. It has another popular name now called "Fine-Tuning", but the idea is same. When one freezes the layers of the large model then it is referred as Transfer Learning and when one doesn't freeze the layers and allows layers to be modified it is called Fine Tuning. So when one wants to add additional knowledge without modifying existing knowledge in a LLM, then Transfer Learning is a good choice. When one wants to update LLMs knowledge then Fine-Tuning is better.

Despite its advantages, active learning faces computational efficiency challenges. Iteratively querying the model for the most informative samples can be resource-intensive. Striking a balance between the computational cost of active learning and the model's learning efficiency is crucial for practical implementation.

Supervised learning has been effective in building efficient ML models, but it gets hindered due to lack of labeled data. RL (Reinforcement Learning) has been able to find a mid way where one can generate dynamic labels / feedback through an iterative process. It has been highly effective in solving ML solutions where there is a reactive process like self-driving cars, playing a game, as after every move one can know was it a right move or a wrong move. Even the phenomenal breakthrough in Gen-AI has been brought by RLHF, where humans provided continuous feedbacks to improve Instruction tuned LLMs and the result is chatGPT and other such models. RL will continue be a prominent tool for ML models where one can provide continuous feedbacks.

Deep reinforcement learning combines RL with deep neural networks to handle complex and high-dimensional input data. The RL process typically involves the following steps: 1. Observation: The agent observes the current state of the environment. 2. Action: The agent selects an action based on its policy. 3. Transition: The environment transitions to a new state based on the agent's action. 4. Reward: The agent receives a reward or penalty based on the new state. 5. Learning: The agent updates its policy based on the observed outcomes to improve its decision-making in the future. Indeed, reinforcement learning comes with its own set of challenges, and addressing these problems is crucial for the successful application of RL algorithms.

Online Learning: Online learning updates the model in real-time as new data arrives in a continuous stream. It is particularly useful when the data is dynamic and evolving, enabling the model to adapt quickly to changing patterns. Ensemble Learning: Ensemble learning combines multiple individual models, called base learners, to make predictions collectively. Retraining can involve updating the base learners with new data or replacing some of the base learners with new ones. This technique improves model performance and reduces overfitting.

When dealing with LLMs with billions or trillions of parameters, fine-tuning is computationally intensive and often limited by the available hardware. Parameter Efficient Fine-Tuning (PEFT) offers an alternative to full fine-tuning by updating only a subset of the model weights. A popular PEFT technique called Low-Rank Adaptation (LoRA) involves freezing the original model weights (say self attention layer), using matrix factorization to create two low-rank matrices, and then training the smaller matrices. During inference, the smaller matrices' product matrix weights are added to the original frozen weights and replaced in the model. It significantly reduces memory requirements and costs and is less susceptible to catastrophic forgetting.