How do you update AI models for changing data?

Data drift is an important concept when machine learning & AI models are in production. A model is trained with a given data (based on training & testing scenarios). Once the model is validated, it is productionize. In extremely simple terms, the data drift scenario occurs, when the model is not able to identify a specific data scenario and the overall model accuracy drops. It is the change in the data - which can occur gradually or suddenly depending on what is the source of the data, at frequency it is coming in, the volume, and the variety of the data. Using anomaly detection models, data drift monitors, and other ways, data drift can be detected, and accordingly AI models can be retrained for understanding new scenarios.

Consider a sales prediction model for a retail store. It could experience concept drift if customer behavior shifts due to factors like new product offerings or pricing changes. Covariate shift might happen if there's a change in customer demographics, such as if the store starts catering to a new population. Label drift could emerge due to alterations in inventory systems, while domain drift may stem from expansion into new locations, where the customer preferences may be different.

Data drift is something that is bound to happen specially if the model is runs for long a time in production. It is essential to mal out a pipeline that periodically looks for anomaly and then steps to fix these changes in data can be taken it. It can be a complex pipeline like an MLOPS solution that continously feeds new data and improves the model. Or, it could be something simple like looking at the output say once a month and determine if the accuracy metrics reflect a change in data and if any retraining or fine tuning is needed.

There are different kinds of data drifts: Concept drift - if the business definition of something changes.Solution is to align with business. Input/Output data drift -if the input/output(target) data distribution changes. Ways to handle tabular input/output data drifts: 1. Frequent retraining and performance comparisons using defined metrics.Flipbook Backtesting over time helps a lot. 2. Do training vs production data distribution checks to detect out of distribution scores using strong statistical tests such as KS test for continuous features, chisquared test for categorical features.Aggregation at a data level can be done using more conservative approach such as bonferroni inequality or relaxed approach such as False Discovery Rate (FDR).

No se puede decir que un modelo está puesto en producción si no se está monitoreando. Dentro del marco MLOps el monitoring es fundamental puesto que nos servirá para conocer cuando debe reentrenarse nuestro modelo y así ser robusto. Para ello existen conceptos como el Data Drifting o el Concept Drifting: el primero de ellos para medir si los datos con los que fue entrenado nuestro modelo son similares a los que están llegando en producción; el segundo, para conocer si nuestro modelo puede haberse vuelto obsoleto debido a cambios en el entorno que al fin y al cabo afectarán al desempeño, por ejemplo la llegada del COVID seguro que afectó a muchos modelos ya en producción.

Try to build the model initially to take into account known potential changes. When possible, use adaptive coding rather than hard coding for values that may need flexibility.

To keep AI models updated for changing data and situations, a process called "fine-tuning" is often used. This involves retraining the model with new data to adapt to evolving patterns and contexts. Regular monitoring, feedback loops, and continuous training help ensure that the model stays relevant and accurate over time.

Navigating data drift involves weighing options like re-training and re-building AI models. While re-training with tools like TensorFlow or scikit-learn is faster and cheaper, it may not suffice for complex models or significant drift. Re-building with platforms like PyTorch or H2O.ai is time-consuming but necessary for radical drift or outdated models. Factor in tools' capabilities for optimal adaptation.

Retraining: One straightforward approach is to retrain the model periodically on a mix of old and new data. This ensures that the model stays up-to-date with recent data trends. Incremental Training: For some models (e.g., online learning models), you can incrementally train on new data without starting from scratch. Fine-Tuning: If the model architecture allows, you might only need to fine-tune certain parts of the model on new data, rather than retraining the entire model. This approach is often used with deep learning models. Ensemble Methods: Sometimes, adding a new model trained on recent data to an ensemble of existing models can improve performance without discarding the knowledge captured by the older models.

Retraining or rebuilding a model automatically is essential to maintain model performance as data evolves. One popular Python package for this is MLflow. It provides tools for tracking model versions, automatically retraining when new data arrives, and deploying updated models seamlessly. By setting up an automated pipeline with MLflow, you can ensure that your machine learning models adapt to changing data patterns, improving their accuracy and relevance over time.

Validate and Test: Before deploying updated AI models, validate and test them rigorously. Ensure they align with the data and problem definition while evaluating their performance and robustness. Utilize methods like cross-validation, error analysis, or confidence intervals. Compare the updated models with previous versions and benchmarks to assess improvements and value.

Before deploying updated AI models, thorough validation and testing are crucial. Validation ensures alignment with data and problem definition, while testing assesses performance and robustness. Methods like cross-validation, error analysis, and confidence intervals aid in validation and testing. Comparing updated models with previous versions and benchmarks gauges improvement and value, ensuring effective deployment.

Cross-Validation: Employ techniques like cross-validation to assess how well your updated model generalizes to unseen data. Testing Scenarios: Test the model under different scenarios and potential edge cases that may arise with changing data.

Firstly we will check the model performance by looking at decile view (split of population and target in 10 equal parts) like which decile is impacted the most, for eg: we can see their is a major change in top two decile then immediately we can do missing value imputation for that particular variable. This can be done by checking at the significant variables and treat the variable separately and parallely we can investigate the root cause like is it a one time change or that variable will be changed frequently on the basis of that we can treat the new values coming separatley as a quick work around till the time model will be refreshed. Check for model assessment on weekly/monthly basis to see the performance.

After updating the models, it's important to validate their performance using a separate validation dataset. Rigorous testing ensures that the updated models generalize well to new and unseen data. Techniques like cross-validation can be employed to assess model performance.

Monitoring AI models is critical! This phase gets sometimes neglected by project team. However, the impacts of letting AI models unmonitored could be significant. There is the exmaple of Zillow AI, which uses AI in real estate to value the home market where the model drifted because it was not updated to reflect changes in market conditions, which results in significant losses.

Deploying an AI model using FastAPI and monitoring with Prometheus is a robust approach. FastAPI simplifies API creation, while Prometheus offers powerful monitoring capabilities. Start by creating a FastAPI app to serve your model predictions, ensuring RESTful endpoints. Then, instrument your app with Prometheus client libraries to expose metrics like request counts and latencies. Prometheus can scrape these metrics for real-time monitoring and alerting. With this setup, you can deploy, monitor, and scale your AI model effectively, ensuring its reliability and performance.

Validate, test and monitor your models... continuously! Data do drift... models need to be adapted! And improve them continuously on the fly! PS: you probably want to use some kind of hybridisation to do that...

Deployment: Once validated, deploy the updated model in a controlled environment. Gradually transition to the new model in production. Monitoring: Implement continuous monitoring to detect any unexpected behavior or performance degradation in the deployed model.

Deploy the updated models to production systems and implement real-time monitoring mechanisms. Monitoring allows for the tracking of model performance, and alerts can be set up to notify of any deviations from expected behavior, indicating potential issues.

Documentation is crucial for model pipelines, but documentations should also involve a business specific documentation describing the business logic, requirements and vital stakeholders. Business documentation is able to provide context, explaining how the model aligns with objectives, its ROI, and potential risks. Of course there is also a need of technical documentation which outlines the pipeline's architecture, data flows, and model details, aiding collaboration among data scientists and engineers. It ensures reproducibility and troubleshooting. Both types guide decision-making, foster transparency, and help stakeholders understand, trust, and effectively utilize the AI system, which is vital for a data science team.

Document the changes made to the models, outlining the reasons for the update and detailing the impact on performance. Effective communication of updates to stakeholders, including data scientists and users, is essential. Provide documentation on any new features, improvements, or considerations introduced with the updated models.

Documentation: Keep comprehensive records of the update process, including changes in data, model versions, and performance metrics. Communication: Inform stakeholders, users, and team members about the update. Clearly articulate the benefits and implications.

As with all technical change, the more you can involve the users in the process, the less shock there is for the business. Internal representatives of the real customer are a good proxy when you're talking about thousands of customers, but the 'traditional' skills of UX, change management and customer collaboration don't go away just because you're using AI.

To document and effectively communicate the update of an AI model, it is necessary to follow a structured approach: 1 - Record the technical details of the update, including changes made to the data, algorithms, and model parameters. 2 - Prepare a clear and accessible summary explaining the potential impacts of the update on outcomes, as well as the reasons for these changes. 3 - Communicate this information transparently to stakeholders, using appropriate channels such as reports, meetings, or publications on internal or public platforms, to ensure that everyone understands the implications of the update.

Even more, plan that data do drift from the beginning... and if you need to use some kind of hybridisation to do that, well... do that! It is one of the reason why hybridisation, in particular hybridisation of ML and OR, is so powerful!

Schedule Regular Updates: Establish a schedule for routine model updates, considering the rate of data change and business requirements. Automate When Possible: Automate the update process where feasible, reducing manual effort and ensuring timely updates.

Pour planifier les futures mises à jour des modèles d'IA, il est nécessaire de : - Identifier les besoins spécifiques de mise à jour en fonction des évolutions technologiques et des retours d'expérience des utilisateurs. - Établir un calendrier de mises à jour régulières en intégrant des phases de test et de validation pour minimiser les risques d'erreurs. - Mettre en place une veille technologique pour anticiper les innovations et ajuster les modèles en conséquence. - S'assurer de documenter chaque mise à jour pour une traçabilité et une compréhension claire des modifications apportées.

Few more areas to focus on when any AI/ML models are into production: 1. Monitoring the data drift constantly 2. Monitoring the model drift 3. Ensuring the model is fine-tuned and optimized to understand different varieties (scenarios) about the data 4. Scheduling model re-training at any given frequency (to ensure model keeps on improving itself time & again). 5. Keeping a track of the overall statistical metrics for the model & making sure it is aligned with the business goals.

Often in production, we can get data which was incorrectly fed to the pipeline. These user errors are very difficult to ascertain but can greatly impact the performance. It is vital during the initial phase to see how users are interacting with the model and have some heuristic approach to filter out user errors. A good and extensive testing procedure can help, but users can always surprise you!

An important aspect is to define triggers to identify that the current ML model is performing poorly on newer data. Also, this change in performance is not short term seasonality. Once a valid trigger is received, the model re-training pipelines can be run and the newer model be evaluated based on its overall performance accuracy over older and newer data. It is often important to move very old data out from the re-training pipeline while assimilating newer data, to keep the model relevant while drifting.

A recent article (DZiQ) stated that 87% of AI projects don't go to production because productionalization is often an afterthought as most organizations focus on model experimentation or building POC. MLOps framework should be part of end to end Data Science workflow. 1. Build a solid DevOps framework (focussed on productionalizing ML projects) 2. Version control your code, training data and model 3. Train and deploy the best model (with traceability from for each experiments ex: MLFlow) 4. Monitor the data drift and model drift 5. Retrain models either on a schedule or whenever drift happens 6. Review retrained model performance via shadow or canary deployment 7. Release the best performing model and repeat the process again as required!