How do you leverage AI and ML to enhance your BI capabilities and insights?

While most use AI and ML for data analysis, there is potential in using these technologies to convert unstructured documents into structured data. According to Microsoft, 72% of an organization's data is stored in unstructured files. By utilizing AI and ML, companies can streamline their processes and gain valuable insights from this previously untapped data. Microsoft Syntex allows businesses to unlock their unstructured data, using ML and AI technologies to map and extract data from various document types, including PDFs and emails. The extracted data can then be seamlessly integrated with other structured data sources, enabling a holistic view of the organization's data and enabling more efficient operation.

IT leaders & technology savvy leaders from high value silos within very large organizations will be hungry for this solution. Expectations of big returns will remain vaporware unless/until the data playing field is leveled & made uniform. Pockets of noncompatible data won't do. Taking the "conclusion" out of the job description for this solution also makes success more likely. A good one.

You know the saying. Data Scientists love clean data, but they don't love cleaning data. About 70% of an ML project is spent on "data wrangling" We use Machine Learning models extensively to clean, normalize, and enrich data. In fact, we don't require data to be clean, complete, or in a data lake or data warehouse. Often that is a 9-12 month pre-requisite of getting the data in order before they begin. Using Machine Learning to assist in making sense of the data is a great accelerator.

The risk of understating the importance of including high-quality natural language processing (NLP) for this solution is a big one, and one to be avoided. If business leadership can't understand the insights, then the value of the insights is greatly diminished. The combination of Generative AI and Large Language Models lowers the difficulty of defending against this risk.

I have found that the power of AI/ML for data analysis is that it's dynamic. Most BI-based solutions are rules based. That doesn't scale when it comes to anomalies, really large or complex data sets. When an anomaly occurs (like Covid) you can throw your rules out of the window. You also will attempt to add more and more rules and that is unmanageable. With ML, you likely just need to retrain the model. The more data you have the better the prediction. There is a theoretical limit on how much data (and data sources) you can practically manage with BI. ML doesn't have the limit.

Leaving the task of creating the dashboard to AI makes a lot of sense. My personal experience struggling with the tasks of using the PowerPivot capability of Microsoft Excel and similar features of PowerBI amounted to a lot of wasted effort. Successfully using these features requires a pretty high level of "citizen developer" expertise. Offloading their operation to AI & ML will spread the benefits across a much wider audience.

Broadly speaking business leaders with big expectations for BI need to maintain a persistent connection to the dictum "garbage in, garbage out". At the same time they need to also familiarize themselves with the many forms "garbage" can take: ~ Incompatible data stores ~ Big repositories of unstructured data waiting to be tagged ~ Dashboards demonstrating data relationships off topic AI & ML are becoming much more useful as methods of cleaning up all this "garbage"

We need to demystify AI/ML. It's not the magic that many believe it is. It's actually some really simple concepts involving linear algebra and probability and statistics. It's also a technology (like mobile, wireless, etc). It's not a silver bullet. It's also something that will become ubiquitous over time. The biggest challenge is bringing data scientists into the fold. Applications developers are the most evolved and ahead of the curve with innovation to make their job more efficient. IT infrastructure folks are finally getting it with infrastructure as code. Data Scientists are very different animals. They use environments like Data Bricks, Spark Notebooks and even things like CICD and version control are foreign to them.