Jason Fishbein’s Post

Synthetic data is a controversial topic in AI circles, but I’m very bullish on it. Here’s why ⬇️ 1. Cost-efficiency 💰 Collecting and processing data is extremely expensive. It is labor-intensive, prone to human error, and time-inefficient (often taking weeks / months to collect the right data). Synthetic approaches reduce these costs to compute, which is magnitudes cheaper. 2. Increased privacy 🔒 Synthetic data retains the statistical characteristics of real-world data without compromising personally identifiable information. This creates safer, more compliant models. 3. Increased representativeness ↔ In testing environments, having access to datasets that cover a wide range of use cases is crucial for validating the robustness and adaptability of a model. Synthetic data facilitates the generation of scenarios that might be rare or challenging to capture in the real world. 4. Mitigated bias ⚖ Biased training data leads to biased models, potentially impacting real people in dramatic ways. Synthetic data mitigates this by generating examples that represent a more comprehensive spectrum of the population. Synthetic datasets can also be endlessly updated and refined with new information and considerations, which is much harder to achieve with expensive real-world data collection. 5. Better simulations 🕹️ Simulating counterfactuals is a powerful way to make an AI system more robust and capable. Synthetic data can help us come up with better counterfactuals, such as unexpected user behaviour in a recommender engine or unforeseen collisions in a self-driving car. 6. Faster development cycles ⚡ As mentioned in (1), generating data quickly and at scale eliminates the need for teams to spend weeks trying to get their hands on a dataset. This enables more experiments, iterations, and deployments. 7. Data sharing and collaboration 🤝 This goes in hand with privacy considerations. Unlike real-world datasets, which are legally and ethically constrained, synthetic data can be shared, modified, and updated effortlessly between various stakeholders. ⚠️ A few caveats: - Not all synthetic data is good data. - Synthetic data ≠ fake data. It still captures the features, signals, and distributions of real-world data. - Synthetic data does not mean the end of real-world data. Quite the contrary — it forces us to find new and better ways to collect high-quality data in the real world. On the long run, hybrid datasets will win. - Many situations can’t be represented with synthetic data. In fact, many situations aren't accurately represented with real data in the first place. - All synthetic data comes from a real-world kernel. In that sense, synthetic data is a reflection of the original data. - To the previous point: there is a natural trade-off between fidelity and privacy. High fidelity = closer to the real world, easier to trace back to real people. Low fidelity = more creative data, further from the original dataset, but also potentially less reliable.

Optimizing the Potential of Generative Artificial Intelligence with Broader Data Utilization In several seminars, I've often noticed that some participants are amazed at how cool generative AI can be with its language, reasoning, and translation capabilities. But unfortunately, this AI, which is built from public data, can't reach its full potential in the corporate world unless it's combined with company data storage... right? ChatGPT and Gemini Pro are really smart when asked about mathematical or general things, but when it comes to questions involving our company's data, they seem clueless... hehe. Now, most companies today store a huge amount of data, both on-premises and in the cloud. Many of these businesses already have data science practices using structured data for traditional analytics, such as making predictions. However, to get the most value from generative AI, these companies need to start exploring unstructured and semi-structured data as well. According to a February 2021 MIT report titled “Tapping the Power of Unstructured Data,” 80 to 90 percent of data is unstructured—hidden in texts, audio, social media, and other sources. Companies that can leverage this data can gain a competitive advantage, especially in this era of generative AI. The question is, how aware are our companies of this and what have they prepared to extract these unstructured and semi-structured data? Has the MIS (Management Information System) been able to handle unstructured data? So how do you collect comprehensive data? First, you need to understand the types of data sources that can be used in the future: First-party data is internal data generated through daily business interactions with customers and prospects. Second-party data is generated in collaboration with trusted partners, such as product inventory data shared with e-commerce or retail sales channels. Third-party data can be obtained from external sources to enrich internal data sets, such as manufacturing supply chain data and financial market data. So, if you want to boost the potential of generative AI in your company, start by opening up to unstructured and semi-structured data, and don't forget to maximize all available data sources. For more details, check out the nearest article for further information! Keep exploring and stay ahead, folks! 🚀✨ Pondok Aren May 20, 2024 Ha-Er-Weh

"The first problem you solve is how you will secure the data.." Heads of Data Architecture Heads of Artificial Intelligence Heads of Information Technology Heads of Data Science On average, I speak with 2-4 leaders in IT, AI, and Data daily to gain a comprehensive view of their challenges in moving AI initiatives forward. Data security is problem #1 Generative AI behaves far differently than other systems In a business even foundational systems of AI like Machine learning, natural language process, and deep learning AI's ability to come to plausible-sounding conclusions creates a problem in terms of understanding how AI makes decisions. If enabled to have access to the Sensitive databases and systems of a company Will AI understand what it should and should not provide As an answer to a prompt from a mid-level employee Who is asking a question about business strategy AI in its access to the company's financial data might interpret a short-term cash crunch due to an acquisition with a need to disclose to the employee that the business needs to focus on short-term profits due to financial pressures. That is a concern that requires creative problem-solving and The goal of the organization's use of AI is the most important factor Each of the big 4 requires a different approach to generating Usable synthetic data sets to obtain desired outcomes 1) Revenue Generation Market-level data is available from multiple government sources that track key economic trends that can be modeled against a synthetic data set that models customer behavior correlations using sentiment analysis across correlating data points to purchasing drivers. 2) Cost Reduction A synthetic data set can be created to establish the mean, standard deviation, upper control limit (UCL), and lower control limit (LCL) and apply statistical process controls to gauge the accuracy of operating costs integrated with real-time interest rates and financial markets data. 3) Operating Efficiencies Accurately measuring the existing process inputs and output and translating those observations into the UCL, LCL, mean, and standard deviation that defines the synthetic operating model of the company and training AI to interpret the synthetic data to provide recommendations against the publically available Lean, Six Sigma, SPC, Kaizen, and continuous improvement methodology data. 4) Increase capabilities The objective of the exercise define the dataset requirements, if the innovation is revolutionary there may not be enough existing data to accurately project the market-level reaction to the idea. The synthetic dataset would need to resemble the dataset of an LLM, AI would need fine-tuning to see possibilities where others have seen challenges rooted in philosophical thinking. If data security is the greatest challenge to AI adoption Then we may first look to synthetic to harness the power of AI Without exposing sensitive company data to the model AI only needs contextual understanding

Data governance and data quality are essential in building effective AI models. As highlighted in the article, ensuring your data is accurate, structured, and compliant is critical to the success of any AI strategy. Poor data leads to poor outcomes. Organizations must establish strong governance frameworks to manage data quality before it's ingested by AI models. Remember, the quality of your AI outputs depends on the quality of your inputs—garbage in, garbage out. https://lnkd.in/eg3VtgjV

𝐓𝐡𝐞 𝐕𝐢𝐭𝐚𝐥 𝐑𝐨𝐥𝐞 𝐨𝐟 𝐌𝐨𝐝𝐞𝐫𝐧 𝐃𝐚𝐭𝐚 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦𝐬 𝐢𝐧 𝐚 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈-𝐃𝐫𝐢𝐯𝐞𝐧 𝐖𝐨𝐫𝐥𝐝 #dataplatform #Data #AI In the rapidly evolving landscape of technology, the advent of generative AI has ushered in a new era of innovation and transformation. As these AI systems become increasingly sophisticated, the role of modern data platforms becomes crucial, serving as the backbone for these technologies. A modern data platform not only supports the heavy demands of generative AI but also enhances its capabilities, leading to more efficient, innovative, and intelligent applications. 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 𝐌𝐨𝐝𝐞𝐫𝐧 𝐃𝐚𝐭𝐚 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦𝐬 A modern data platform integrates various data management and processing tools into a cohesive framework. It is designed to handle vast volumes of data, support real-time processing, and provide robust data security and compliance features. These platforms typically include components like data lakes, data warehouses, and advanced analytics tools that work seamlessly to support data-driven decision making. 𝐓𝐡𝐞 𝐈𝐦𝐩𝐨𝐫𝐭𝐚𝐧𝐜𝐞 𝐨𝐟 𝐃𝐚𝐭𝐚 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦𝐬 𝐢𝐧 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 Generative AI, which includes technologies capable of producing content like text, images, and music, relies heavily on large datasets. These AI models learn from the data they process, making the quality, speed, and scalability of the data platform critically important. Scalability and Performance: As AI models grow in complexity, the data platforms must efficiently scale to handle increased loads and perform high-speed data processing to train AI models without bottlenecks. Data Quality and Diversity: A robust data platform ensures that the data fed into AI systems is not only high-quality but also diverse. This is crucial for training generative AI models that are unbiased and effective across various scenarios. Security and Compliance: With AI models accessing vast amounts of potentially sensitive data, modern data platforms must adhere to strict security protocols and compliance regulations to protect data integrity and privacy. The future of modern data platforms in the generative AI world looks promising. With advancements in cloud technologies and machine learning algorithms, these platforms are set to become more intelligent, autonomous, and integrated. This evolution will likely usher in a new wave of innovations and might redefine what machines can do. As we delve deeper into the generative AI age, the importance of robust, efficient, and secure data platforms cannot be overstated. They are not just supporting infrastructure but strategic assets that drive the AI capabilities forward, making groundbreaking advancements possible across all sectors of society. As these technologies continue to develop, the synergy between modern data platforms and generative AI will become even more critical, shaping the future of technology and its impact on the world.

Do you think companies that use synthetic data generated by AI would be at a bigger risk of AI models collapsing and getting dumber? Using synthetic data generated by AI can offer significant benefits for training and validating models, such as improving data diversity and overcoming privacy issues. However, it also introduces specific risks that could affect the performance and reliability of AI models. Here’s a closer look at the potential risks and considerations: Potential Risks of Using Synthetic Data Data Quality and Representativeness: Overfitting to Synthetic Data: If AI models are primarily trained on synthetic data, there’s a risk they might overfit to the synthetic patterns rather than generalizing well to real-world data. Synthetic data might not capture all the nuances and variability of real-world scenarios. Bias and Inaccuracy: Synthetic data might inadvertently introduce biases or inaccuracies if the generation process does not adequately reflect real-world distributions or if the underlying model used to generate the synthetic data has biases. Loss of Real-World Nuance: Limited Complexity: Synthetic data may not always capture the full complexity of real-world environments, especially if the generative model is not sophisticated enough. This can lead to AI models that perform well in controlled scenarios but fail in more complex or varied real-world situations. Validation and Testing Challenges: Validation Limitations: Relying heavily on synthetic data for model validation and testing can be problematic if it doesn’t accurately represent real-world conditions. This could result in models that perform well on synthetic benchmarks but poorly in practical applications. Risk of Collapse: If the AI models are trained on synthetic data that doesn’t align well with real-world data, they might encounter issues when deployed. The models could exhibit unexpected behavior or performance degradation, potentially leading to what might be perceived as "collapsing" or becoming less effective. Generative Model Dependency: Quality of Generative Models: The quality of synthetic data depends on the generative models used to create it. If these generative models are flawed or limited, the synthetic data they produce might not be reliable, affecting the performance of the AI models trained on such data. Mitigation Strategies Hybrid Approaches: Combining Synthetic and Real Data: To mitigate risks, companies can use a combination of synthetic and real-world data. Synthetic data can supplement real data, especially in cases where real data is scarce or sensitive, but it should not replace it entirely. Robust Validation: Cross-Validation with Real Data: Ensure that models trained with synthetic data are rigorously validated with real-world data. This helps in assessing their performance and robustness in practical scenarios.

"Aligning Data Governance and Generative AI" -- Article By Kenneth Chisholm -- Published By Daneshmand Enterprises "As we have all seen in the last few months there have been tremendous breakthroughs and awe-inspiring demos in the field of Generative AI. ChatGPT, Google Gemini, Perplexity, Github Copilot have demonstrated Generative AI solutions for answering questions, content creation, holding conversations, sentiment analysis, text classification, entity recognition, code generation and more. The results, while not always perfect, are enough to start solutioning for a head start in the AI race. The question is how can Data Governance connect and align with Generative AI? If we look past the hype and properly account for Generative AI’s strengths and current weaknesses, we can see there are strategic points of alignment. ..." --Further Reading at https://lnkd.in/eQdRTiSS

Check out the article I wrote for my friend's company blog about "Aligning Data Governance and Generative AI"

Businesses from diverse domains are turning to synthetic data generation for their AI and ML operations: a game changer in serving their needs for high-quality data without compromising privacy. In alignment with the latest trends in data for AI, AI-DAPT EU Project will develop the Synthetic Data Generation Engine, led by Suite5 Data Intelligence Solutions. A tool that will: 1) Facilitate synthetic data generation, addressing the problem of limited and inappropriate data for AI; 2) Allow the assessment of synthetic data fitness-for-purpose, aiming to close the simulation-to-real gap of synthetic data and eventually lead to improved model performance. Interesting? Check the following article for a quick glimpse at trends in synthetic data: https://lnkd.in/dFmmTG2 #AI #Adaptation #SyntheticData #ML #DataGeneration

Kamal Ahluwalia, Ikigai Labs: How to take your business to the next level with generative AI AI News caught up with president of Ikigai Labs, Kamal Ahluwalia, to discuss all things gen AI, including top tips on how to adopt and utilise the tech, and the importance of embedding ethics into AI design. Could you tell us a little bit about Ikigai Labs and how it can help companies? Ikigai is helping organisations transform sparse, siloed enterprise data into predictive and actionable insights with a generative AI platform specifically designed for structured, tabular data.   A significant portion of enterprise data is structured, tabular data, residing in systems like SAP and Salesforce. This data drives the planning and forecasting for an entire business. While there is a lot of excitement around Large Language Models (LLMs), which are great for unstructured data like text, Ikigai’s patented Large Graphical Models (LGMs), developed out of MIT, are focused on solving problems using structured data.   Ikigai’s solution focuses particularly on time-series datasets, as enterprises run on four key time series: sales, products, employees, and capital/cash. Understanding how these time series come together in critical moments, such as launching a new product or entering a new geography, is crucial for making better decisions that drive optimal outcomes.  How would you describe the current generative AI landscape, and how do you envision it developing in the future?  The technologies that have captured the imagination, such as LLMs from OpenAI, Anthropic, and others, come from a consumer background. They were trained on internet-scale data, and the training datasets are only getting larger, which requires significant computing power and storage. It took $100m to train GPT4, and GP5 is expected to cost $2.5bn.  This reality works in a consumer setting, where costs can be shared across a very large user set, and some mistakes are just part of the training process. But in the enterprise, mistakes cannot be tolerated, hallucinations are not an option, and accuracy is paramount. Additionally, the cost of training a model on internet-scale data is just not affordable, and companies that leverage a foundational model risk exposure of their IP and other sensitive data.   While some companies have gone the route of building their own tech stack so LLMs can be used in a safe environment, most organisations lack the talent and resources to build it themselves.  In spite of the challenges, enterprises want the kind of experience that LLMs provide. But the results need to be accurate – even when the data is sparse – and there must be a way to keep confidential data out of a foundational model. It’s also critical to find ways to lower the total cost of ownership, including the cost to train and upgrade the models, reliance on GPUs, and other issues related to governance and data retention. All of this leads to a very different set of solutions than wha...