Florin Lungu’s Post

Data powers our world today in ways both seen and unseen. However, getting data from different systems to work together can be challenging! In my blog https://buff.ly/3IbMeWu, I tried to expand on the art of ELT (Extract, Load, Transform) to enable seamless data integration across disparate sources. Learn how the right ELT approach helps you: ➡️ Reduce data mapping complexity ➡️ Continuously integrate new data sources ➡️ Maintain accuracy with incremental data updates I have included some real-world examples, market research, use cases, and actionable tips you can apply to integrate your own data more effectively. Let me know your biggest data integration challenges in the comments. #datascience #datadrivenbusiness #challengeyourself #learnsomethingnew #ELT #datagovernance #dataintegration

🚨 New Post Alert: The Data PM Gazette🚨 After a month's break, writing about a topic closer to my heart: analytics for platform products. TL;DR: it's not the same as user metrics that you measure generally. So... 👉 What should you be measuring? 👉 How can you gather that data? 👉 How it's different from what the Engineering team measures? I'll answer that and much more in my latest substack. Read below. #datapm #dataproducts #platforms

🏁 Day - 98 Semantic and Metrics Layers: When data engineers think about serving, they naturally tend to gravitate toward the data processing and storage technologies—i.e., will you use Spark or a cloud data warehouse? Is your data stored in object storage or cached in a fleet of SSDs? But powerful processing engines that deliver quick query results across vast datasets don’t inherently make for quality business analytics. When fed poor-quality data or poor-quality queries, powerful query engines quickly return bad results. Where data quality focuses on characteristics of the data itself and various techniques to filter or improve bad data, query quality is a question of building a query with appropriate logic that returns accurate answers to business questions. Writing high-quality ETL queries and reporting is time-intensive, detailed work. Various tools can help automate this process while facilitating consistency, maintenance, and continuous improvement. Fundamentally, a metrics layer is a tool for maintaining and computing business logic. (A semantic layer is extremely similar conceptually, and headless BI is another closely related term.) This layer can live in a BI tool or in software that builds transformation queries. Two concrete examples are Looker and Data Build Tool (dbt). For instance, Looker’s LookML allows users to define virtual, complex business logic. Reports and dashboards point to specific LookML for computing metrics. Looker allows users to define standard metrics and reference them in many downstream queries; this is meant to solve the traditional problem of repetition and inconsistency in traditional ETL scripts. Looker uses LookML to generate SQL queries, which are pushed down to the database. Results can be persisted in the Looker server or in the database itself for large result sets. dbt allows users to define complex SQL data flows encompassing many queries and standard definitions of business metrics, much like Looker. Unlike Looker, dbt runs exclusively in the transform layer, although this can include pushing queries into views that are computed at query time. Whereas Looker focuses on serving queries and reporting, dbt can serve as a robust data pipeline orchestration tool for analytics engineers. How it started: 👇 https://lnkd.in/gFtwbqkV #dataengineering #dataengineer #dataanalytics #datascience #datanerd

Customer: “Should I use LookML or an ELT tool, like DBT or Dataform, to model and govern my data analytics?” Me: “Yes!” (both) LookML’s capabilities are often conflated with those of in-warehouse “ELT” transformation tools like Dataform and DBT.  As these tools appear to be similar in nature, it is often thought that users need to choose one over the other. This new blog post outlines why customers should be using both LookML and ELT tools in their data analytics stack, with a specific focus on the importance of LookML https://lnkd.in/gZaW4n2X #looker #analytics #googlecloud #dbt #bigquery #dataengineering #lookml

No one wants to admit it, but the universal semantic layer everyone is looking for is tables in your data warehouse. Looker ushered in the idea of the modern, cloud-native semantic layer. Obviously, it was tightly coupled with the BI tool. Even with APIs to query the model in other tools, 99.999% of the usage was in the tool itself. And when a new team outside the BI tool wants to use business logic, the natural idea is some disconnected semantic layer. The problem is that for a universal semantic layer to actually take root, the entire ecosystem needs to evolve to use it's semantics - every tool in your tool chain needs to tightly couple with said semantic layer. A handful of tools may do that, but the reality is most of your toolkit will not, and you're back to square one with yet another layer to babysit (and now you have two layers trying to work perfectly together to solve problems that could be in one place). The good news is that we already have a universal layer that every tool knows how to talk to - tables in your data warehouse and JDBC. The entire data ecosystem was built around these simple standards. Are they perfect at semi-additive measures or fan outs or other nuanced stuff - no they aren't, but they are universally accessible and every use case upstream was built to interface with them. So leave the nuanced semantics in your BI layer (ahem, Omni), but when it needs to be universal, make it universal.

Data is king, but managing it effectively is the real challenge. This post dives into ETL and ELT, two common approaches to data processing, helping you choose the right one for your business needs. Here's the gist: * ETL (Extract, Transform, Load): Ideal for structured data, complex transformations, and data quality control. Think batch processing and regulatory compliance. * ELT (Extract, Load, Transform): Perfect for real-time analytics, big data, and diverse data sources. Think stock market analysis and customer personalization. Choosing the right approach depends on factors like: > Data volume, velocity, and variety > Real-time requirements > Existing infrastructure and skills > Business goals and future scalability Still confused? Check out the full post for detailed explanations, use cases, and key considerations for selecting the best data strategy for your organization! https://lnkd.in/gyF-MU5b #etl #elt #datamanagement #analytics #businessintelligence #DataManagement

In today's business world, data is everywhere, but turning raw information into valuable insights requires more than just collecting it. With ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform), you can streamline your data transformation process for improved business intelligence. Choosing between ETL and ELT can significantly impact your company's performance. Want to dive deeper into these methodologies and see how they can empower your business?   Read more on our blog: https://lnkd.in/gYr2BiTK   Bhawana Khater Ritesh Dalmia Nilabh Bajpai   #letspriorise #ETL #ELT #BusinessIntelligence #DataTransformation #DataScience

The modern data stack is a powerful pattern for businesses to collect, store, and analyze data. It is a collection of tools and technologies that work together to make it easier to get insights from data. The modern data stack is typically cloud-based and includes tools for data extraction, transformation, loading, and warehousing. In this blog post, we will provide a short intro into the modern data stack. We will discuss the different components of the stack, as well as the benefits of using a modern data stack. Read more - https://lnkd.in/dRrY6h-A #dataengineering #datapipeline #datadomains #moderndatastack

How can your business leverage #genAI? Integrate CockroachDB into your strategy! Our latest Change Data Capture feature allows seamless integration with OLAP engines like Google's #BigQueryML / #VertexAI. Check out our reference architecture in the article below to learn more about how CockroachDB can enhance your genAI strategy. https://lnkd.in/g7r3atVu

In the age of Copilot, it is understanding and application which truly counts. Copilot (Data Factory, Data Science and Data Engineering, Data Warehouse and Power BI) is designed to enhance productivity, reduce complexity, and ensure that users can focus on generating insights and driving business value.