dbt Labs’ Post

Raise your hand if you think "Governance" is a four letter word! Now get ready to run when the four letter word of "Data" prefixes it! Why do most firms run from this endeavor? Do we really have preference to live in a place with issues like data silos, inconsistent reporting & metrics, inefficient processes, as well as a host of other ill effects? How do we get over the huge obstacles which often come with "Data Governance"? ...you know, things like - meaningless scope, negative bias, intimidating effort, stigmas of "unachievable". I fully acknowledge that I don't have all the answers. However, I was taught by my elders to face challenges head-on! The only way you can eat a elephant-sized meal is one bite at a time. My team and I chose to implement dbt Labs because we saw the potential of how it could bring data unification at Avetta. Now, we have recently began a fresh "Data Governance" journey. I am extremely excited to continue to leverage dbt Labs to help us overcome the elephant-size obstacles of "Data Governance".

The modern data stack problem 3/3: Scale Over the past week, I've highlighted two of the most pressing data challenges faced by scale-ups and large enterprises: skills and speed. These issues converge on a third, crucial challenge. As your data operation scales, untangling analytics engineering bottlenecks becomes a priority. ↓ You often skill analysts and hire analytics engineers (♡) to maintain consistent data models and follow proper engineering workflows. ↓ But as your company grows, so does the number of analytics engineers, and consequently, the costs. ↓ Eventually, the migration project to dbt will conclude. But business needs will continuously require data model changes, and at the same time, require analysts to return to their core expertise: analyzing data. ↓ And whenever you let go from your strict procedures, you find the same issues with duplicates, inconsistencies, and silos reemerging, now in your dbt models. Here's a surprising fact: → Nearly every large company I speak with ends up shifting dbt responsibilities back to engineers, away from analysts! *** As we bring dbt into our daily practices, we find that inconsistencies and conflicts remain an issue, and sometimes might become exacerbated by the additional workflow to keep dbt and the BI tools in sync. We have the engineering tools and solid understanding for data modeling within dbt and syncing up with BI tools like Looker, Tableau, or Power BI. But a reverse flow -- capturing the inception of core business logic crafted by analysts and shifting it into dbt -- remains a territory awaiting innovation. *** Thanks for following up on this mini-series on the modern data stack. Check out this past week's posts in the comments. What do you think? P.S. Euno expands dbt's governance capabilities into the BI layer, enabling analytics engineers to govern business logic proactively and retroactively without slowing down business analysts. DM me if this resonates with you.

LookML or ELT? Three reasons why you need LookML LookML is a powerful tool for business logic and governance in data analytics. However, its capabilities are often confused with in-warehouse "ELT" transformation tools like Dataform and DBT. It's important to use both LookML and ELT tools in data analytics stack, with specific focus on the importance of LookML. **(50 words)** Read more on https://lnkd.in/dyqjxwfn

This is a great post from Jamie Davidson. We've seen a lot of innovation in data in the past few years and this is great. So many new ideas and ways of working with data. Consolidation is coming though. It's simply better for companies to manage fewer relationships with vendors and it's likely that the trend will be in this direction. Omni is a BI tool that does a lot more than the average BI tool - check it out!

This weekend, I got to thinking about how traditional RAG frameworks often don't quite hit the mark when it comes to pulling up the most relevant content from big data sets. So, I rolled up my sleeves and put together my own new framework for RAG: the Automatic Metadata-based Retriever-Augmented Generator (AutoMeta RAG) framework. Automatic Metadata based Retriever-Augmented Generator (AutoMeta RAG) framework is designed to optimize the retrieval of data chunks from a large dataset using LlamaIndex and by leveraging dynamic metadata schemas. The framework initiates by analyzing the dataset and anticipated user queries to suggest two types of metadata schemas: file-level and chunk-level. File-level metadata remains constant across all data chunks within a single file, providing a macro-view of the data attributes. Conversely, chunk-level metadata is unique for each data chunk, allowing for fine-grained indexing and retrieval. Following the schema suggestion, the framework automates the extraction of metadata for each data chunk according to the defined schemas and stores these, along with the data vectors, into the Qdrant VectorDB since Qdrant supports metadata based filtering which is one of the many reasons I love Qdrant. On the inference side, the framework extracts metadata from user queries based on the pre-determined schemas. It utilizes this metadata to perform targeted search in QdrantDB with filtering, ensuring that the retrieved data chunks are the most relevant to the user's request. Although evaluation and benchmarking is yet in progress, this method would significantly improve the precision and speed of data retrieval by reducing the search space. Read More: https://lnkd.in/dQD7cuHK Github: https://lnkd.in/d-sQS95n

dbt Labs' latest version of the semantic layer offers exciting possibilities in data-driven decision-making. The semantic layer connects data platforms with business intelligence (BI) tools, providing a single, verified source of truth for data. Empowering businesses with universally defined and accessible metrics and concepts, it will help break down data silos, improve data consistency, and simplify analytics. Here’s more. https://bit.ly/471GqK2 #dbtSemanticLayer #DataAnalytics #BusinessIntelligence #TrustedData

Understanding the Right Knowledge Graph for Your Data / RAG Needs When choosing a graph structure, consider two key factors: A. Is your data dynamic? B. Is your schema dynamic? Fixed Data/ Fixed Schema: Perfect for static data like document hierarchies. Build once, use forever! Dynamic Data/ Fixed Schema: Ideal for continuously changing data, like patient records. Set a schema, let data flow in! Fixed Data/ Dynamic Schema: Great for evolving relationships in static text. Use LLMs to iteratively build schemas. Dynamic Data/ Dynamic Schema: For complex, ever-changing data. Requires robust graph management tools. At WhyHow.AI, we focus on creating tools that help developers manage their RAG pipelines with precision. Interested? Contact us at team@whyhow.ai, find our calendly / follow our newsletter at WhyHow.AI, or join our Discord for more insights! https://lnkd.in/eJNtvfAX

𝐃𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧𝐚𝐥 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠: 𝐒𝐭𝐢𝐥𝐥 𝐭𝐡𝐞 𝐂𝐨𝐨𝐥 𝐊𝐢𝐝 𝐨𝐫 𝐘𝐞𝐬𝐭𝐞𝐫𝐝𝐚𝐲'𝐬 𝐍𝐞𝐰𝐬? In the realm of data architecture, the data lakehouse has surged in popularity by merging the advantages of data lakes and data warehouses. As more organizations adopt this model, a common question arises: Is traditional dimensional modeling still preferable? The answer, it turns out, hinges on balancing use case enablement and query patterns with best practices derived from past experiences. 𝐓𝐡𝐞 𝐑𝐨𝐥𝐞 𝐨𝐟 𝐃𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧𝐚𝐥 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠: Dimensional modeling, known for its star and snowflake schemas, has been a staple in data warehousing due to its simplicity and efficiency in handling analytical queries. This model is particularly effective for business intelligence (BI) tools and reporting. However, in a data lakehouse environment, the varied range of use cases means that no single modeling approach will fit all scenarios. 𝐔𝐬𝐞 𝐂𝐚𝐬𝐞-𝐃𝐫𝐢𝐯𝐞𝐧 𝐃𝐞𝐜𝐢𝐬𝐢𝐨𝐧𝐬: When determining whether to use dimensional modeling in a data lakehouse, it’s essential to consider specific use cases and query patterns: ✅ BI and Reporting: If the primary use case involves traditional BI, with users generating reports and dashboards with predictable query patterns, dimensional modeling remains highly beneficial. ✅ Data Exploration and Machine Learning: For use cases that involve ad-hoc data exploration, data science, or machine learning, a more flexible, schema-on-read approach may be preferable. In practice, a hybrid approach often works best. For instance, maintaining a dimensional model for well-defined, high-frequency analytical queries while adopting flexible design patterns for special use cases. Ultimately, if the database doesn’t support 80-90% of use cases and query patterns, the data model is flawed, even if it adheres to all best practices.

It’s time for a thoughtful rebundling of the data stack. The modern data stack broke apart the legacy monolithic BI stack. Best-in-class point solutions were pieced together. At first, it was ETL (Fivetran or Stitch), data warehouse (Redshift, then Snowflake or BigQuery), and business intelligence (Looker, Mode or Periscope). Then we added orchestration, dbt, data catalogs, data observability, and reverse ETL not to mention adjacent workflows like ML / AI. Analytics alone has been broken apart into SQL tools, visual discovery tools, governed reporting, customer-facing, spreadsheets, notebooks, and augmented analytics with AI. There is value in having a point solution that is materially better for valuable use cases. But increasingly, I’ve spoken to more and more data leaders who are questioning whether the value is worth the cost. The cost in complexity, in integration, in siloing, in overlapping licenses. I recently spoke with a data team that has a dedicated procurement person to manage the 15+ relationships they have with vendors. 😬 With analytics, there is value in bringing complimentary use cases together. SQL is great for exploration for analysts, spreadsheets are great for operations. Reporting should be standardized and managed. The same model you use to power internal reporting should be leveraged to power customer-embedded reporting. Omni is rebundling the BI stack.