dltHub’s Post

🔧 Wrapped up the second session of the dltHub August 2024 Workshop today. In today’s session, we covered: •Custom Incremental Loading: Efficiently updating data. •Schema Configuration: Managing data types and contracts. •Tracing & Logging: Debugging with Sentry. •Performance Optimization: Speeding up data processing. •Custom Sources: Integrating external APIs. •Reverse ETL: Pushing data back into systems. •Deployment: Using tools like Lambda, Dagster and Airflow. Thanks to the dltHub team for the practical insights. Check out dlt if you work with data pipelines. [https://lnkd.in/dQa4RBaG]

In addition to the Model Runtime we've already introduced (a unified interface for model connection). 🔜 These are some awesome standalone module coming soon: RAG Pipeline Modularization: This involves managing various aspects of RAG, such as ETL, embedding, index building, data recall, etc., offering developers considerable customization options. Tools: Dify comes loaded with built-in tools, and it also supports integration of additional tools that comply with the OpenAPI Specification API standard. Workflow: Dify focused on providing customizable workflow configurations, tailor-made for specific industries and scenarios. Interested in learning more? 👉🏻 Check out our blog: https://lnkd.in/gfZH-7bd

🚀 Level Up Your Data Ingestion Process With An ETL pipeline For GraphRAG Applications! 🚀 In my latest Medium article, I explored the transition from a basic manual data ingestion pipeline to a fully automated, scalable solution. While our previous approach to feeding data into a Neo4j database was a great starting point, it's time to take things up a notch! ETL pipeline for data ingestion into Neo4j database using Docker, Inc and docker-compose In this new guide, I break down how to build a robust ETL pipeline tailored for GraphRAG applications. We'll be diving into Docker and Docker Compose to streamline your data processing, enhance scalability, and boost overall efficiency. Ready to transform your data workflow? Check out the article and let's build this essential piece together! 💡 #DataEngineering #ETL #Neo4j #Docker #GraphRAG #TechInnovation #MediumPost #CodeWithPrince

Ready to elevate your skills in handling JSON data? Join us for a session on leveraging TiDB for storing, retrieving, and manipulating JSON data! 🚀 Discover how TiDB’s unparalleled scalability makes it an ideal JSON storage solution and learn to use SQL for dynamic reporting. Find out why TiDB is a game-changer compared to traditional NoSQL solutions. Don’t miss this chance to revolutionize your approach to JSON data management! 📈🔧 👉 Sign Up Today and unlock the full potential of TiDB!

Explore Structured outputs for LLM-powered ETL, RAG, and agent pipelines! You can now use #llama_index for structured extraction with any LLM setup, with full support for async and streaming. Here's what you need: 1. Define a Pydantic object. 2. Attach it to your LLM using `as_structured_llm(…)`. This setup lets you perform structured extraction seamlessly, whether you’re working with chunk-level or document-level RAG. You can see your structured JSON being filled in real-time, which is incredibly useful for handling large JSON outputs efficiently. Cookbook :https://lnkd.in/gBFvdnTf Docs :https://lnkd.in/g6z5HtaW

Yesterday, we launched our new No-Code CI integration for *all* data teams to run data diffs during their pull request process. We built this because we want to empower all teams to automate their data testing — not just those who use dbt. Watch Nick Carchedi demo how it works below 👇 https://lnkd.in/g8PmUwPE

I believe using Ibis Data instead of pandas or polars can help avoid debates over which dataframe engine to use. Ibis provides a unified interface that allows engineers and data scientists to work with the same codebase, while easily switching engines as needed. For example, you can develop with Pandas and switch to Polars in production without any code changes. The main missing feature is support for rlang. If Ibis adds support for the R-lang dataframes, it could potentially cover the entire dataframe ecosystem.

Oops 🐒 we dropped a new product! 📯 📯 Datafold's No Code CI makes it easy to automatically compare data between development vs production environments (or any two data environments) and catch data errors before merging code changes. We built this so teams can add data diffing to their testing arsenal using their existing data infrastructure, without writing a line of code. I genuinely think this is a delightful new workflow--you will see in the video. This tool, and Datafold in general, has absolutely no dependency on dbt. 🙌 All we require is connectivity to your data. Blog 🖋 : https://lnkd.in/gXwy4xmY Vid 📽 (feat. Nick): https://lnkd.in/ge9yYmWm P.S. A data diff is like a git diff, but for data. Let me know if you want to try this out!

Introducing a dedicated dbt job type within Civis Platform, making it easier than ever to integrate #dbt into your existing workflows! Now you can streamline your data transformations, automate processes, and collaborate effortlessly—all within a single platform. Read our latest blog post to learn more! #DataScience

Love querying the data using GraphQL from the Propel API Playground Saves me so much time. I get: ◆ GraphQL schema autocompletion ◆ Code examples ◆ Access token generator