Robert Hodges’ Post

Terraform Weekly newsletter #159 (sponsored by CAST AI) Reduce K8s costs by 60%+ with CAST AI! Lightning-fast autoscaling, spot VMs, and many more. Optimize your first cluster for FREE. https://lnkd.in/e6gisPkF * Deploying Super Mario on #Kubernetes by Ajay Kumar Yegireddi * Deploying a Containerized App to ECS #Fargate Using a Private ECR Repo & Terragrunt by Stéphane Noutsa * Introducing mlinfra — a hassle-free way to deploy ML Infrastructure by Ali Abbas Jaffri * AWS Lambda #Serverless Processor: Rust, Fargate by Darryl R. * Building a dynamic AWS security group solution with CSV in Terraform by Anthony Wat Open-source projects: * dishavirk/GitOps-with-EKS-ArgoCD-TF - An advanced GitOps workflow using GitHub Actions, ArgoCD, Terraform, and AWS EKS for a Python Flask application that converts temperatures from Celsius to Fahrenheit. Subscribe - https://lnkd.in/deZHPqs5 Read online - https://lnkd.in/dgdQ_WEn

*Breathes out* Last week was all about Mage and Docker, Inc. In the second week of the Data Engineering Zoomcamp by Alexey Grigorev, I learned a whole lot about Mage: from setting up with Docker, connecting to a plethora of data sources, building blocks for workflows, and scheduling them. Prior to last week, I never thought I'd ever choose any orchestration tool over Airflow. But boy, Mage is surely giving it a run for its money. My favourite feature is the concept of reusable blocks. It saves time and energy for data engineers by allowing the same task/block to be used across multiple pipelines. So, If a parameter in a major transformation step changes, you'd only need to edit the shared block, and all dependent pipelines will be updated automatically. Anyway, so over the weekend, I developed an ETL (Extract, Transform, Load) pipeline that extracts data from various API sources, harmonizes them, applies defined transformation steps, before loading it into a PostgreSQL database and a Google Cloud (GCP) Bucket, for downstream operations. And, all of this was done on Mage! If this is something you're interested in, feel free to read more about it here https://lnkd.in/dy_Mv47S #dataengineering DataTalksClub #mage #docker

💡 At LoopStudio, we believe that efficient data processing architectures are the backbone of any successful ML solution. Recently, we implemented a powerful system that integrates Snowflake, DBT, Apache Airflow, and AWS services like SQS, Lambda, and Aurora to solve complex data challenges for one of our clients. Here’s a sneak peek of what we covered: 1) Data Collection: High-quality data is the foundation for any ML project. 2) Data Processing: Efficiently transforming raw data into valuable insights. 3) Data Storage: Choosing the right solution to ensure scalability and performance. 4) Machine Learning in Action: Leveraging data to drive innovation and business decisions. Want to know more? Check out the full breakdown in our latest blog post and learn how we build scalable, resilient architectures that push the boundaries of what's possible ➡️ https://lnkd.in/drM2R86W

What do you get from a Sync demo? When you request a demo with our team, you get a first hand look at how to gain complete control over your #Databricks ecosystem — all while freeing up dev time with automated infrastructure and cost management. Request a demo here: https://hubs.ly/Q02qDLgR0 #dataengineering #databricksworkspace

🚀 Introducing our first infographic in a series on tools used in data engineering! Today, we're diving into a simple but fundamental building block for engineering, the game-changing containerization platform - Docker, Inc 🐳 Discover how Docker simplifies application deployment, enhances portability, and boosts efficiency. Perfect for your CI/CD pipelines and microservices! Stay tuned for more insights on top data engineering tools. #DataEngineering #Docker #TechTools #Containers #techinfograph

Join me tomorrow to see what we've learned building a platform for big and fast data on Kubernetes. #opensource #realtime #analytics #kubernetes #bigdata

DataOps: Build AWS Pipelines in 2024 | Basics of DataOps https://lnkd.in/gRSi8GmV

CICD on Microsoft Fabric ==================== Don't hesitate to leverage deployments pipelines - Game changer for me as unified and low code compliant cicd system Most of others data systems needs Terraform or Azure devops (yaml) No worries for datacitizens to use them (dep pipelines) ✔️Dev Workspace is synchronized with Git(hub) sources control a main branch <=== Hoping soon for WS switch alongside branches (main / features) to avoid annoying STATIC relationships (something dynamic like on Azure datafactory console) ✔️Push items alongside WS environments (pilot with APIs if you want) ✔️Rules to change values (connections names and cicd parameters ( <== hoping for all artifacts rules support as soon as possible) ✔️Delta codes control ( before / after by lines) <=== A wish for some items : stop random lines change for my text codes Sometime all my lines are on red color , just due to a small update <=== Hoping for one API to perform this check ✔️ Deployments History 🚒 Hoping for Dataflowgen2 a soon support (with cicd parameters on this one) 🚒 Hoping soon for an integrated devops best practices ( pbi semantic model or report / LH / WH / dfgen2 / pipeline / notebook) controls service <== game changer feature #powerbi #microsoftfabric #azure

Leverage Redis for processing Workato Recipe understanding and analysis of hardcoded values in recipes using Workato copilot resource. https://lnkd.in/gFV5_Ayr https://lnkd.in/gtkdQk-p https://lnkd.in/gkeHR2aB #productivityhack #workato #automation #integration #genaiusecase

🚀 Day 36 of #90daysofdevopschallenge: Managing Persistent Volumes in Your Deployment! 💥 Excited to delve into the intricacies of Persistent Volumes in Kubernetes and optimize storage utilization for enhanced application performance. Let's harness the power of Persistent Volumes to ensure seamless data storage and retrieval within our deployments. 🙌🔥 #devops #learningandgrowing #90daysofdevops #90daysofdevopschallenge #trainwithshubham #LearningIsFun #k8s #Kubernetes #LearningJourney #configmap #secrets #pvc #pv #persistentsvolume #persistantvolumesclaim 🛠️📂