Green Software Foundation’s Post

Great to see, how the measurement of emissions in serverless computing is developing. Thanks, Franziska Warncke and Denis Angeletta, for your pioneering work and this interesting article!

Great timing on this new Amazon Web Services (AWS) launch given recent unpredictable #weather events such as the rains in #Dubai - Read this link to see the 2023 list of #ExtremeWeather events https://lnkd.in/edp_s8tF 👍 #ClimateChange #Sustainability AWS for Industries #WeatherPrediction

Numerical Weather Prediction has been one of the standard workloads of #hpc for decades. And my Colorado neighbors over at NSF NCAR - The National Center for Atmospheric Research have been key to advancing research in this area. One of their most popular developments in support of these efforts is the open source Weather Research and Forecasting (WRF) model, with > 50k users from over 160 countries. See below for detailed instructions on how you can perform your own #weather prediction on Amazon Web Services (AWS) using AWS ParallelCluster, Amazon FSx for Lustre, and WRF.

Heading to AWS re:Invent? Don’t miss our talk on building intelligent real-time systems with InfluxDB! 🗓️ Thursday, December 5 🕜 1:30 PM 📍 Venetian | Hall B | Expo | Theater 4 We'll explore the world of time series data and how InfluxDB powers intelligent real-time systems. We'll cover the essentials of time series databases and showcase how InfluxDB tackles complex data challenges with ease. 🔗 https://bit.ly/4eZEmpE #InfluxDB #AWSreInvent #TimeSeries

Last week at AWS re:Invent 2024, I had fascinating conversations with engineering and data leaders about their real-time data integration challenges. These discussions echoed major themes from AWS’s keynotes—like challenges moving inference pipelines into production, addressing the complexities of coordinating LLM-powered agents, and simplifying architectures by hiding complexity behind foundational primitives. In my latest blog, I dive into my takeaways from the event. One highlight is the need to go beyond vector search by incorporating real-time structured data into RAG pipelines, unlocking the full potential of large language models. https://bit.ly/4gsL8om

🚀 Upcoming Session at AWS Cloud Weekend: "Building Data Pipelines in a Serverless Way" with Vesko Vujovic! 🚀 🗓️ Join us for a deep dive into serverless data engineering this May 11th at the Faculty of Electrical Engineering, Podgorica! 🔧 This session dives into serverless data engineering, exploring how to create, orchestrate, and transform data without managing servers and using step functions as orchestrators. We'll showcase the power of this approach by building a scalable data pipeline specifically for handling the Do Not Call numbers registry for various countries. This practical example will demonstrate how serverless architectures can streamline data processing for various use cases, allowing you to focus on data insights and not infrastructure. 📈 🔗 Learn more and secure your spot: https://cloudweekend.me/ ✨ Don’t miss out on this transformative session—see you there!

Dataspaces in Systems Engineering - having worked in software and systems engineering for a long time, I have developed a special interest in dataspaces. During the years, improving specific tools with custom functionality, integrating the tools within one company, building fusioned systems engineering repositories and supporting the exchange of data across the boundaries of companies, I am intrigued by the possible solutions that dataspaces bring to the table. So in COOPERANTS, itemis is looking especially in the application of Gaia-X Association for Data and Cloud (AISBL) in the system engineering lifecycle for several companies. Below is a sketch of the use cases of systems engineering interaction between companies in terms of data exchange and (AI) service access we are considering.

Thrilled that mine and Qian Li's paper was just published in the Communications of the ACM! The big challenge we've been working on since grad school is how to make serverless computing work well for stateful applications. This is tricky because it's hard to make long-lived stateful workflows reliable in ephemeral serverless execution environments. The solution, which we talk about in this paper and are building at DBOS, Inc., is using database transactions to make programs durable. The key idea is to store the execution state of a serverless workflow in a database. That way, if the workflow is interrupted, it can automatically recover from where it left off by looking up its execution state in the database. If you know that stateful programs are durable, it becomes a lot easier to make them serverless. You can deploy them to ephemeral executors and leverage their durability to autoscale them and move them between executors as needed. That way, you can build incredibly reliable stateful programs that also benefit from the scalability and convenience of serverless.

Excited to announce the availability of logs transformation and enrichment in CWL. 🚀 Customers can enhance their analytical experience by adding structure to their logs using pre-configured templates for common AWS services or build custom transformers with Grok. Customers can also enrich their logs with additional metadata 🚀. Customers can leverage transformed logs with support for field indexes, discovered fields in CWL-I, alarming using metric filters and forwarding via subscription filters.  🚀 Thanks to all who worked on building this feature ! 🚀 Learn more: https://lnkd.in/gHs2r_jv

Last week at #AWSreInvent, we had fascinating conversations with engineering and data leaders about real-time data integration challenges. These discussions aligned with key themes from AWS's keynotes—like moving inference pipelines into production, coordinating LLM-powered agents, and simplifying architectures with foundational primitives. In our latest blog, by Nate Stewart he shares takeaways from the event, including a key insight: the need to go beyond vector search by incorporating real-time structured data into RAG pipelines. This approach unlocks the full potential of large language models and drives better outcomes. Read more about our perspective and how real-time data is shaping the future of AI and data engineering. https://bit.ly/4f8JIi2