Apache Sedona

You can use Apache Spark and Apache Sedona to cluster points with the DBSCAN algorithm. DBSCAN stands for Density-Based Spatial Clustering of Applications with Noise and Sedona uses this algorithm to cluster geometries in a DataFrame. The following example shows how to cluster points in a Spark DataFrame. Outliers are assigned to cluster -1.

🚀 From Finding Cafés to Processing Satellite Imagery: My Journey with Apache Sedona 🌍 When I first started using Apache Sedona for geospatial data processing, I had no experience with PostGIS or large-scale spatial computations. Fast forward, and I found myself processing satellite imagery, handling raster-vector intersections, and even contributing to Apache Sedona’s API improvements! 🎯 This journey wasn’t just about learning a new tool—it was about solving real-world geospatial challenges and optimizing workflows for massive datasets. 📖 In this article, I cover: ➡️ Setting up Apache Sedona for large-scale geospatial processing ➡️ Handling raster and vector data efficiently with Spark ➡️ Fixing real-world geospatial computation issues (invalid intersections, clipping errors) ➡️ Raising an official feature request in Apache Sedona! 🎉 Whether you're a data engineer, GIS enthusiast, or someone diving into big data geospatial analysis, this guide will give you practical insights into using Apache Sedona for scalable geospatial computations. 🔗 Check it out here: https://lnkd.in/gijYiZSw #GeospatialData #BigData #ApacheSedona #DataEngineering #GIS #Spark #CloudComputing Apache Sedona The Apache Software Foundation Matt Johnson, Bram Desoete, Hardeep Arora, Dede T., Prateek Dubey

I’ve put together a tutorial for R users who occasionally work with large geospatial datasets. In this tutorial, I demonstrate how Apache Spark, Apache Sedona (developed by Wherobots), and Delta Lake can significantly speed up your analysis. Apache Spark is a powerful distributed computing engine, while Apache Sedona extends Spark with spatial capabilities, enabling efficient large-scale geospatial processing. Delta Lake, on the other hand, provides a robust storage system that ensures reliability and performance when handling big data. For this tutorial, I use the New York City cab dataset, which is commonly used to predict trip durations. I show how you can use pickup and drop-off coordinates to determine the neighbourhoods where trips start and end, using a shapefile from NYC Open Data. I then use these neighbourhoods to link median household income data to each pickup and drop-off location. Additionally, I demonstrate how to obtain population density data at each location using raster data from WorldPop. Lastly, I show how to retrieve Local Climate Zones (LCZ) for pickup and drop-off locations using the World Urban Database and Access Portal Tools (WUDAPT). You can find the tutorial here: https://lnkd.in/gVewxhej. Hope you find it useful!

You can create an Apache Iceberg table with a geometry column using Sedona and Spark. Just create an empty table and then append a DataFrame with the format set to Iceberg. Iceberg provides several advantages vs. data lakes like reliable transactions, versioned data, time travel, schema enforcement, and DML operations. More posts on these features coming soon!

🚀 Apache Sedona 1.7.1 is out! 🚀 We’re excited to announce the release of Apache Sedona 1.7.1, featuring: ✅ SQL interface for GeoStats (ST_DBSCAN, ST_GLocal, ST_LocalOutlierFactor) ✅ Broadcast join support for distributed KNN Join ✅ STAC catalog & OpenStreetMap (OSM) PBF reader ✅ New ST functions like ST_RemoveRepeatedPoints This minor release includes new features, improvements, and bug fixes with no breaking changes. 📖 Release notes: https://lnkd.in/d9ai93Ph #ApacheSedona #Geospatial #BigData #OpenSource

We know that handling large-scale spatial data can be daunting 😔, which is why we’ve teamed up with O'Reilly to bring you this comprehensive guide, designed to simplify geospatial data. This content will help boost your spatial analytics expertise and transform the way you work with geospatial data! 💪 🆕 Our newest chapter, focusing on vector data analysis using spatial SQL, is now available. Check it out here, and don’t forget to revisit the earlier chapters: https://bit.ly/4gkm4AU If you've already accessed the previous chapters, be sure to check your inbox for the latest one! 📧

We know that handling large-scale spatial data can be daunting 😔, which is why we’ve teamed up with O'Reilly to bring you this comprehensive guide, designed to simplify geospatial data. This content will help boost your spatial analytics expertise and transform the way you work with geospatial data! 💪 🆕 Our newest chapter, focusing on vector data analysis using spatial SQL, is now available. Check it out here, and don’t forget to revisit the earlier chapters: https://bit.ly/4gkm4AU If you've already accessed the previous chapters, be sure to check your inbox for the latest one! 📧

Introducing the new Sedona STAC reader feature! This innovative addition addresses the typical hurdles associated with integrating STAC datasets. By streamlining data ingestion processes, it enhances analytical workflows for smoother operations. #sedona #STAC #GIS #geospatial #wherobots

Happy Monday! Just a friendly reminder about the events we have going on this week: 🌎 TOMORROW 3/4: Join us for our monthly community office hour, where we'll share the latest news and updates for Apache Sedona. Bring your questions if you have any, as we'd love to hear about the cool projects you're working on! https://bit.ly/3WCyagO 📈 WEDNESDAY 3/5: Learn how companies like Comcast are using Apache Sedona to optimize their ETL pipelines and how it outperforms other tools like GeoPandas and PostGIS to boost productivity. https://lnkd.in/g2HG7FvV 📺 These sessions will be recorded, so make sure to RSVP, and we'll send you the recording if you're unable to attend. See you there!