Apache Sedona

Sedona makes it easy to compute the distance between two points. Just use the ST_Distance function to compute the distance between points in a Cartesian plane. Stay tuned for posts on other functions Sedona has to compute the distance between latitude/longitude coordinate that factor in the curvature of the earth!

Sedona recently introduced a Sedona GeoPandas API that will let you write code with GeoPandas syntax. This will be a great alternative for GeoPandas developers who want to take advantage of the great performance/scalability of Sedona without learning Sedona syntax.

NEXT WEEK: The Apache Sedona office hour is coming up soon! We just released Sedona 1.7.1, and there are plenty of exciting new features to cover this time around! 😎 ✅ SQL interface for GeoStats (ST_DBSCAN, ST_GLocal, ST_LocalOutlierFactor) ✅ Broadcast join support for distributed KNN Join ✅ STAC catalog & OpenStreetMap (OSM) PBF reader ✅ New ST functions like ST_RemoveRepeatedPoints Mark your calendar for Tuesday, April 1st to tune in. You won’t want to miss this! https://bit.ly/3UBmxFY ✉️ P.S. If you can't make it live, no worries! Just sign up and you'll get the recordings delivered to your inbox to watch when you can.

Iceberg geometry support makes it easy to perform spatial delete operations. For example, you can delete any linestrings that cross a given polygon. Iceberg is a Lakehouse storage system (aka "open table format"), so the delete operation just rewrites the impacted files. This is more efficient than delete operations on data lakes. See the following code snippet for an example delete operation with Apache Sedona and Apache Iceberg:

You can use Apache Spark and Apache Sedona to cluster points with the DBSCAN algorithm. DBSCAN stands for Density-Based Spatial Clustering of Applications with Noise and Sedona uses this algorithm to cluster geometries in a DataFrame. The following example shows how to cluster points in a Spark DataFrame. Outliers are assigned to cluster -1.

🚀 From Finding Cafés to Processing Satellite Imagery: My Journey with Apache Sedona 🌍 When I first started using Apache Sedona for geospatial data processing, I had no experience with PostGIS or large-scale spatial computations. Fast forward, and I found myself processing satellite imagery, handling raster-vector intersections, and even contributing to Apache Sedona’s API improvements! 🎯 This journey wasn’t just about learning a new tool—it was about solving real-world geospatial challenges and optimizing workflows for massive datasets. 📖 In this article, I cover: ➡️ Setting up Apache Sedona for large-scale geospatial processing ➡️ Handling raster and vector data efficiently with Spark ➡️ Fixing real-world geospatial computation issues (invalid intersections, clipping errors) ➡️ Raising an official feature request in Apache Sedona! 🎉 Whether you're a data engineer, GIS enthusiast, or someone diving into big data geospatial analysis, this guide will give you practical insights into using Apache Sedona for scalable geospatial computations. 🔗 Check it out here: https://lnkd.in/gijYiZSw #GeospatialData #BigData #ApacheSedona #DataEngineering #GIS #Spark #CloudComputing Apache Sedona The Apache Software Foundation Matt Johnson, Bram Desoete, Hardeep Arora, Dede T., Prateek Dubey

I’ve put together a tutorial for R users who occasionally work with large geospatial datasets. In this tutorial, I demonstrate how Apache Spark, Apache Sedona (developed by Wherobots), and Delta Lake can significantly speed up your analysis. Apache Spark is a powerful distributed computing engine, while Apache Sedona extends Spark with spatial capabilities, enabling efficient large-scale geospatial processing. Delta Lake, on the other hand, provides a robust storage system that ensures reliability and performance when handling big data. For this tutorial, I use the New York City cab dataset, which is commonly used to predict trip durations. I show how you can use pickup and drop-off coordinates to determine the neighbourhoods where trips start and end, using a shapefile from NYC Open Data. I then use these neighbourhoods to link median household income data to each pickup and drop-off location. Additionally, I demonstrate how to obtain population density data at each location using raster data from WorldPop. Lastly, I show how to retrieve Local Climate Zones (LCZ) for pickup and drop-off locations using the World Urban Database and Access Portal Tools (WUDAPT). You can find the tutorial here: https://lnkd.in/gVewxhej. Hope you find it useful!

You can create an Apache Iceberg table with a geometry column using Sedona and Spark. Just create an empty table and then append a DataFrame with the format set to Iceberg. Iceberg provides several advantages vs. data lakes like reliable transactions, versioned data, time travel, schema enforcement, and DML operations. More posts on these features coming soon!

🚀 Apache Sedona 1.7.1 is out! 🚀 We’re excited to announce the release of Apache Sedona 1.7.1, featuring: ✅ SQL interface for GeoStats (ST_DBSCAN, ST_GLocal, ST_LocalOutlierFactor) ✅ Broadcast join support for distributed KNN Join ✅ STAC catalog & OpenStreetMap (OSM) PBF reader ✅ New ST functions like ST_RemoveRepeatedPoints This minor release includes new features, improvements, and bug fixes with no breaking changes. 📖 Release notes: https://lnkd.in/d9ai93Ph #ApacheSedona #Geospatial #BigData #OpenSource

We know that handling large-scale spatial data can be daunting 😔, which is why we’ve teamed up with O'Reilly to bring you this comprehensive guide, designed to simplify geospatial data. This content will help boost your spatial analytics expertise and transform the way you work with geospatial data! 💪 🆕 Our newest chapter, focusing on vector data analysis using spatial SQL, is now available. Check it out here, and don’t forget to revisit the earlier chapters: https://bit.ly/4gkm4AU If you've already accessed the previous chapters, be sure to check your inbox for the latest one! 📧