wildflow

wildflow coral -- our first product, a comprehensive digital twin of coral reef ecosystems based on multimodal foundation models for biodiversity. 🔵 Why? Billions of years of evolution created Earth's vibrant ecosystems -- they are inconceivably rich and incredibly complex! We have so much to learn from this beautiful complexity. We want to build a future where humans and nature thrive and enrich each other. That’s why we chose to start with coral reefs. They are the most complex, beautiful, endangered, and important ecosystem. They are home to 25% of marine life while occupying only 0.1% of the ocean. Sadly, we’re on track to lose 90% of coral reefs by 2050. The time to act is now. Over half a billion people rely on coral reefs. Already vulnerable communities are the first to suffer the consequences. We have to make a lot of critical decisions about them! They are a key to model entire biosphere (protect nature) down the road. 🔵 How? We take all the data across all modalities, such as 3D photogrammetry, bioacoustics, underwater videos, remote sensing, eDNA, environmental data like currents, and more, to create the ultimate digital twin of any coral reef ecosystem and make it available to the world. And do it at a planetary scale. Core tech is multimodal foundation models for biodiversity. 🔵 What? This enables deep modelling of complex ecosystem dynamics, such as population dynamics, predator-prey dynamics, energy transfer, phenological events (like spawning), and computes the ecosystem's health and resilience metrics. Via rigorous analysis of multimodal data, it uncovers precise mechanisms driving ecosystem change, offering humans irrefutable evidence to steer their actions. It shows quantitatively how coastal development, agriculture runoff, invasive species, pollution, increasing water temperatures, and other pressures affect the coral reef ecosystem. It shows what the coral reef ecosystem gives us back through its services, such as coastal area protection and oxygen production. It coordinates conservation and restoration efforts worldwide. We know which practises work and which don’t. It guides human activities to understand, protect and restore coral reef ecosystems. 🔵 Next steps: ↳ ✅ 3D "Street View" for coral reefs prototype [DONE] ↳ 3D digital twin of coral reefs ↳ multimodal twin of coral reefs ↳ model ocean ecosystems ↳ model biosphere ↳ become the first generation that actually leaves behind nature better than we found it! 🐳 🌎

🌊 Coral Reefs are some of the most critical ecosystems in our oceans, yet they're disappearing at an alarming rate. I spoke to wildflow CEO, Sergei Nozdrenkov about their use of Radiance Fields to create a "3D Street View" for Coral Reefs. This represents the beginning of both visualization, but also larger deep (no pun intended) learning about our oceans and the environments that drive them. And yes, Radiance Fields do work underwater. Join me as we continue to examine the real world applications of Radiance Field based technologies and the people building with them. Full article here 👇 https://lnkd.in/dU_XJqUm

Sneak peek into the new larger 3D "Street View for Coral Reefs" model: - ❌ geometry is not good - ❌ no cleanup of wacky splats - ✅ used 200 photos from 2 GoPros (almost one swim-through) - ✅ 5 x 7 sq metres --> 15 x 3 sq metres (link in the comment)

A way to protect/restore Earth's vibrant Coral Reefs, Amazon Forest.. Many years of building AI for biodiversity at Google X and wildflow led me to this: we need a Planetary Digital Nervous System to survive and thrive. This is a cybernetics approach to conservation. 🔵 Why? Life on Earth has been evolving for billions of years, it's so precious, beautiful and incredibly complex! We losing nature, but there's a way to become the first generation that actually leaves behind nature better than we found it! 🌎 🔵 How? We need a Planetary Digital Nervous System that connects all sensors measuring the biosphere, such as eDNA, underwater cameras, bioacoustic microphones, remote sensing... (and environmental data like currents, temperatures...), to real-world actions like restoring coral reefs, creating marine protected areas, preventing algal blooms, managing invasive species, precise fishing policies, etc. 🔵 Cybernetic loops Measure -> Model -> Act are everywhere. 🔵 Pest Management in Ag: ↳ Measure: where all insects are using pheromone traps, acoustic traps, drones... ↳ Model: population dynamics, predict outbreak ↳ Act: targeted spray, biocontrol, mating disruption 🔵 Marine Protected Areas: ↳ Measure: BRUVs, satellite/acoustic tags, ↳ Model: find cluster of the most important fish, ecological niche ↳ Act: create and enforce MPA polygon 🔵 Algal Bloom: ↳ Measure: eDNA, remote sensing ↳ Model: forecast population dynamics of algae ↳ Act: remove nutrients from the water, etc 🔵 When you touch something hot and instantly snatch your hand away, you also follow Measure (touch smth) -> Model (realise it's super hot) -> Act (snatch your hand away). For you it takes milliseconds. For marine protected areas it takes at leat 5 years! 🤯 🔵 Measure 📈✅: a lot of incredible companies collecting the data: NatureMetrics, Flying Fish Technologies Pty Ltd, Saildrone, Stream Ocean, Rainforest Connection (RFCx) and many more. Nature data grows exponentially (supply)! 🔵 Act 📈✅: massive demand here. How do you find data for nearby park? It's all fragmented. 🔵 Model ❌: huge bottleneck in data aggregation and modelling natural ecosystems. We need data platforms like Cecil, federated data networks, and companies and decision engines like Nala Earth. Majority of Measure orgs also creating platforms and data consortiums. Even NOAA: National Oceanic & Atmospheric Administration at Oceans Decade complaining it's hard to justify data collection, we need intermediaries. Really hard to bridge this gap! 🔵 wildflow was founded to comprehensively understand nature and empower humans to make better decisions about protecting our ecosystems. Planetary Digital Nervous System is our NorthStar. We start with multimodal foundation models for Coral Reefs (comment below). 🔵 We can protect and restore nature! We need more passionate people to build this ultimate digital twin of natural ecosystems to bridge the gap! 🌎 We can do it together!! 🙌

Try our 3D "Street View for Coral Reefs" prototype (3D Gaussian splatting)! We've hit a new milestone today! A small step towards our mission after months of work.. So stoked to tell you \o/ 🔵 Why? Coral Reefs are so precious, beautiful, incredibly complex and threatened ecosystems. They are dying fast.. But there's a way to protect and restore them! Also they are a key to protecting other ecosystems. A lot of amazing people fighting for their life. We need to understand deeply how Coral Reefs function, what methods work, what don't, to coordinate precise action. That's why we're building a digital twin of coral reef ecosystems -- our first product. 🔵 https://lnkd.in/gekBPU_c -- today you can try our first scrappy prototype (works best on computer), 10% done. 🔵 What you see: it's a "Google Maps / Street View / Earth for Coral Reefs". Someone swam around a reef with a few GoPros, and you can now see high-res 3D model of coral reef in the browser! It's a super cheap way of monitoring. Soon you will fly over square kilometres of reefs and see them in centimetre resolution. You can compare reefs in different countries and see how they evolve over time, easy way of getting rid of biases in data. 🔵 The prototype is terrible. Still work in progress (10%), just a sneak peek. E.g. we didn't even clean up the model at all. Used only 70 images out of 8000 for this reef. You see only 5x7m area. 🔵 Yet, I don't think you can find a better resolution reefs, so that you can just go and see them in the browser. Especially without using "beefy" computers on backend. In our case all the rendering is happening on the client side, we don't even have any backend servers. It fetches static data directly from Google Cloud buckets. Super easy to scale! 🔵 Next steps: more data, larger scenes, 3D segmentation models, classifying coral species, adding other modalities (acoustics, eDNA, geospatial)... Allowing anyone to run analytics against all our data... Foundation models for biodiversity... Becoming the first generation that actually leaves behind nature better than we found it! 🪸 🔵 This prototype wouldn't happen without these incredible people: - Ben Williams - it's all Ben's fault I'm doing this now, his idea and support - Pavel Rodionov - trained 3D Gaussian splats, the model you see - Jason Lynch - photogrammetry and Metashape guru - Andrew Mogg - all the photogrammetry heavy lifting - Rindah Talitha Vida - leading the study for the reef you see https://lnkd.in/gq_u46HD - Michael Vasilkovsky - cutting-edge 3DGS theory stuff - Tim Lamont -- coordinating everything from the academia side - Mark Kellogg -- amazing 3DGS viz library we use - Mars buildingcoral.com team for restoring this reef you see and many many more fantastic people! 🔵 Everything you see is open-source (MIT licence, just copy-paste us...). Give us a star on GitHub: https://lnkd.in/gncgfVvZ 🔵 Exciting times ahead!! 🪸🌎

Since I left Google X I've been exploring Oceans + Biodiversity + AI space building my startup wildflow.ai (foundation models for coral reefs and beyond). It's been a journey... Finally, my path has crystalised! Now I know where I'm going, and most importantly how. Stoked to tell you! 🥳 🪸 🔵 https://lnkd.in/eixESM8p -- company vision doc 🔵 I'll start with why 🙋 • Life on Earth has been evolving for billions of years. • Amazon rainforest, coral reefs and other ecosystems are incredibly complex. • But it's not just random things, it's a beautiful complexity. • They are inconceivably rich! • Unfortunately, we don't really know how ecosystems work. • The ocean is still like a black box. • Poor understanding leads to poor decisions that eventually harm nature. • Sadly, we already lost 69% of living things on the planet in the past 50 years. • The good news, we now have enough knowledge and tools. 🔵 For the first time, we could become the generation that actually leaves behind nature better than we found it! 🔵 "Understand nature. Empower action."   • Wildflow is on a quest to comprehensively understand nature and empower humans to make better decisions on protecting our ecosystems, starting with coral reefs. • We’re building multimodal foundation models for biodiversity, a “brain” of the digital nervous system for our planet. • This system deeply understands complex ecosystem dynamics and guides human activities, enabling billions to flood into conservation/restoration and more. • Most importantly it brings balance between humans and nature. • And in the future it gives us superpowers to make a jungle out of a desert and even terraform Mars! 🔵 Here's the company vision doc: https://lnkd.in/eixESM8p We can do it together! 🌎 🤗

The panel discussion and Q&A session titled "World Regeneration" powered by AI x Impact, explored the transformative power of AI on global challenges. Moderated by Dijana Galijasevic 🌍, the panel featured insights from Sergei Nozdrenkov, Anastasia Volkova, PhD, Špela Poklukar, and Sabine Wiesmüller, PhD. The discussion highlighted how AI can drive significant positive change and contribute to solving critical issues, offering a range of perspectives on the future of AI in world regeneration. #AISummit #KissTheFuture #AIxImpact #PanelDiscussion #Sarajevo2024

TL;DR: how to render high-res 3D models of coral reefs (>100GB) using 3D Gaussian splatting? Any ideas? Background: There’s a huge amount of high-res coral reef photogrammetry data in the world (e.g. created with Metashape, Zephyr, etc). It would be great to have an open-source/open-data platform to host 3D models so that anyone could see different reefs, zoom in, rotate, etc. And, of course, run the scientific data pipelines/analytics against all the data. Point clouds out of Metashape look great! I’m considering 3 ways of rendering: - Point cloud: looks amazing from far away, but if we zoom in, becomes too sparse, so hard to make sense of what’s happening there. - Mesh: the most standard way, but reefs are noisy in terms of geometry and color; the output is not fantastic. - 3DGS: From what I’ve seen, this looks the best for the noisy models. Planning to go ahead with this one. We need to precalculate splats. The process is going to be like this: - (images, poses, point cloud) -> 3DGS. We take original images, reconstructed camera poses, our high-res point clouds, and train Gaussian splats. - Initialise splats with point cloud. - Use a gradient descent against the similarity function, comparing the original image against how the splats look from that camera pose. Now, how do we build this? I see these components: - Backend: a smart way of organising splats and serving them. - Client: client-side rendering of the splats in the browser, convenient navigation, etc. - Transmitting the data in a smart way. - A pipeline to precalculate splats and clean them a bit. Backend: - Something like Octree comes to mind. We also need different LOD (levels of detail). - Cesium seems nice for mesh, we need smth like that for splats. Potree does this for point clouds. - Anything out there for 3DGS? Perhaps need to build my own. Client: - Pure WebGL/WebGPU could be fun, but I hope there’s something good open-source out there. Tree.js and Playcanvas seem great. What else? - How do I know what nodes of Octree to query? Do I query everything in front of the camera of everything around camera from all sides? - How do I know which LOD to load from Octree? I guess it will be empirically measured and we decide based on the distance. Transmission: - I guess we use TCP/WebSockets. - Could even do my binary format and decode using Rust/WASM. Luma seems to do it that way. Impressive how many details 10MB of data could transmit. - Will start simple, just serve a file from S3/GCS via HTTP haha Splats creation: - Do we train every LOD separately? Or is there a way to downsample high-res splats? - Hopefully, we won’t get too many weird artefacts that must be removed (like massive shard-looking splats). Playcanvas could help? An alternative to all of this would be rendering the scene on the backend (e.g. using GPU) and then stream video to the client. Like Nira does: https://lnkd.in/eTGJmthq. But that's more expensive. Any recommendations?