ScyllaDB’s Post

💡 Tuesday tech tip: At #ScyllaDB, we're big fans of adding flexibility to the data model. In our free ScyllaDB University lesson, you can learn how user-defined types allow users to define more complex structures and attach multiple data fields to a single column. #NoSQL #NoSQLdatabase #TechTip

💡 Tuesday tech tip: At #ScyllaDB, we're big fans of adding flexibility to the data model. In our free ScyllaDB University lesson, you can learn how user-defined types allow users to define more complex structures and attach multiple data fields to a single column. #NoSQL #NoSQLdatabase #TechTip

Searching through one million records is very fast thanks to Elasticsearch and asynchronous data mutation operations https://lnkd.in/dUhCjV3T #Dotnet #Elasticsearch #Microservices

Searching through one million records is very fast thanks to Elasticsearch and asynchronous data mutation operations

Today we are looking into the influence of defining a schema when loading data like JSON, CSV and PARQUET. You will be surprised about the performance improvement you can get out of it loading 😊 https://lnkd.in/dX25CZpa In this session we will: - Have a look how a schema looks like - Evaluate the performance gain of schema definitions - Identify other benefits defining a schema Did you miss intro video into File Formats? https://lnkd.in/dQ6VViFc You want to master Data Engineering with PySpark? Subscribe here: https://lnkd.in/duVbCwRz Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.    Video link here: https://lnkd.in/dX25CZpa   #spark #pyspark #dataengineering #dataengineeringessentials       

🔍 Writing queries in #AzureCosmosDB just got easier! Discover the enhanced error messaging in Data Explorer that helps you find and fix issues quickly. Full details here: https://lnkd.in/ei8aT4mg

🔧 Behind the Scenes: The Complexity of Processing Time Series Data for Monitoring and Observability 🔧 When it comes to monitoring and observability, time series data plays a pivotal role. But what most don't see is the complexity behind processing and storing this data at scale. ⏱️ Handling massive streams of metrics, logs, and events in real-time requires a solution that's not only fast but also efficient in managing disk space, query performance, and data retention policies. In my latest article, I explore tstorage(https://lnkd.in/gmb5iS_S), a time series embedded database that highlights how things works in terms of storage, performance and compression used in storing massive amount of data. Check out my deep dive article in https://lnkd.in/gn8BXh3N #Monitoring #Observability #TimeSeriesData #DataEngineering #TechInsights #TStorage #Performance

The "shuffle" is an expensive operation Spark sometimes needs to do. It takes place for transformations with a "wide dependency". This can be the case for `join` and `groupBy` operations, for example. It cannot happen for other operations, like `filter` or `union`. You can look at the physical execution plan of a query to see if a shuffle is done. Call `explain` on your dataframe and look for "Exchange" (see screenshot). Each "Exchange ..." line indicates a shuffle. A wide dependency means a single partition of a child RDD uses all parent RDD partitions as input. But a Spark "task" operates on a single partition, not multiple. Spark solves this by shuffling data in the parent partitions to reorganize their data. It prepares one "intermediate" partition with a "narrow dependency" to each target partition in the child RDD. Further processing is then done with one task per intermediate partition. Shuffling is expensive because it involves: - sorting the data to align with target partitions - disk IO to write the sorted data to a file on disk - network IO to send data around, when parent partitions live accross different nodes in a cluster Shuffles often cannot be prevented, but it's always good to be aware of them. #dataengineering #softwareengineering

Use case alert - Learn how Route powers always-on data for 1+ billion orders with #CockroachDB. https://lnkd.in/giNqwhk3

Data.olllo Version 6.0 Release Notes Multi-Core Support P Core: The classic core, optimized for datasets up to tens of millions. V Core: Tailored for datasets in the billions, excelling with large HDFS files. X Core: Enhanced classic core, designed for terabyte-scale data, supporting GPU acceleration and multi-threading. Basic P Core features are free and cover most data processing scenarios. New Features Enhanced core functions. Introduced a comprehensive data visualization module. Added a Tools Map for easier navigation and usage. Introduced a Regex Map for advanced filtering capabilities. Added a subscription model for advanced features. Bug Fixes Resolved issues with English language hints. Fixed bugs affecting existing functionalities. Version Details Release Version: 6.0