Dremio adds first generative AI-infused tool, intros others

Two to come

While Text-to-SQL is now available, Autonomous Semantic Layer and Vector Lakehouse are still in the preview stage.

Autonomous Semantic Layer aims to automate the cumbersome process of modeling and cataloging data. To make data accessible and usable, all data is defined and categorized as it is ingested and integrated.

Those semantics -- the definitions and categorization -- make data points and datasets easier to find amid countless other data points and thousands of datasets. In addition, they make relationships between data points and datasets easier to discover.

But defining and categorizing data is monotonous and time-consuming.

Dremio, therefore, is developing Autonomous Semantic Layer with generative AI to automate much of data cataloging that otherwise has to be done manually. The generative AI technology automatically learn the details of users' data and develops the semantic descriptions of datasets, columns and relationships.

The result is that data engineers no longer have to manually catalog all of their organization's data. The tool does it for them.

As with Text-to-SQL, the aim of Autonomous Semantic Layer is to improve efficiency to ultimately reach insights more quickly, according to Shiran.

"Data teams spend most of their time getting their data cleaned, documented and prepared, which slows down the time to insight," he said. "The Autonomous Semantic Layer aims to solve this problem through automatic documentation, automatic data acceleration and semantic search."

Vector Lakehouse, meanwhile, is designed to improve semantic searches, recommendation systems and anomaly detection, according to Shiran.

He noted that one of generative AI's unique capabilities is its ability to automatically create vector embeddings -- numerical representations of unstructured data such as text, images and videos. Those representations essentially brings structure to the unstructured data points, which enables them to be combined with structured data, categorized for discoverability and ultimately used for analysis.

Without generative AI, data teams are generally forced to deploy standalone vector databases, leading to isolated data repositories. But by combining Dremio's data lakehouse platform with generative AI, the vendor aims to enable customers to store their unstructured data along with their structured data for easier access and more efficient analysis.

"Companies will be able to manage and process [vector] embeddings alongside source data in an open data lakehouse architecture," Shiran said.

Future plans

The combination of generative AI with data management and analytics capabilities is in its nascent stage, as Bond noted.

Therefore, there's no specific set of generative AI-infused capabilities that Dremio and all other data management vendors need to build in order to remain competitive. Each is in its exploration phase and coming up with its own ideas.

"We are still very early days," Bond said. "But there is a lot of opportunity for what will be possible."

Tagging and classification -- which Dremio says it's addressing -- is one area that can benefit significantly from generative AI, he continued. So is using natural language to not only translate queries into code but also to develop entire data pipelines.

Meanwhile, "Dremio's roadmap is centered around cost efficiency and speed-to-insight," Shiran said of the vendor's plans.

One tool the vendor has under development that aims to help organizations control their cloud computing costs is Dremio Artic. It is a lakehouse catalog designed to reduce infrastructure costs by eliminating the need to replicate data across development, testing and production environments.

When the same data is needed in three different environments, costs triple.

So Artic, which provides users with branches within the same environment rather than forcing them to build three separate cloud environments, has the potential to significantly lower organizations' cloud computing costs, according to Shiran.

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.