Global retailer enlists Zyte for data-driven, AI-powered pricing intelligence

In 2022, we kicked off an ongoing managed data extraction project for a large global retailer (that, sadly, we cannot share – we take privacy very seriously), and the project has since matured into a collaborative and innovative partnership exploring exactly what is possible with web data today.

The client needed weekly pricing intelligence data from hundreds of websites to help set unique prices of products across stores in 30+ countries. To do that accurately, they needed more than just standard pricing data. Unusually, they needed some hard-to-access product attributes which were often buried inside completely unstructured text (such as description text), making it difficult to extract using standard practices.

These requirements and constraints, as well as the large number of global target sites, required the creation of a unique framework and hundreds of scripts, and some unique custom technology leveraging Zyte’s API's infrastructure.

Their project requirements included…

• Gathering product data from hundreds of different retail websites
• Gather data buried in unstructured text for each product
• Transform data to the client’s preferred format
• Ensure data is ethically and legally obtained
• Deliver weekly data feeds

Zyte’s data delivery team took care of everything, making it so easy the client could simply provide the list of target websites and SKUs and wait for the data to flow. 

For each product page, we used our powerful AI to extract all the product details matching our standard product schema, and then used an LLM (that we built) to go the extra mile and extract the non-standard items from the page.

The client has gotten exactly what they need, with some enhancements.

• Gathering product data from hundreds of different retail websites -- The machine learning model powering Zyte API’s AI Scraping accelerated spider implementation and successfully extracted most of the client’s data needs. Not only did we achieve this milestone, we did it three times faster using our AI-powered automatic product data extraction technology to accelerate the process, leaving only a handful to be manually created by our delivery team.

• Gather data buried in unstructured text for each product -- Zyte’s data delivery team developed and used an LLM-based solution to extract the relevant data from descriptions and unstructured text. The unstructured description text item was first extracted using our AI Scraping and then passed through the LLM’s special prompts created by the team.

• Transform data to the client’s preferred format -- After extracting the unstructured data via the LLM, it was transformed to the client’s standard using a Python library. The final transformed data was then incorporated into the final data schema for delivery.
• Ensure data is ethically and legally obtained -- Zyte’s world-class legal team is involved in every Zyte Data project. Before any work is started, they provide legal guidance on the project and evaluate the client’s data requirements ensuring it can be ethically and legally obtained.
• Deliver weekly data feeds -- Zyte’s data delivery team created a seamless integration point to the client’s cloud storage provider. This allowed the weekly data feeds to be delivered automatically.

The Result

• Vast amounts of complex and hard-to-collect data, successfully and consistently delivered to our client in record time using some of the most advanced technology available today.

• They wanted the best data available to price their products globally. Zyte delivered on-time, and at scale. We continue to explore new ways to enhance their data capabilities and leverage the latest technology confidently and reliably at scale. 

We want to hear about your project!

Reach out to tell us about your project. We love finding solutions to new challenges, and we have the brainpower and developer resources to get it done.