Build with Catalyst: Web crawler for ecommerce using Catalyst SmartBrowz

Continuing the Build with Catalyst series, today we dig into the necessity of incorporating a web-crawler function into your business and what you can gain from it. You can build a crawler using SmartBrowz.

Before we start, if you want to learn how you can build an AI-based lead scoring engine using Catalyst Quick ML, check out our previous post in the series. Now, let's get started.

What does SmartBrowz do?

SmartBrowz takes care of headless browser management, browser isolation, and related dependencies. This ensures you can implement browser automation functionalities without worrying about memory issues and server provision.

With SmartBrowz you can:

• Automate using a headless browser.
• Generate PDFs and screenshots using templates.
• Leverage the power of browser automation.

Web crawler for ecommerce

Let's take a look at something that comes across in your day-to-day life: ecommerce. With the advent of ecommerce, there's been an increasing number of brands that sell their products via various ecommerce marketplaces. Buyers surf multiple ecommerce platforms before making a purchase. To ensure consistency and earn confidence, businesses selling online need to ensure:

• Their products are available across all zip codes (inventory is available across multiple warehouses).
• They are on the buy box, and nobody else is selling what they are for an unreasonably lower price, especially if they're selling directly.

Traditionally, brands rely on reports generated by the retailer to monitor such information. However, in the age of flash sales and one-click purchases, this information is best gathered as quickly as possible. Given the scale of operations (number of products, zip codes), it becomes difficult to monitor this information manually.

To counter this issue, you need a scalable, reliable, and easily manageable browser automation solution.

Enter Smartbrowz! Using Catalyst Smartbrowz, you can automate the crawl function. Imagine having an automated service that opens the URL for your product/s, checks if it's available to "Buy Now" for a whole list of zip codes, and even monitors the price for you. If it finds that the products are being sold for a lower price by a different seller instead of your own authorized seller, it takes a screenshot, generates a PDF, and informs you.

Here's how you can set up a price crawler in a step-by-step manner:

1. Browser Logic in SmartBrowz helps you create custom functions for your needs. Here, you can create functions that can crawl and scrape to help you gather information you require from the web. This could range from the price, to availability or the percentage in price change, percentage of offer provided on the price, and more.
2. You can easily write a simple script in Browser Logic with browser automation libraries and automate repetitive tasks. This ensures the actions are carried out at regular intervals with minimal intervention needed.
3. You can generate charts, dashboards, and reports with the gathered data for easy understanding and interpretation. With the dynamic data (JSON), you can load and generate charts, dashboards, and reports first with the JS and convert them into PDFs or screenshots.

This capability can be adopted across various industries and for various use cases; get creative, automate smartly, and make informed, data-driven decisions.

Feel free to share your thoughts in the comments section or get a detailed 1:1 session. We will come back next week with another compelling use case. Stay tuned to this space to see how you can take full advantage of Catalyst and its capabilities.