How to Scrape eBay Pages for Price Intelligence

Gabriel Cioci on Aug 10 2021

Let me paint a picture for you. You just launched your rare retro video game cartridge online shop, and you don’t really know how to handle the product pricing.

Setting the price for a product is always a challenge: if the price is too low, you cut from the profit margin. If the price is too high, customers will not be interested in the product and will buy it from a competitor.

Decisions, Decisions. What can we do? Given this problem, you have two solutions:

Spend hours every day to research the prices from competitors
Be smart about it and create an automatic script that does this mundane task for you automatically

Can you guess which option we’ll pick for this article? Because time is money and I like to make my life easier whenever I can, we’ll go with the second option.

If the first option sounds more fun to you, you might as well read this article anyway. It’s clear that you have too much free time on your hands.

How scraping eBay will help your online store

eBay is the second largest online eCommerce in the US, covering 19.7% of the market. With 182 million active users, eBay is one of the best sites we can use for price intelligence.

For this example, we will scrape only the product’s price, but eBay is a data treasure. You can always scrape the reviews section, extract negative keywords, do a sentiment analysis, and determine why customers are unhappy with the product they bought.

You can also scrape the People who viewed this item also viewed section to find out what customers want. You can use this information to add new products to your shop or create discounted bundles with more products. People will love them!

How to scrape eBay with WebScrapingAPI

One of the biggest hurdles in web scraping is avoiding a block from the website you’re interested in. Luckily, you don’t have to worry about that when you have the right tool. In this case, WebScrapingAPI is the best solution for three simple reasons:

We offer 1,000 requests per month with the free package, no card required.
The API has a proxy pool of 100M+ rotating IPs, guaranteeing no request overlap.
You can integrate with their API with a single line of code.

You might wonder why we use a scraping API instead of building our own. The reason is plain simple: building a scraping tool from scratch might take you some weeks.

The biggest problem is not making the tool itself but fine-tuning it, so sites don’t flag it as a bot.

应用程序接口访问密钥和身份验证

Go to the WebScrapingAPI website and create an account. You will get an API key and 1,000 requests for free.

Install the dependencies

Our pet project is lightweight and requires only two dependencies: axios and cheerio. Axios is a promise-based HTTP client for Node.JS. We use this library to call the WebScrapingAPI. The second library, cheerio, is a lean implementation of jQuery for the server, and we use it to parse the product page. Use this command to install the dependencies:

npm install axios cheerio

Bootstrap the project

Create an index.js file and paste the following code into it:

const cheerio = require('cheerio');
const axios = require('axios');

const api_key = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
const product_url = 'https://www.ebay.com/itm/174620280276?epid=4444&hash=item28a82e05d4:g:UpMAAOSwfY5gYDr6';
const api_url = `https://api.webscrapingapi.com/v1?api_key=${api_key}&url=${encodeURIComponent(product_url)}`;
const hour_in_ms = 1000 * 60 * 60;

let product_price = null;

const check_price = async () => {

	// Scrape the document

	// Parse the document

	// Process the results

	// Check for any price change in one hour
	setTimeout(check_price, hour_in_ms)

}

(async () => check_price())()

As you can see, we use a recursive function with a timeout to scrape the product page every hour and check for any price change.

Scrape the page(s)

Replace // Scrape the document with this code:

let response;

try {
	response = await axios.get(api_url);
} catch (error) {
	console.log(error);
	process.exit();
}

const $ = cheerio.load(response.data);

This code uses the WebScrapingAPI to scrape the eBay product page and converts the results to a cheerio instance.

Inspect the source code

We scraped the entire product page but we need just the product price. We can use the cheerio instance to get the price value. The selector for the product price is #mainContent [itemprop="price"].

Parse the HTML

Replace // Parse the document with the following code:

let price = $('#mainContent [itemprop="price"]').html()

Process the results

The price comes in this format: CURRENCY SYMBOL AMOUNT. We need to extract only the amount. Replace // Process the results with this code:

price = parseInt(price
	.split(' ')[1]
	.replace('$', '')
	.replace('&nbsp;', '')
	.split(',').join('')
	.split('.')[0])

if(!product_price) {
	console.log(`Initial product price:`, price)
} else {
	if(product_price !== price) {
    		console.log('New price for product:', price)
	}
}

product_price = price

This code will clean the price by doing the following actions:

Remove the currency (US)
Remove the currency symbol ($)
Remove any whitespace ( )
Remove commas from the number
Remove the decimals

Once the price is parsed, we log the initial cost, or the new price if it changed between the scraping sessions.

结论

With a few lines of code and a fantastic tool like WebScrapingAPI, we managed to have a script that fetches the price of any product from eBay. From here, the sky's the limit. You can take the script to the next level and implement the following improvements:

Scrape the product title for easier price classification
Scrape multiple products and compare the price
Scrape another eBay website (like ebay.de or ebay.ca) and compare the price for the same product
Flip the table and scrape the Amazon product page and make a comparison for prices between the two sites for the same product
Implement email notifications; you should use nodemailer to accomplish this
Implement Slack notifications; you can use this library to complete the task

The best part about this applied solution for a real use case is that you can do all this without spending neither money nor precious time. That’s because WebScrapingAPI has a two-week free trial that lets you use all its functionalities for free!