High-quality data is expensive — and that’s a good thing.

1. It’s difficult to get the data

To get competitor pricing data, a tool called a spider “crawls” the internet and finds the information you’re requesting. This tool needs to be smart because if a website changes its design, for example, the spider might not find the information it needs.

To keep the spidering tool up-to-date, you need a team of people monitoring the web and consistently updating the tool.

2. There are two sources of competitor pricing data

First, you can get competitor prices straight from the source through direct scraping. This method goes to your competitors’ websites and reports back exactly how much they charge for each product. You’ll get accurate prices, but you won’t know how the market as a whole prices that product.

Alternatively, you can get competitor prices from comparison shopping engines (CSEs). These prices highlight different market dynamics at play, but if your competitor doesn’t advertise a product on a comparison shopping engine, you won’t know how they price that product.

We recommend using a combination of direct scraping and comparison shopping engine data.

3. Not all data is the same

When you compare two data services, take note of the service they provide. There’s a big difference between managed scraping services and unmanaged scraping services. Managed scraping services have higher costs, but generally produce higher quality data and save your team significant amounts of time.

4. Lots of vendors sell bad data

As with many things, that too-good-to-be-true price is just that: too good to be true. Data of all forms is in high demand, and there are plenty of vendors willing to sell you low-quality data for an astonishingly low price. These vendors skip out on vital safety checks that keep your data clean, organized, and up-to-date