Caveat to my wasted effort comment:

Your'e an e-commerce site that has a problem with people buying goods (especially virtual goods, ebooks, gift cards, etc)[1] with stolen credit cards. You need a solution.

The hardest thing I've ever had to crawl (as I mention in another comment in this thread) has been linkedin and Facebook. Why? Because I have to be logged in to get the data I want.

If you want to stop crawlers you can also put the good stuff behind auth, but you need a solid auth mechanism. You can't just do email verification because I'll use shit like https://www.mailinator.com to generate a ton of fake emails to sign up for your site.

[1] Why virtual goods? You can't stop shipping or track down the person once the card is reported stolen.

> [1] Why virtual goods? You can't stop shipping or track down the person once the card is reported stolen.

In the meantime, virtual good are also zero-cost : when you sell a ebook and the transaction is cancelled by the bank, you didn't lose anything, it's not like the buyer was willing to pay anyway.

Until you see someone do it with buying gift cards. Those people are bastards. In that case using a service that helps detect fraudulent card usage is bueno.

I dont stop crawlers, I only randomly feed damaged/wrong data to crawlers.

I especially loving doing this for e-commerce sites. Now the table has turned. Try guess which fraction of your scrapped data were wrong.

My point is you’re not detecting the ones that you should be the most concerned with.

Problem is as developers, sometimes we only have hammers for these screws.

How would this stop crawling by users who cannot afford the API subscription or don't want to pay for it?

I think your suggestion would reduce crawling, but not prevent or block it.

Build a revshare API.

I assume the people that can't afford it are using it for non-revenue generating purposes. I don't know that I'd care about those people if they aren't taking away from my bottom line.

Because you'd presumably need to authenticate (and have a paid account) to access the data.

I think they meant, since they can't afford the API, they'd just keep crawling your HTML. Combo move the API and putting your stuff behind auth and you solve a lot of the problem.

You've gotta get that good yummy content publicly accessible for google though, so you'll rank. So, that's a balancing act.