Note that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
Definitely agree. I'd also recommend using mobile sites for scraping as they tend to be simpler and load faster (assuming they contain all of the data you are interested in).
How do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?
You reverse engineer the application, or you run it in a debugger.
If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions.
If it features HMAC signing, you decompile the app, find the key, reverse engineer the algorithm that chooses and sorts the paramaters for the HMAC function, and rewrite it outside the app. If the key is generated dynamically you reverse engineer that too, and if it's retrieved from native .so files you're going to have a fun time, but it's still quite doable.
All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.
Once you do the one-time work of pulling out the key, you can just add something like, "secret_key=foobar" to your requests, and you're back to happily parsing JSON.
If they keep changing it up, I'm sure you could automate the decompiling process. The reality is that this technique is security by obscurity at its core, and is therefore never going to succeed.
In many cases, if all you're after is the data, you don't even need to reverse-engineer much; I'm not familiar with the mobile world but in Windows you can just send input events to the process, essentially simulating a user, and read its output in a similar fashion. You can still treat the app like a black box and regardless of how much obfuscation they put in the code, you still get the output.
(This technique is useful not just for scraping data, but for UI testing of your own apps.)
> All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.
This is totally true, but the original premise was to do it just with a MITM. I was being generous and assuming most apps do dynamic generation of their keys. I'm probably wrong now that I think about it.
I feel like at a certain point this crosses the line from unintended use of a private API to unethical hacking.
If the data owner went through the trouble of encrypting the traffic between it and it's app they have a certain expectation of private communications that you'd better have a damn good reason for violating.
When an application you have legally installed on your own computer is communicating with the outside world, it seems a fundamental right to inspect the exchanged data to check that it is not leaking personal information. If the data is encrypted or obfuscated, this could make us suspicious (why hiding if there is nothing to hide ?) and gives additional motivation to audit the security.
Once the api is reverse engineered, we might be tempted to improve the usability of the application by adding some features (scraping all data). If this hurts the server (huge resource consumption), this becomes unethical and may become illegal.
It's certainly unintended use of a mobile API, but it's not hacking; it's reverse engineering. HMAC is used for client integrity verification as a signing algorithm; it's not used for generating hashes or ciphertexts of confidential user data. Furthermore, even if it were, it's operating on data that we are sending to the server in the first place. We aren't actually breaking encryption or cracking hashes for confidential user data, we are choosing to manually sign messages to the server using the same methodology as the application itself. Cryptographically speaking there is a very large difference in utility here. The only actual encryption present is the TLS, but both you and the server ultimately see the data.
Reverse engineering occupies a much more ethically and legally grey area than outright hacking because you are fundamentally taking software in your possession and modifying it. There are strong arguments that people should have the right to do this. If can lead to hacking, and it's useful for security research, but it is not in of itself an attack on the application's security (you could make a case that it is an attack on the application's trust model, however).
Now, if the developers relied on the privacy of the API as a form of implicit authorization (i.e. by forging requests from the client I can retrieve another user's data using an insecure direct object reference on a username paramater), and I proceed to do that - yes, that's hacking. You're accessing confidential data in an unauthorized manner, just as you would be if an insecure direct object reference were present on the website. The developers made a mistake in conflating client validation with user authorization, but you've still passed a boundary there.
It is arguable that this is unethical or at least amoral, but if all you're doing is scraping publicly available data using the public mobile API, it is at least legally defensible until the other party sends you a C&D for using their API in an unauthorized manner (so long as you haven't agreed to a TOS by using the mobile API, which really depends on whether and how prominently they have a browserwrap in place). I think the spirit of your point is that someone probably just shouldn't be using an API if they're not authorized to do so, but it's a very important legal and technical distinction to make here that you aren't hacking by reversing the embedded HMAC process.
Not off the top of my head. The knowledge isn't tribal, but it's certainly scattered (few blog posts will give you take you the whole way) and the tools are...spartan.
I recommend you read CTF writeups (there was one hosted on GitHub where a team retrieved the request signing key for Instagram IIRC). Those are usually very tutorial-like, though they tend to take some level of knowledge for granted even if they don't intend to.
The other thing to do is pick up apktool, JD-GUI, dex2jar and maybe IDA Pro, Hopper or JEB and learn them as you you go.
Android tends to be easier to decompile if you want to discover stored keys, etc. I've done it a few times by using Charles on desktop + setting up a proxy to run my iOS connection through my laptop, and then running the app on mobile.
>All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.
Do note that if you become annoying and/or conspicuous enough, they can use legal force to stop you, and if the case actually goes through the process, you'll almost surely lose. This is true at least in the United States and Europe.
If they're smart enough to collect data in the app to detect when a request to their api is actually coming from their app (versus one coming from curl or from a script), then your reverse engineering problem just went from painful to unprofitable (i.e. not worth your time).
I'm not sure I understand this. The point of the reverse engineering is to simulate (exactly) requests from the app, using your script, thus making it impossible for them to detect. And the data collected in the app has no bearing on your scripted calls to the API outside the app.
I suppose I can envisage a scenario where the app sends a counter as a request id, and my script will then send the next value of the counter as its id, causing the next request from the app to re-use a counter value and thus fail, but the server API and the app have no way of knowing this is due to my script and not, say, network issues, and therefore it should still not affect my reverse engineering abilities...
Maybe, taking this further, the app could have baked in a certain number of unique, opaque 'request tokens' that are valid for one API call only, and when my script has used all of them it will cease to work, and in doing so cause my copy of the app to become useless, but again, not an insurmountable barrier.
Never assume something is impossible. Tokens/counters that are generated at runtime inside the app is a start at countering bots. But there are MUCH more advanced techniques and big businesses that are built upon helping mobile and webapps detect bots and other scripted requests.
Start with the mitm. You'd be surprised how many apps have no certificate pinning. If there's pinning, you just need a jailbreak so you can override the https method to ignore certificate pinning. See: iseclabs ssl killswitch
Some apps use private keys baked into the app, but you can usually recover those from memory. Do this using lldb and a remote debug server on a jailbroken iPhone over USB.
Or you can treat the app as a black box and use a phone (or multiple) as a "signing server" to sign API requests before sending them via, e.g. curl or python. To do this you install the app on a jailbroken phone, tweak it to expose an http server on the phone that listens for API requests, feeds them to the "sign API client request" method, and then returns the signed request. This method has the benefit of being resilient to frequent app updates.
I guess I was being generous in my assumption that the apps will generate keys dynamically, making that not useful for a repeat attack as it were. I'm probably wrong though, most apps probably use a single baked in key.
The app and api can share a secret generation method. I once wanted to use the api of an iOS app that sent a timestamp + dynamic token based on that timestamp in the request headers. The timestamp/token combination was validated by the server, which had a tolerance of five minutes (so replaying the same timestamp/key combination that you observed via MITM would stop working within five minutes). Rather than try to work out the algorithm, the approach was to dump the app headers and work out where the key was being generated, then used cycript to attach to the running app and invoke the `tokenForTimestamp:` method to generate valid timestamp/token pairs at one second intervals out for the next several years. Still working a year on :)
Most of the "worthwhile" apps have certificate pinning and HMAC-signing in place these days, so it's a bit more of an onerous process than simply booting up Charles and playing with the API. But yeah, it's definitely better to deal with JSON outputs than have to parse HTML.
> you realize if you piss off anyone doing this, they can have you sent to jail right?
Are you talking about Aaron Swartz? There were complicating factors. Typically, no, scraping is not a criminal matter. If you need a lawyer, however, you should talk to one.
> Automated scraping is unlikely to qualify as fair use in court.
Scraping is not [usually] the problem. It is what you do with the scraped content that is the problem. Additionally, the nature of the scraped content itself is at issue: facts are not copyrightable, for instance, just particular expressions.
Fair use is a hugely, hugely complex topic. If you ever ask someone if something is fair use and they give you a straight "yes" or "no" answer - don't trust it unless they are also showing you a court decision that covers your exact scenario and is valid in your jurisdiction. Any good answer will include a healthy does of "it depends."
The only way to conclusively determine if something is fair use is going through the fair use "run-time" - which is a court decision on the topic. There is no other way - none - to determine if something is fair use and claims to the contrary are false. To be clear: fair use is a judicial determination and it is highly fact specific.
Disclaimer: I am a lawyer, but I am not your lawyer. If you need one, you should get one.
Perfectly feasible technically. However aren't you stepping into a legal minefield? After all, it's a "private" API. Furthermore, since you're explicitly going around the hurdles they've laid out for the public (captchas, T&C), you're surely painting a target on your head?
Note, I'm not debating whether this is a good or bad thing, just that in the current environment this is surely a legally dodgy manoeuvre.
Reminds me of when app developers found the private API of the Pokemon GO app and used it to create their own Pokemon locater apps. Many of them made it to the top free charts and would have raked in hundreds of thousands of downloads, if not millions.
Another fun aspect of that - most banks don't release public APIs giving safety as a reason. On the other hand, each one of them have mobile apps with APIs inside.
