I think there is a huge, huge gap between TwiDiBook's needs and the needs of the overwhelming majority of sites on the Internet. There are plenty of applications which do real things for real people which will never have scaling issues associated with them. The vast majority of the remainder of applications will have issues which are tractable by fairly simple solutions. And then we get to the pathbreakers who have to invent new methods of engineering to keep up with their growth curves.
The party is just getting started for "Hey, check it: we can put a cache in front of our database" for that broad middle of the curve. (This includes applications where relational databases are a great fit but for certain performance requirements in small bits of the program. For example, you really don't want to have to implement a university backend system in a key/value store. Just trust me on this one. It is virtually made for SQL, but while 99% of the use of the system is less than performance intensive, having resources in place to support that last 1% costs the university hundreds of thousands of dollars. I think we could slash that with creative caching.)
I'd agree with all that, but add a disclaimer. The rise of cheap and easy solutions for handling Big Data makes a lot of new applications feasible. All those cool correlations that Amazon does to suggest new books? You can do that at home now. Got a problem that would benefit from running an analysis of hundreds of millions of pages? You don't have to be Google to implement that.
It is a bit maddening to see people applying the NoSQL hammer to problems that don't need scalability, but it's partly driven by the realization that they open up opportunities to build entirely new applications.
Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it. Maybe it is for Amazon which got plenty of data to analyze, but not for a startup.
Personally I don't trust NoSql solutions because they aren't mature enough. I've had little problems using MySql/Postgres and never had loss of data ... and I can always come up with a non-relational scheme latter built on top ... when I really need it.
The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start.
I've also used MySql instead of Memcached successfully for a website with tens of millions of hits per day, preferring it because Memcached crashes on me a lot and there are big differences between versions I don't want to worry about ... and yeah, I know async reads are problematic with Mysql/Postgres, but I just threw a web service in front of it, and when using the Memory storage engine it really kicks ass.
> Amazon's correlations aren't done in real time, and I don't get why it would be an issue using MySql for it.
Many of the algorithms for recommender systems are especially known for being parallelizable and would be extremely difficult to reliably and quickly implement on top of MySQL (or Postgres or Oracle). I can't even think of a way to do so without making a fairly complex ETL cycle with a separate DWH and a main serving DB (with points at which the site becomes degraded and possibly unavailable). As I've mentioned earlier, with a system like Hadoop, you could start off by implementing these algorithms in Perl or Python (making use extensive libraries already available in the language).
Recommendations may not be real time, but there's a huge time difference between doing an even fairly simple "market basket"-type analysis for a data set orders of magnitude difference smaller than Amazon's in MySQL and doing this on top of Hadoop + HBase/Voldemort/Cassandra (i.e., compute in Hadoop, serve from a NoSQL db which offers Hadoop integration). You could even start by doing the computation in Hadoop and loading the data into MySQL (while, perhaps, serving the data from a slave db).
> Maybe it is for Amazon which got plenty of data to analyze, but not for a startup.
Your start-up doesn't need to have tons of users to have tons of data. This is especially true if you're building a start-up to make sense of the data that's already out there (some YC examples are Flightcaster, Directed Edge). For a consumer start-up, ability to have cheap and scalable storage means you can record data much more frequently.
> The other problem I've had with Nosql is that capabilities vary from implementation to implementation, and my software really evolves a lot ... I don't know what my needs are from the start.
Isn't this the point of separating your data model from your presentation layer? Begin with MySQL (or even simpler, with BerkeleyDB) -- change it to whatever else suits your needs.
I think what the parent meant is that there are now tools available to implement the algorithms used by Amazon et al for recommendations. It isn't about having more users, it's about being able to keep track of great deal more data and use that data at great deal more points in the applications. That's what Hadoop, NoSQL et al allow you to do (e.g. Voldemort's and HBase's ability efficiently load and serve data prepared in Hadoop; afaik Cassandra has Hadoop integration now as well).
Now even if you are okay with low throughput (you don't see yourself become a major Internet destination), you can still benefit from non-relational approaches.
Implementing a recommender system or PageRank on top of a relational database is simply painful and requires expertise in that system's SQL dialect (someone who's done this on top of Oracle won't be able to do this -- in the same amount of time -- on top of MySQL or Postgres). With Hadoop
you can start with a Perl or Python script (and make use of libraries like scipy or pysvm) and then gradually re-write the performance critical parts in Java. Even if you aren't and will never get to Facebook's or Amazon's scale, you're now able to build intelligent applications on top of your data.
You can also sleep sound at night (i.e., not in front of a terminal tailing your logs or watching ganglia) knowing that a load spike (which could be start of a new financial quarter, holiday shopping season; something as positive as a NYTimes article or -- much more likely -- as negative as a DDOS attack) won't bring you to an outage.
The algorithms themselves, however, are very difficult and require a background in machine learning; that is even to use an existing implementation of these algorithms (e.g. Apache Mahout). I am not an expert in these topics. I took a graduate course in machine learning, but that only demonstrated to me how little I know and how much more I have left to learn. It's this expertise that coupled with ability to implement a scalable infrastructure with open source/in-house developed software (instead of relying on commercial vendors) that makes your a business idea (you know, the one that the "business guys" think they can hire bunch of MSCEs to build in ASP.NET/J2EE/PHP/RoR) defensible from some "kid" in Harvard dorms (especially if that kid declined a $1mm starting bonus to work for Microsoft instead of going to Harvard).
Don't get me wrong. SQL is an amazingly example of a domain specific language; RDBMs are wonderful works of software engineering. SQL RDBMs are great for many applications (e.g., ERP systems and their functional equivalents like the university registration system patio11 mentioned). Yet they're not made to solve the kinds of problems/create the kind of applications that simply didn't exist when they were created. The fact that they can at times be used for this doesn't mean they should: I can write Russian with Latin instead of Cyrillic Alphabet (or write Lisp in vi instead of emacs), but it's a lot better to use the correct tool.
Better even is that PostgreSQL 9.0 will have solid master/slave async replication. PostgreSQL already supports better write concurrency and multi-core scaling than InnoDB or MyISAM, and the replication will bring web-style read scaling into the picture. PostgreSQL has a "free-er" license, a more unified community, and isn't burdened by possibly hostile ownership.
I don't think we can say PostgreSQL is truly 'better' than the non-relational databases which are beginning to permeate the web.
I believe the decision is more of a mySql/postgreSQL vs. non-relational databases.
Sure, PostgreSQL may have better implementations of certain features than MySQL, but for most web developers, mysql is good enough.
How would put PostgreSQL against non-relational dbs?
I'm only just getting into my first nosql (Cassandra), so I've got no real-world data on it, but I think it would perform in the "well enough" category in performance. Just implement the tables with two columns: a key and a value, and index the crap out of them.
It would also have the benefit of better tools and the option to use a JOIN or a transaction where it made sense. Of course, you could do the same with MySql, too.
I'm not advocating PostgreSQL over a NoSQL. It certainly won't scale like Cassandra. I'm advocating it over MySQL. With the 9.0 release, it will be to a point where there won't be any reason to chose MySQL. PostgreSQL is easier to deploy and simpler to configure, has more features, is less likely to screw up your data, and has more scalable performance.
Yes, some parts of the market don't need the kind of scale talked about here (you'd think the domain name would tip you off). And plenty of people get by with MS Access, still. I don't really think either point adds to the discussion.
"Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores"
In other words, to get MySQL to superduper scale, you have to turn it into NoSQL anyway. At which point, the only reason to be using MySQL at all is inertia, because it's pretty clunky compared to the other alternatives. Back in the day before Cassandra and etc, there wasn't a concept for NoSQL, and sharding MySQL looked like an exotic way of using a relational DB. Actually, no. It's a way of crudely emulating something that isn't a relational DB.
The party is just getting started for "Hey, check it: we can put a cache in front of our database" for that broad middle of the curve. (This includes applications where relational databases are a great fit but for certain performance requirements in small bits of the program. For example, you really don't want to have to implement a university backend system in a key/value store. Just trust me on this one. It is virtually made for SQL, but while 99% of the use of the system is less than performance intensive, having resources in place to support that last 1% costs the university hundreds of thousands of dollars. I think we could slash that with creative caching.)