My experience with MongoDB hasn't been the most pleasant in a write-heavy environment. Until they fix the write lock properly, MongoDB is pretty much useless for many high throughput applications in my opinion...
The new DB level locking introduced in this release is a joke. There's not much difference between that and the old global write lock unless you split your database in dozens of smaller ones. What a pain. I wish they'd just stop pretending and addressed the issue properly once and for all.
I really want to like and use MongoDB because the way data is represented and how it can be queried is awesome.
It's inaccurate that there's no difference between the global lock and database level locking. Whilst it's true that locking is now down at the database level and you get benefits from splitting into multiple databases, the real benefit is from the new PageFaultException architecture. Even using a single database you will see significant performance improvements.
That's good to know. It still feels like an half-assed solution to a very serious problem, however.
I definitely won't be looking at MongoDB again until this is fixed for good, and until I know replication is more reliable. Had terrible problems that that too unfortunately.
It might "feel" like a bad solution not granular enough but if you actually test the implementation then locking is no longer a real concern. There are still improvements to be made and this is the first step towards that. Changing from a global to database lock required huge amounts of work which sets the foundations for the next releases with collection and document level locking.
In my experience replication has been extremely stable and reliable for several point releases so I'm not sure what problems you had.
It's not half-assed so much as a first step. 10gen have previously stated that the plan is to gradually increase the granularity or write-locks over successive releases.
One of the reasons DB-level locking is actually quite important is that the replication oplog is on the 'local' database and your data is on other databases -- so your application is no longer fighting with replication for the write lock.
David knows what he is talking about, they use MongoDB in really high scale. Also, getting database level locking was like 90% of the work according to the CEO. Doing collection and finally document level locking should be somewhat trivial now. 10Gen just want's to take it slow, and make sure database level locking is working 100% correct before moving onto more granular locking.
Yep they sure are taking it nice and slow... at this pace, collection-level locking is probably only 2 more years away. Other nosql databases are nipping at their heels.
David may use MongoDB at really high scale, but since when is that a reason to call someone an expert.
DB Level Locking (https://jira.mongodb.org/browse/SERVER-4328) is a dependency of Collection Level Locking (https://jira.mongodb.org/browse/SERVER-1240). For a critical, tricky core piece of functionality like this it makes sense to tackle one major concurrency issue at a time, do a release, stabilise the code and verify there are no issues introduced by the new locking code and only then release the next step in locking enhancements. Doesn't seem like they're "pretending" about anything as far as I can tell.
Actually, there is a lot of difference but not just because of the global lock to database lock transition. There is also a lot of work done on yield-before-lock: http://blog.serverdensity.com/goodbye-global-lock-mongodb-2-... And don't forget that replication also happens in a different database...
The problems with early rev SQL servers were similar (in inconvenience) but, with feedback from users and hard work from developers, they soon moved beyond MVP.
I'm willing to suck up some inconvenience with MongoDB, it's one of the most inspiring technologies of the last decade. Web application persistance feels more like a fully connected limb now.
I can understand the gripes but do urge people to give 10gen some slack and appreciate that what they are building takes time - it's clear that the team are working hard on features driven by community feedback.
It's more than inconvenience when your whole service is down because some database queries take up to 30 seconds to complete (and often time out), and when replication on some servers is days behind (and some are up to date). We also had to completely flush our data a few times because of some corruption we couldn't recover from.
The truth is, MongoDB is awesome if your dataset fits in memory, but if you're in a write-intensive environment with 400-500GB of data, it's just not there yet.
Not true. We're processing 12TB of data each month with thousands of writes per second and sub-millisecond response times. You simply have to understand how to use MongoDB correctly which isn't difficult with the out of the box settings and by reading some of the documentation. Figuring out your working set is the most difficult part but that doesn't take long to calculate based on understanding the queries you're doing and creating the appropriate indexes.
Since you seem to be fairly negative about MongoDB but light on details, perhaps you should write up your experiences so others can learn from what you did wrong.
I have used Mongodb a bit, but only for applications that mostly need to read a lot. What issues might I run into if I need to do lot of writing as well?
None, almost certainly -- if you have enough RAM to fit your working data set, then MongoDB's writes are just changing memory. If it does need to read data from disk before changing it, the new yielding architecture that others have mentioned will help prevent lock contention.
I've been playing around with the Aggregation Framework lately (using the release candidate). The performance seems to be pretty reasonable, especially when compared to similar tasks with the old MR framework. A quick and dirty benchmark number in case anyone is interested:
* Obligatory unscientific, probably not meaningful, etc. disclaimer.
Results: The time range matched against above matches 42,466 documents within the collection. The average response time over 50 runs is 419ms. Not exactly "Big Data OLAP" stuff just yet, but plenty fast enough for most use cases involving reasonably small sets of data. Great job to the MongoDB team!
Out of curiosity, did you have an index on the ts field that you are matching with?
By the way, if you (or anyone else for that matter) come up with useful benchmarks I'd love to get a copy of them at mathias@10gen.com. I have a few of my own, but I'd like to get some real-world workloads from the community to test potential optimizations against.
could you go into why tag aware sharding is a "HELL YES" for you? I was kindof already of the opinion that mongo sharding is much too complicated (http://i.imgur.com/c3Dpq.jpg), this seems to compound that. Is it something you've needed/wanted so bad that its worth it?
For startup projects I love Mongo because we can get a product up and running very quickly. However I always feared in the back of my head we would have to move off of it if our service got too big. Maybe it is all the complaints from a small portion of heavy users. Regardless, big updates like this are going a long way to help make me feel content on continuing to use it as we grow.
> However I always feared in the back of my head we would have to move off of it if our service got too big
I like to call that "nice problem to have" territory. There is nothing to fear from being too successful, or from having the problems of success. Success problems can be solved by the application of people, time and money. And like all optimization you measure first to make sure you really have the problems you think you do, Amdahl's law etc.
Far more likely you'll have the problems of not being successful, such as no attention or money from customers and investors. Or not being in the business you think you are in. There isn't really that much point putting in infrastructure just in case you get successful, if that same infrastructure takes time to develop, and slows development.
With MongoDB, you are able to shard your system. This means you can grow your databases horizontally. This is not something you can easily (or cheaply) do in the world of RDBMS. You will see much better scalability with MongoDB than with something like MySQL.
You can shard RDBMS relatively easily -- basically you wind up pushing a part of your database structure into your clients, so your clients can decide which shard to use.
The cost, though, is that you wind up having a difficult time doing some things that MongoDB can't do. (For example: Renormalizing your database... Does that even mean anything for MongoDB?)
There's something to be said, of course, for simplifying your design. But it's probably a good idea also to make sure your design reflects your requirements.
Indeed you can distribute data to multiple independent RDBMS, but balancing when new nodes are added is probably a manual process (or a lot of custom code) that is likely to require downtime. To avoid downtime, your application would need to write to both chunks while it is balancing/migrating (and then delete the old data/chunks once it is migrated to a the RDBMS). Essentially, you would need to write what is already in MongoDB.
You would also have to write a parallel query engine.
I too am a fan of simple designs, but I think rolling your own sharding on top of a RDBMS would likely be a massive chunk of time.
There are really expensive commercial products working on horizontally scaling RDBMS... but personally, I prefer open source and document oriented databases :-)
In all this talk of locking were missing the real benefit to a heavy write environment, better yielding. this is going to be huge for those one off writes that in the past would have held the lock. Also I am looking forwards to playing with the new aggregation framework.
It is, it's next in fact, but it was decided to make sure DB level locking worked first (and gets the bugs shaken out of it) before moving on to the next level of locking. Dwight gives a decent description of the thinking in this presentation:
Not many do. There are quite a few things I build myself if I really really want to use it. A lot of the time it isn't worth it.
I would totally be using MongoDB for some projects (it fits the bill -perfectly-), but my resources for those are usually limited to Solaris on SPARC.
I do have an x86 desktop with Linux as my main PC at work, so I don't miss out on all the fun completely.
However, I do mostly sysadmin stuff, so a lot of the things I use come in the form of scripts, so it's often cross-platform.
Though I do come across some install scripts that just blatantly assume that everything is running Bash on Linux. Those are fun.
I have very little expectation that the MongoDB team will ever push a non x86 version. They do a lot of optimisations deep inside that rely on architecture specific things. But one can hope :)
Lots of big companies, mostly infrastructure like telecom or tv, use Solaris still. Unless the project aims to support those uses, it usually isn't worth their time.
I think you'd be quite surprised if you ever found out just how much of the world, including your very own existence, depends heavily on platforms and systems that you consider to be "dead".
I totally agree there are a lot of big apps running on Sparc... but I think that it would be much cheaper to buy new x86 hardware for a all products/systems going forward (i.e., those written for MongoDB). Sun (errr... Oracle) also sells x86 Solaris servers (a platform MongoDB supports).
The new DB level locking introduced in this release is a joke. There's not much difference between that and the old global write lock unless you split your database in dozens of smaller ones. What a pain. I wish they'd just stop pretending and addressed the issue properly once and for all.
I really want to like and use MongoDB because the way data is represented and how it can be queried is awesome.