This article is a good overview-- one of the most informed I've seen from the HPC side of the world.
I don't think "HDFS is very slow and very obtuse" is true, though. If you use HDFS for what it was intended for-- reading large files that are mostly local-- it does pretty well. MapReduce and Impala explicitly support this paradigm by looking at where files are and scheduling computation there. If your workload has a ton of metadata operations, this can become a bottleneck, but that is not unique to HDFS. For example, if you use "ls" with the wrong options on Lustre, things can get quite slow due to the large number of metadata operations generated: https://wikis.nyu.edu/display/NYUHPC/Lustre+FAQ
Lustre is a cool filesystem in many ways, and has been battle-hardened for HPC. However, I think Lustre is relatively unlikely to gain ground in Hadoop due to the complexity of administering and upgrading its in-kernel server. Ceph might have a chance, though-- we'll see.
The article also doesn't discuss a lot of the areas in which Hadoop is way ahead. Hadoop handles faults in individual nodes, rather than just assuming we will use gold-plated hardware with infalliable RAID. Node failures happen, and HDFS and Hadoop can recover-- even from NameNode failures, these days. Google's Spanner paper points the way towards having strong consistency without sacrificing (as much) availability.
I agree that cramming traditional HPC problems into the Map-Reduce framework is not really a great idea. Instead, it would be better to use something like Spark (which, to his credit, Lockwood mentions). Spark is getting a huge amount of attention right now, and I think it's going to resolve some of these issues.
I don't think "HDFS is very slow and very obtuse" is true, though. If you use HDFS for what it was intended for-- reading large files that are mostly local-- it does pretty well. MapReduce and Impala explicitly support this paradigm by looking at where files are and scheduling computation there. If your workload has a ton of metadata operations, this can become a bottleneck, but that is not unique to HDFS. For example, if you use "ls" with the wrong options on Lustre, things can get quite slow due to the large number of metadata operations generated: https://wikis.nyu.edu/display/NYUHPC/Lustre+FAQ
Lustre is a cool filesystem in many ways, and has been battle-hardened for HPC. However, I think Lustre is relatively unlikely to gain ground in Hadoop due to the complexity of administering and upgrading its in-kernel server. Ceph might have a chance, though-- we'll see.
The article also doesn't discuss a lot of the areas in which Hadoop is way ahead. Hadoop handles faults in individual nodes, rather than just assuming we will use gold-plated hardware with infalliable RAID. Node failures happen, and HDFS and Hadoop can recover-- even from NameNode failures, these days. Google's Spanner paper points the way towards having strong consistency without sacrificing (as much) availability.
I agree that cramming traditional HPC problems into the Map-Reduce framework is not really a great idea. Instead, it would be better to use something like Spark (which, to his credit, Lockwood mentions). Spark is getting a huge amount of attention right now, and I think it's going to resolve some of these issues.