Spark and Tez Highlight MapReduce Problems
2014-03-27 14:15
162 查看
On February 3rd, Cloudera announced
support for Apache Spark as part of Cloudera Enterprise. I’ve blogged
about Spark before so I won’t go into substantial detail here, but the short version is Spark improves upon MapReduce by removing the need to write data to disk between steps. Spark also takes advantage of in-memory processing and data sharing for further
optimizations.
The other successor to MapReduce (of course there is more than one) is Apache Tez. Tez improves
upon MapReduce by removing the need to write data to disk between steps (Sound familiar?). It also has in-memory capabilities similar to Spark. Thus far Hortonworks has thrown its weight behind Tez development as part of the
Stinger project.
Both Tez and Spark are described as supplementing MapReduce workloads. However, I don’t think this will be case much longer. The world has changed since Google published the original MapReduce paper in 2004. Memory prices have plummeted while data volumes and
sources have increased, making legacy MapReduce less appealing.
Vendors will likely begin distancing themselves from MapReduce for more performant options once there are some high profile customer references. It remains to be seen what this means for early adopters with legacy MapReduce applications.
Thanks to Josh Wills at Cloudera for helping clarify the advantage provided by Spark & Tez.
Ref: http://blogs.gartner.com/nick-heudecker/spark-tez-highlight-mapreduce-problems/
support for Apache Spark as part of Cloudera Enterprise. I’ve blogged
about Spark before so I won’t go into substantial detail here, but the short version is Spark improves upon MapReduce by removing the need to write data to disk between steps. Spark also takes advantage of in-memory processing and data sharing for further
optimizations.
The other successor to MapReduce (of course there is more than one) is Apache Tez. Tez improves
upon MapReduce by removing the need to write data to disk between steps (Sound familiar?). It also has in-memory capabilities similar to Spark. Thus far Hortonworks has thrown its weight behind Tez development as part of the
Stinger project.
Both Tez and Spark are described as supplementing MapReduce workloads. However, I don’t think this will be case much longer. The world has changed since Google published the original MapReduce paper in 2004. Memory prices have plummeted while data volumes and
sources have increased, making legacy MapReduce less appealing.
Vendors will likely begin distancing themselves from MapReduce for more performant options once there are some high profile customer references. It remains to be seen what this means for early adopters with legacy MapReduce applications.
Thanks to Josh Wills at Cloudera for helping clarify the advantage provided by Spark & Tez.
Ref: http://blogs.gartner.com/nick-heudecker/spark-tez-highlight-mapreduce-problems/
相关文章推荐
- Big Data, MapReduce, Hadoop, and Spark with Python
- MapReduce\Tez\Storm\Spark四个框架的异同
- MapReduce\Tez\Storm\Spark四个框架的异同
- MapReduce\Tez\Storm\Spark四个框架的异同
- mapreduce/spark/storm/Tez 框架
- Spark and Tez, out of phase
- MapReduce、Tez、Storm、Spark四个框架的异同
- 简单说说MapReduce, Tez, Spark
- MapReduce和Spark写入Hbase多表总结
- A summary: how to use bit manipulation to solve problems easily and efficiently
- Hadoop2.0/YARN深入浅出(Hadoop2.0、Spark、Storm和Tez)
- spark 2.1 Encoders and UploadBlock
- How to Plan and Configure YARN and MapReduce 2 in HDP 2.0 如何配置YARN
- 各种编程模型,hadoop,pig,tez,spark
- Multiple strings.xml files cause order and incomplete problems in android
- spark 2.1 TaskSetManager, Schedulable And Pool
- Jaw Crusher Operating Problems and Handling Method
- Symmetric NAT and It’s Problems
- hadoop and mapreduce.
- Spark vs. MapReduce 时间节约66%,计算节约40%