Storm 配置图文解析

Storm 配置图文解析

参考阅读：http://www.xiaofateng.com/?p=959

==============================
|     sample-topology        |
|  ------------------------  |   Task 1           Task 2          Task 3
|  |   Worker Process 1   |  |     T1               T2              T3
|  |           +--------+ |  |    Spout     =>     Bolt      =>    Bolt
|  | +------+  | +----+ | |  | parallelism      parallelism     parallelism
|  | |  T3  |  | | T2 | | |  |    hint=2          hint=2          hint=6
|  | +------+  | +----+ | |  |
|  |           | +----+ | |  |    combined parallelism = 2 + 2 + 6 = 10
|  | +------+  | | T2 | | |  |
|  | |  T3  |  | +----+ | |  |  Each of the 2 worker processes will spawn 10/2=5 threads
|  | +------+  +--------+ |  |
|  |                      |  |    T1: parallelism hint = initial executors
|  | +------+  +--------+ |  |
|  | |  T3  |  |   T1   | |  |    T2: the T2 bolt was configured to use 2 executors and four tasks.
|  | +------+  +--------+ |  |        For this reason each executor runs two tasks for this bolt.
|  ------------------------  |
|                            |
|  ------------------------  |    Config conf = new Config();
|  |   Worker Process 2   |  |
|  |           +--------+ |  |    // run as 2 workers on 2 supervisors
|  | +------+  | +----+ | |  |    conf.setNumWorkers(2);
|  | |  T3  |  | | T2 | | |  |
|  | +------+  | +----+ | |  |    // T1: 2 executors for spout
|  |           | +----+ | |  |    topologyBuilder.setSpout("T1-spout", new T1Spout(), 2);
|  | +------+  | | T2 | | |  |
|  | |  T3  |  | +----+ | |  |    // T2: 2 executors for bolt with total 4 tasks
|  | +------+  +--------+ |  |    topologyBuilder.setBolt("T2-bolt", new T2Bolt(), 2)
|  |                      |  |        .setNumTasks(4).shuffleGrouping("T1-spout");
|  | +------+  +--------+ |  |    // T3: 6 executors for bolt (default 1 task for 1 executor)
|  | |  T3  |  |   T1   | |  |    topologyBuilder.setBolt("T3-bolt", new T3Bolt(), 6).shuffleGrouping("T2-bolt");
|  | +------+  +--------+ |  |
|  ------------------------  |    StormSubmitter.submitTopology("sample-topology", conf, topologyBuilder.createTopology());
==============================

说明：

一个worker进程(process)会产生N个线程(executor)，那么并行度(parallelism)就是所有线程的数目。setNumWorkers

任务(task)是线程执行的工作队列，线程的任务数说明线程的吞吐能力。一个线程的各个任务之间并不是并发的。setNumTasks

线程(executor)是执行任务的上下文环境。

参照上图理解各个配置的含义。