Spark运行各个时间段的解释
2015-10-03 17:25
295 查看
package org.apache.spark.ui
private[spark] object ToolTips {
val SCHEDULER_DELAY =
"""Scheduler delay includes time to ship the task from the scheduler to
the executor, and time to send the task result from the executor to the scheduler. If
scheduler delay is large, consider decreasing the size of tasks or decreasing the size
of task results."""
val TASK_DESERIALIZATION_TIME =
"""Time spent deserializing the task closure on the executor, including the time to read the
broadcasted task."""
val KSHUFFLE_READ_BLOCED_TIME =
"Time that the task spent blocked waiting for shuffle data to be read from remote machines."
val INPUT = "Bytes and records read from Hadoop or from Spark storage."
val OUTPUT = "Bytes and records written to Hadoop."
val STORAGE_MEMORY =
"Memory used / total available memory for storage of data " +
"like RDD partitions cached in memory. "
val SHUFFLE_WRITE =
"Bytes and records written to disk in order to be read by a shuffle in a future stage."
val SHUFFLE_READ =
"""Total shuffle bytes and records read (includes both data read locally and data read from
remote executors). """
val SHUFFLE_READ_REMOTE_SIZE =
"""Total shuffle bytes read from remote executors. This is a subset of the shuffle
read bytes; the remaining shuffle data is read locally. """
val GETTING_RESULT_TIME =
"""Time that the driver spends fetching task results from workers. If this is large, consider
decreasing the amount of data returned from each task."""
val RESULT_SERIALIZATION_TIME =
"""Time spent serializing the task result on the executor before sending it back to the
driver."""
val GC_TIME =
"""Time that the executor spent paused for Java garbage collection while the task was
running."""
val JOB_TIMELINE =
"""Shows when jobs started and ended and when executors joined or left. Drag to scroll.
Click Enable Zooming and use mouse wheel to zoom in/out."""
val STAGE_TIMELINE =
"""Shows when stages started and ended and when executors joined or left. Drag to scroll.
Click Enable Zooming and use mouse wheel to zoom in/out."""
val JOB_DAG =
"""Shows a graph of stages executed for this job, each of which can contain
multiple RDD operations (e.g. map() and filter()), and of RDDs inside each operation
(shown as dots)."""
val STAGE_DAG =
"""Shows a graph of RDD operations in this stage, and RDDs inside each one. A stage can run
multiple operations (e.g. two map() functions) if they can be pipelined. Some operations
also create multiple RDDs internally. Cached RDDs are shown in green.
"""
}
private[spark] object ToolTips {
val SCHEDULER_DELAY =
"""Scheduler delay includes time to ship the task from the scheduler to
the executor, and time to send the task result from the executor to the scheduler. If
scheduler delay is large, consider decreasing the size of tasks or decreasing the size
of task results."""
val TASK_DESERIALIZATION_TIME =
"""Time spent deserializing the task closure on the executor, including the time to read the
broadcasted task."""
val KSHUFFLE_READ_BLOCED_TIME =
"Time that the task spent blocked waiting for shuffle data to be read from remote machines."
val INPUT = "Bytes and records read from Hadoop or from Spark storage."
val OUTPUT = "Bytes and records written to Hadoop."
val STORAGE_MEMORY =
"Memory used / total available memory for storage of data " +
"like RDD partitions cached in memory. "
val SHUFFLE_WRITE =
"Bytes and records written to disk in order to be read by a shuffle in a future stage."
val SHUFFLE_READ =
"""Total shuffle bytes and records read (includes both data read locally and data read from
remote executors). """
val SHUFFLE_READ_REMOTE_SIZE =
"""Total shuffle bytes read from remote executors. This is a subset of the shuffle
read bytes; the remaining shuffle data is read locally. """
val GETTING_RESULT_TIME =
"""Time that the driver spends fetching task results from workers. If this is large, consider
decreasing the amount of data returned from each task."""
val RESULT_SERIALIZATION_TIME =
"""Time spent serializing the task result on the executor before sending it back to the
driver."""
val GC_TIME =
"""Time that the executor spent paused for Java garbage collection while the task was
running."""
val JOB_TIMELINE =
"""Shows when jobs started and ended and when executors joined or left. Drag to scroll.
Click Enable Zooming and use mouse wheel to zoom in/out."""
val STAGE_TIMELINE =
"""Shows when stages started and ended and when executors joined or left. Drag to scroll.
Click Enable Zooming and use mouse wheel to zoom in/out."""
val JOB_DAG =
"""Shows a graph of stages executed for this job, each of which can contain
multiple RDD operations (e.g. map() and filter()), and of RDDs inside each operation
(shown as dots)."""
val STAGE_DAG =
"""Shows a graph of RDD operations in this stage, and RDDs inside each one. A stage can run
multiple operations (e.g. two map() functions) if they can be pipelined. Some operations
also create multiple RDDs internally. Cached RDDs are shown in green.
"""
}
相关文章推荐
- java7 invokedynamic学习笔记
- prefast 使用
- Servlet入门之HelloWorld
- 1007. Maximum Subsequence Sum (25)
- Spark技术内幕:Stage划分及提交源码分析
- 第4周项目5-循环双链表应用
- thinkphp中的参数绑定
- leetcode Construct Binary Tree from Inorder and Postorder Traversal
- URL::to() 与 URL::toRoute()
- C++学习笔记之RTTI(运行时类型识别)
- Java中单链表的实现
- 【设计模式】里氏代换
- andriod开发点滴(使用Bundle,Intent在Activity间传递map)
- mysql limit
- 1006. Sign In and Sign Out (25)
- 2015长春网络赛hdu5444Elven Postman搜索二叉树
- 大数据系统梳理
- [前端]JS时钟实例
- 2.Python深入_上下文管理器
- HDU5493 Queue【线段树】