spark core 2.0 Executor
2017-01-10 10:33
197 查看
Executor如果在非本地运行,则设置非捕获异常的处理。如果任何线程因为未捕获异常结束,则把整个执行器进程杀死,避免奇怪的情况。
if (!isLocal) { // Setup an uncaught exception handler for non-local mode. // Make any thread terminations due to uncaught exceptions kill the entire // executor process to avoid surprising stalls. Thread.setDefaultUncaughtExceptionHandler(SparkUncaughtExceptionHandler) }
/** * The default uncaught exception handler for Executors terminates the whole process, to avoid * getting into a bad state indefinitely. Since Executors are relatively lightweight, it's better * to fail fast when things go wrong. */ private[spark] object SparkUncaughtExceptionHandler extends Thread.UncaughtExceptionHandler with Logging { override def uncaughtException(thread: Thread, exception: Throwable) { try { // Make it explicit that uncaught exceptions are thrown when container is shutting down. // It will help users when they analyze the executor logs val inShutdownMsg = if (ShutdownHookManager.inShutdown()) "[Container in shutdown] " else "" val errMsg = "Uncaught exception in thread " logError(inShutdownMsg + errMsg + thread, exception) // We may have been called from a shutdown hook. If so, we must not call System.exit(). // (If we do, we will deadlock.) if (!ShutdownHookManager.inShutdown()) { if (exception.isInstanceOf[OutOfMemoryError]) { System.exit(SparkExitCode.OOM) } else { System.exit(SparkExitCode.UNCAUGHT_EXCEPTION) } } } catch { case oom: OutOfMemoryError => Runtime.getRuntime.halt(SparkExitCode.OOM) case t: Throwable => Runtime.getRuntime.halt(SparkExitCode.UNCAUGHT_EXCEPTION_TWICE) } } def uncaughtException(exception: Throwable) { uncaughtException(Thread.currentThread(), exception) } }
相关文章推荐
- spark core 2.0 CoarseGrainedSchedulerBackend SchedulerBackend ExecutorAllocationClient 源代码解析
- spark core 2.0 UnifiedMemoryManager
- spark core 2.0 BypassMergeSortShuffleWriter
- spark core 2.0 PartitionCoalescer, PartitionGroup, DefaultPartitionCoalescer
- spark core 2.0 ChunkedByteBufferOutputStream
- spark core 2.0 StorageMemoryPool
- spark core 2.0 RedirectableOutputStream
- spark core 2.0 SortShuffleManager
- spark core 2.0 CheckpointState RDDCheckpointData Checkpoint LocalRDDCheckpointData
- spark core 2.0 ExecutorSource Metrics统计
- spark core 2.0 ExecutionMemoryPool
- spark core 2.0 Partition and HadoopPartition
- spark core 2.0 TaskSchedulerImpl 源代码解析
- spark core 2.0 Executor ClassLoader
- spark core 2.0 MemoryManager
- spark core 2.0 DiskBlockObjectWriter
- spark core 2.0 Executor Heartbeat
- spark core 2.0 TimeTrackingOutputStream
- spark core 2.0 MemoryLocation
- spark core 2.0 MemoryBlock Soruce Code Analysis