spark core 2.0 LongArray
2017-01-22 10:58
134 查看
LongArray is an array of long values. Compared with native JVM arrays, this:
1. supports using both in-heap and off-heap memory.
2. has no bound checking, and thus can crash the JVM process when assert is turned off.
The main fields and constructor of LongArray
// This is a long so that we perform long multiplications when computing offsets.
private static final long WIDTH = 8;
private final MemoryBlock memory;
private final Object baseObj;
private final long baseOffset;
private final long length;
public LongArray(MemoryBlock memory) {
assert memory.size() < (long) Integer.MAX_VALUE * 8: "Array size > 4 billion elements";
this.memory = memory;
this.baseObj = memory.getBaseObject();
this.baseOffset = memory.getBaseOffset();
this.length = memory.size() / WIDTH;
}
The set and get method, note that the position the combination of baseOffset and index * WIDTH.
/**
* Sets the value at position {@code index}.
*/
public void set(int index, long value) {
assert index >= 0 : "index (" + index + ") should >= 0";
assert index < length : "index (" + index + ") should < length (" + length + ")";
Platform.putLong(baseObj, baseOffset + index * WIDTH, value);
}
/**
* Returns the value at position {@code index}.
*/
public long get(int index) {
assert index >= 0 : "index (" + index + ") should >= 0";
assert index < length : "index (" + index + ") should < length (" + length + ")";
return Platform.getLong(baseObj, baseOffset + index * WIDTH);
}
1. supports using both in-heap and off-heap memory.
2. has no bound checking, and thus can crash the JVM process when assert is turned off.
The main fields and constructor of LongArray
// This is a long so that we perform long multiplications when computing offsets.
private static final long WIDTH = 8;
private final MemoryBlock memory;
private final Object baseObj;
private final long baseOffset;
private final long length;
public LongArray(MemoryBlock memory) {
assert memory.size() < (long) Integer.MAX_VALUE * 8: "Array size > 4 billion elements";
this.memory = memory;
this.baseObj = memory.getBaseObject();
this.baseOffset = memory.getBaseOffset();
this.length = memory.size() / WIDTH;
}
The set and get method, note that the position the combination of baseOffset and index * WIDTH.
/**
* Sets the value at position {@code index}.
*/
public void set(int index, long value) {
assert index >= 0 : "index (" + index + ") should >= 0";
assert index < length : "index (" + index + ") should < length (" + length + ")";
Platform.putLong(baseObj, baseOffset + index * WIDTH, value);
}
/**
* Returns the value at position {@code index}.
*/
public long get(int index) {
assert index >= 0 : "index (" + index + ") should >= 0";
assert index < length : "index (" + index + ") should < length (" + length + ")";
return Platform.getLong(baseObj, baseOffset + index * WIDTH);
}
相关文章推荐
- spark core 2.0 SortShuffleManager
- spark core 2.0 Executor Heartbeat
- spark core 2.0 MetricsConfig
- spark core 2.0 DiskBlockObjectWriter
- spark core 2.0 MemoryLocation
- spark core 2.0 ChunkedByteBufferOutputStream
- spark core 2.0 MemoryBlock Soruce Code Analysis
- spark core 2.0 TaskSchedulerImpl 源代码解析
- spark core 2.0 RedirectableOutputStream
- spark core 2.0 MemoryManager
- spark core 2.0 TimeTrackingOutputStream
- spark core 2.0 StorageMemoryPool
- spark core 2.0 PartitionCoalescer, PartitionGroup, DefaultPartitionCoalescer
- spark core 2.0 ExecutionMemoryPool
- spark core 2.0 BlockManager dropFromMemory
- spark core 2.0 Executor
- spark core 2.0 BypassMergeSortShuffleWriter
- spark core 2.0 UnifiedMemoryManager
- spark core 2.0 CheckpointState RDDCheckpointData Checkpoint LocalRDDCheckpointData
- spark core 2.0 CoarseGrainedSchedulerBackend SchedulerBackend ExecutorAllocationClient 源代码解析