MATLAB下高效使用内存
2016-04-10 00:00
393 查看
摘要: 高效使用内存 MATLAB
Strategies for Efficient Use of Memory - MATLAB & Simulink - MathWorks 中国
Strategies for Efficient Use of Memory
MAT-Files. Load part of a variable by indexing into an object that you create with the
Text Files. Use the
Binary Files. You can use low-level binary file I/O functions, such as
Image, HDF, Audio, and Video Files. Many of the MATLAB functions that support loading from these types of files allow you to select portions of the data to read. For details, see the function reference pages listed in Supported File Formats for Import and Export.
For a subset of applications that you can break into separate chunks and process independently.
For applications that only rely on the state of a previous block, such as filtering.
use just the one command to do both operations:
Using the
One way to use less memory in this situation is to use nested functions. A nested function shares the workspace of all outer functions, giving the nested function access to data outside of its usual scope. In the example shown here, nested function
If you just need to carry out simple arithmetic and you represent the original data as integers, you can use the integer classes in MATLAB. The following is a list of numeric classes, memory requirements (in bytes), and the supported operations.
Because simple numeric arrays (comprising one
Cell arrays are comprised of separate
Structures require a similar amount of overhead per field (see Array Headers). Structures with many fields and small contents have a large overhead and should be avoided. A large array of structures with numeric scalar fields requires much more memory than a structure with fields containing large numeric arrays.
Also note that while MATLAB stores numeric arrays in contiguous memory, this is not the case for structures and cell arrays.
You can see that this array requires only approximately 4 KB to be stored as sparse, but approximately 8 MB as a full matrix. In general, for a sparse double array with
16 *
12 *
Note that MATLAB does not support all mathematical operations on sparse arrays.
To make more efficient use of your memory, preallocate a block of memory large enough to hold the matrix at its final size before entering the loop. When you preallocate memory for an array, MATLAB reserves sufficient contiguous space for the entire full-size array at the beginning of the computation. Once you have this space, you can add elements to the array without having to continually allocate new space for it in memory.
For more information on preallocation, see Preallocation.
The following statements can require approximately 4.3 MB of RAM. This is because MATLAB might not be able to reuse the space previously occupied by two 1 MB arrays when allocating space for a 2.3 MB array:
The simplest way to prevent overallocation of memory is to allocate the largest vectors first. These statements require only about 2.0 MB of RAM:
The
Strategies for Efficient Use of Memory - MATLAB & Simulink - MathWorks 中国
Strategies for Efficient Use of Memory
Ways to Reduce the Amount of Memory Required
The source of many "out of memory" problems often involves analyzing or processing an existing large set of data such as in a file or a database. This requires bringing all or part of the data set into the MATLAB® software process. The following techniques deal with minimizing the required memory during this stage.Load Only As Much Data As You Need
Only import into MATLAB as much of a large data set as you need for the problem you are trying to solve. This is not usually a problem when importing from sources such as a database, where you can explicitly search for elements matching a query. But this is a common problem with loading large flat text or binary files. Rather than loading the entire file, use the appropriate MATLAB function to load parts of files.MAT-Files. Load part of a variable by indexing into an object that you create with the
matfilefunction.
Text Files. Use the
textscanfunction to access parts of a large text file by reading only the selected columns and rows. If you specify the number of rows or a repeat format number with
textscan, MATLAB calculates the exact amount of memory required beforehand.
Binary Files. You can use low-level binary file I/O functions, such as
fread, to access parts of any file that has a known format. For binary files of an unknown format, try using memory mapping with the
memmapfilefunction.
Image, HDF, Audio, and Video Files. Many of the MATLAB functions that support loading from these types of files allow you to select portions of the data to read. For details, see the function reference pages listed in Supported File Formats for Import and Export.
Process Data By Blocks
Consider block processing, that is, processing a large data set one section at a time in a loop. Reducing the size of the largest array in a data set reduces the size of any copies or temporaries needed. You can use this technique in either of two ways:For a subset of applications that you can break into separate chunks and process independently.
For applications that only rely on the state of a previous block, such as filtering.
Avoid Creating Temporary Arrays
Avoid creating large temporary variables, and also make it a practice to clear those temporary variables you do use when they are no longer needed. For example, when you create a large array of zeros, instead of saving to a temporary variableA, and then converting
Ato a single:
A = zeros(1e6,1); As = single(A);
use just the one command to do both operations:
A = zeros(1e6,1,'single');
Using the
repmatfunction, array preallocation and
forloops are other ways to work on
nondoubledata without requiring temporary storage in memory.
Use Nested Functions to Pass Fewer Arguments
When working with large data sets, be aware that MATLAB makes a temporary copy of an input variable if the called function modifies its value. This temporarily doubles the memory required to store the array, which causes MATLAB to generate an error if sufficient memory is not available.One way to use less memory in this situation is to use nested functions. A nested function shares the workspace of all outer functions, giving the nested function access to data outside of its usual scope. In the example shown here, nested function
setrowvalhas direct access to the workspace of the outer function
myfun, making it unnecessary to pass a copy of the variable in the function call. When
setrowvalmodifies the value of
A, it modifies it in the workspace of the calling function. There is no need to use additional memory to hold a separate array for the function being called, and there also is no need to return the modified value of
A:
function myfun A = magic(500); function setrowval(row, value) A(row,:) = value; end setrowval(400, 0); disp('The new value of A(399:401,1:10) is') A(399:401,1:10) end
Using Appropriate Data Storage
MATLAB provides you with different sizes of data classes, such asdoubleand
uint8, so you do not need to use large classes to store your smaller segments of data. For example, it takes 7 KB less memory to store 1,000 small unsigned integer values using the
uint8class than it does with
double.
Use the Appropriate Numeric Class
The numeric class you should use in MATLAB depends on your intended actions. The default classdoublegives the best precision, but requires 8 bytes per element of memory to store. If you intend to perform complicated math such as linear algebra, you must use a floating-point class such as a
doubleor
single. The
singleclass requires only 4 bytes. There are some limitations on what you can do with the
singleclass, but most MATLAB Math operations are supported.
If you just need to carry out simple arithmetic and you represent the original data as integers, you can use the integer classes in MATLAB. The following is a list of numeric classes, memory requirements (in bytes), and the supported operations.
Class (Data Type) | Bytes | Supported Operations |
---|---|---|
single | 4 | Most math |
double | 8 | All math |
logical | 1 | Logical/conditional operations |
int8, uint8 | 1 | Arithmetic and some simple functions |
int16, uint16 | 2 | Arithmetic and some simple functions |
int32, uint32 | 4 | Arithmetic and some simple functions |
int64, int64 | 8 | Arithmetic and some simple functions |
Reduce the Amount of Overhead When Storing Data
MATLAB arrays (implemented internally asmxArrays) require room to store meta information about the data in memory, such as type, dimensions, and attributes. This takes about 80 bytes per array. This overhead only becomes an issue when you have a large number (e.g., hundreds or thousands) of small
mxArrays(e.g., scalars). The
whoscommand lists the memory used by variables, but does not include this overhead.
Because simple numeric arrays (comprising one
mxArray) have the least overhead, you should use them wherever possible. When data is too complex to store in a simple array (or matrix), you can use other data structures.
Cell arrays are comprised of separate
mxArraysfor each element. As a result, cell arrays with many small elements have a large overhead.
Structures require a similar amount of overhead per field (see Array Headers). Structures with many fields and small contents have a large overhead and should be avoided. A large array of structures with numeric scalar fields requires much more memory than a structure with fields containing large numeric arrays.
Also note that while MATLAB stores numeric arrays in contiguous memory, this is not the case for structures and cell arrays.
Import Data to the Appropriate MATLAB Class
When reading data from a binary file withfread, it is a common error to specify only the class of the data in the file, and not the class of the data MATLAB uses once it is in the workspace. As a result, the default
doubleis used even if you are reading only 8-bit values. For example,
fid = fopen('large_file_of_uint8s.bin', 'r'); a = fread(fid, 1e3, 'uint8'); % Requires 8k whos a Name Size Bytes Class Attributes a 1000x1 8000 double a = fread(fid, 1e3, 'uint8=>uint8'); % Requires 1k whos a Name Size Bytes Class Attributes a 1000x1 1000 uint8
Make Arrays Sparse When Possible
If your data contains many zeros, consider using sparse arrays, which store only nonzero elements. The following example compares the space required for storage of an array of mainly zeros:A = eye(1000); % Full matrix with ones on the diagonal As = sparse(A); % Sparse matrix with only nonzero elements whos Name Size Bytes Class Attributes A 1000x1000 8000000 double As 1000x1000 24008 double sparse
You can see that this array requires only approximately 4 KB to be stored as sparse, but approximately 8 MB as a full matrix. In general, for a sparse double array with
nnznonzero elements and
ncolcolumns, the memory required is
16 *
nnz+ 8 *
ncol+ 8 bytes (on a 64-bit machine)
12 *
nnz+ 4 *
ncol+ 4 bytes (on a 32-bit machine)
Note that MATLAB does not support all mathematical operations on sparse arrays.
How to Avoid Fragmenting Memory
MATLAB always uses a contiguous segment of memory to store a numeric array. As you manipulate this data, however, the contiguous block can become fragmented. When memory is fragmented, there might be plenty of free space, but not enough contiguous memory to store a new large variable. Increasing fragmentation can use significantly more memory than is necessary.Preallocate Contiguous Memory When Creating Arrays
In the course of a MATLAB session, memory can become fragmented due to dynamic memory allocation and deallocation.forand
whileloops that incrementally increase, or grow, the size of a data structure each time through the loop can add to this fragmentation as they have to repeatedly find and allocate larger blocks of memory to store the data.
To make more efficient use of your memory, preallocate a block of memory large enough to hold the matrix at its final size before entering the loop. When you preallocate memory for an array, MATLAB reserves sufficient contiguous space for the entire full-size array at the beginning of the computation. Once you have this space, you can add elements to the array without having to continually allocate new space for it in memory.
For more information on preallocation, see Preallocation.
Allocate Your Larger Arrays First
MATLAB uses a heap method of memory management. It requests memory from the operating system when there is not enough memory available in the heap to store the current variables. It reuses memory as long as the size of the memory segment required is available in the heap.The following statements can require approximately 4.3 MB of RAM. This is because MATLAB might not be able to reuse the space previously occupied by two 1 MB arrays when allocating space for a 2.3 MB array:
a = rand(1e6,1); b = rand(1e6,1); clear c = rand(2.3e6,1);
The simplest way to prevent overallocation of memory is to allocate the largest vectors first. These statements require only about 2.0 MB of RAM:
c = rand(2.3e6,1); clear a = rand(1e6,1); b = rand(1e6,1);
Long-Term Usage (Windows Systems Only)
On 32-bit Microsoft® Windows®, the workspace of MATLAB can fragment over time due to the fact that the Windows memory manager does not return blocks of certain types and sizes to the operating system. Clearing the MATLAB workspace does not fix this problem. You can minimize the problem by allocating the largest variables first. This cannot address, however, the eventual fragmentation of the workspace that occurs from continual use of MATLAB over many days and weeks, for example. The only solution to this is to save your work and restart MATLAB.The
packcommand, which saves all variables to disk and loads them back, does not help with this situation.
Reclaiming Used Memory
One simple way to increase the amount of memory you have available is to clear large arrays that you no longer use.Save Your Large Data Periodically to Disk
If your program generates very large amounts of data, consider writing the data to disk periodically. After saving that portion of the data, use theclearfunction to remove the variable from memory and continue with the data generation.
Clear Old Variables from Memory When No Longer Needed
When you are working with a very large data set repeatedly or interactively, clear the old variable first to make space for the new variable. Otherwise, MATLAB requires temporary storage of equal size before overriding the variable. For example,a = rand(100e6,1) % 800 MB array b = rand(100e6,1) % New 800 MB array Error using rand Out of memory. Type HELP MEMORY for your options. clear a a = rand(100e6,1) % New 800 MB array
相关文章推荐
- 解析在main函数之前调用函数以及对设计的作用详解
- 详解Matlab中 sort 函数用法
- java和matlab画多边形闭合折线图示例讲解
- C#调用Matlab生成的dll方法的详细说明
- 简述Matlab中size()函数的用法
- 从java中调用matlab详细介绍
- 稀疏自动编码器 (Sparse Autoencoder)
- 详解Matlab中 sort 函数用法
- 简述Matlab中size()函数的用法
- VC++与Matlab混合编程的快速实现
- Matlab 矩阵运算
- matlab与opencv部分函数的对照
- matlab神经网络工具箱创建神经网络
- Matlab
- MATLAB 入门教程
- matlab函数_连通区域
- MATLAB中函数模式和命令模式的区别
- MATLAB 添加自定义的模块到simulink库浏览器
- Export Figures for LaTeX Writing