您的位置：首页 > 移动开发 > Android开发

如何在Android中定义容量巨大的数组

2015-06-16 14:49 591 查看

原文链接：点击打开链接

点击打开链接

点击打开链接

避免原文打不开，记录如下：

背景：

本人因为某Android项目需要，需要在Android中定义一个容量为2万的float数组。这两万个float值已经存在某个文件中。

方法：

1.直接在Java文件里定义？

Java单个函数的长度限制为65535字节（不确定具体数值，但肯定是有限制的）。故在Java文件中直接定义如此之长的数组是行不通的。

2.BufferedReader或是Scanner？

可以创建一个文件，每行存放一个float值。通过BufferedReader.nextLine()读取每一行的字符，用Float.parseFloat(String)来将读到的字符转换成float值，最后将float值存取至数组里。也可以通过Scanner.nextFloat()这样的方法来读取文件中存放的float值。这个时候，文件里并不需要将每个float值放在单独的一行里。但是不同的float值至少需要以某些符号隔开（例如，空格）。

经过对比，通过BufferedReader读取2W个float值比较快，使用Scanner会占用大量内存，并且不断调用GC，导致线程暂停。但是，当要读取的文件很大的时候，即使使用BufferedReader也会出现问题：在低端的机子上运行时，Logcat会出现OutOfMemory的错误提示。

3.将2万个float值写入文件后做Java的内存映射。

Java内存映射网上的文章不多，可以参考：http://www.javacodegeeks.com/2013/05/power-of-java-memorymapped-file.html和http://www.studytrails.com/java-io/Channels.jsp。这种方法，需要先将所有float值以byte的形式写入到文件中。接着，使用FileChannel将文件中的byte映射到内存中，生成FloatBuffer。最后在FloatBuffer中读取float值存至float数组中。在通过经测试，这种方法比BufferedReader快很多。但是如果算上写文件的时间的话，不如BufferedReader来得方便。

大家可以借鉴一下我测试对比所用的源码。第一个部分，我使用Scanner；第二个部分，我使用BufferedReader；第三部分，我将第二部分读取到的float值以byte形式存取至文件中；第四部分，我使用MappedByteBuffer将文件中的内容映射至内存，并生成新的float数组。

源码下载地址：https://db.tt/9jCbO653

以下是结果运行截图：

当然，以上测试程序均在PCjava端运行。真正要移植到Android上进行内存映射，还需要：

1.由于从assets里读文件只能获取InputStream，而调用FileChannel.map()必须从FileInputStream中获得，故需要将assets里的内存映射文件转移至sd卡上后再从sd卡上读取它们。

2.在测试文件中，我使用了DataOutputStream.writeFloat()将float数组写入内存映射文件。该方法写入遵循BIG_ENDIAN规则。故当进行内存映射的时候，很有可能是以LITTLE_ENDIAN的方法读取的（这还要看你是如何实现内存映射的）。

以下这段代码，给出了如何将一个存好的float数组（cubeTextureCoordinateData）以LITTLE_ENDIAN的方式写入内存映射文件：

/*WritingMe
20000
moryMapRawData*/

ByteBufferlittleEndian=ByteBuffer.allocateDirect(11160*4).order(ByteOrder.LITTLE_ENDIAN);

i=0;

while(i<11160){

littleEndian.putFloat(cubeTextureCoordinateData[i++]);

}

littleEndian.flip();

try{

FileOutputStreamfos=newFileOutputStream("memorymap_texture_little_endian");

FileChannelfc=fos.getChannel();

fc.write(littleEndian);

fc.close();

fos.close();

}catch(IOExceptione){

e.printStackTrace();

}

PowerofJavaMemoryMappedFile

InJDK1.4aninterestingfeatureofMemorymappedfilewasaddedtoJava,whichallowstomapanyfiletoOSmemoryforefficientreading.AmemorymappedfilecanbeusedtodevelopanIPCtype
ofsolution.ThisarticleisanexperimentwithmemorymappedfiletocreateIPC.

SomedetailsaboutMemoryMappedFile,definitionfromWIKI

Amemory-mappedfileisasegmentofvirtualmemorywhichhasbeenassignedadirectbyte-for-bytecorrelationwithsomeportionofafileorfile-likeresource.Thisresourceistypicallyafilethatisphysicallypresenton-disk,butcanalsobeadevice,
sharedmemoryobject,orotherresourcethattheoperatingsystemcanreferencethroughafiledescriptor.Oncepresent,thiscorrelationbetweenthefileandthememoryspacepermitsapplicationstotreatthemappedportionasifitwereprimarymemory.

SampleProgram

BelowwehavetwoJavaprograms,oneisawriterandtheotherisareader.ThewriteristheproducerandtriestowritetoMemoryMappedfile,thereaderistheconsumeranditreadsmessagesfromthememorymappedfile.Thisisjustasampleprogramtoshow
youtheidea,itdoes’thandlemanyedgecasesbutitisgoodenoughtobuildsomethingontopofamemorymappedfile.

MemoryMapWriter

01	import java.io.File;

02	import java.io.FileNotFoundException;

03	import java.io.IOException;

04	import java.io.RandomAccessFile;

05	import java.nio.MappedByteBuffer;

06	import java.nio.channels.FileChannel;

08	public class MemoryMapWriter {

10	public static void main(String[] args) throws FileNotFoundException, IOException,InterruptedException{

11	File f= new File( "c:/tmp/mapped.txt" );

12	f.delete();

14	FileChannel fc= new RandomAccessFile(f, "rw" ).getChannel();

16	long bufferSize= 8 * 1000 ;

17	MappedByteBuffer mem=fc.map(FileChannel.MapMode.READ_WRITE, 0 , bufferSize);

19	int start = 0 ;

20	long counter= 1 ;

21	long HUNDREDK= 100000 ;

22	long startT =System.currentTimeMillis();

23	long noOfMessage =HUNDREDK* 10 * 10 ;

for

(;;)

26	if (!mem.hasRemaining())

28	start+=mem.position();

29	mem =fc.map(FileChannel.MapMode.READ_WRITE,start,bufferSize);

31	mem.putLong(counter);

32	counter++;

33	if (counter >noOfMessage)

break

36	long endT =System.currentTimeMillis();

37	long tot =endT-startT;

38	System.out.println(String.format( "No OfMessage%s,Time(ms)%s" ,noOfMessage, tot));

MemoryMapReader

01	import java.io.File;

02	import java.io.FileNotFoundException;

03	import java.io.IOException;

04	import java.io.RandomAccessFile;

05	import java.nio.MappedByteBuffer;

06	import java.nio.channels.FileChannel;

08	public class MemoryMapReader {

/**

11	* @paramargs

12	* @throwsIOException

13	* @throwsFileNotFoundException

14	* @throwsInterruptedException

*/

16	public static void main(String[] args) throws FileNotFoundException, IOException,InterruptedException{

18	FileChannel fc= new RandomAccessFile( new File( "c:/tmp/mapped.txt" ), "rw" ).getChannel();

20	long bufferSize= 8 * 1000 ;

21	MappedByteBuffer mem=fc.map(FileChannel.MapMode.READ_ONLY, 0 , bufferSize);

22	long oldSize=fc.size();

24	long currentPos = 0 ;

25	long xx=currentPos;

27	long startTime =System.currentTimeMillis();

28	long lastValue=- 1 ;

for

(;;)

32	while (mem.hasRemaining())

34	lastValue=mem.getLong();

35	currentPos += 8 ;

37	if (currentPos <oldSize)

40	xx =xx+mem.position();

41	mem =fc.map(FileChannel.MapMode.READ_ONLY,xx,bufferSize);

42	continue ;

else

46	long end =System.currentTimeMillis();

47	long tot =end-startTime;

48	System.out.println(String.format( "Last ValueRead%s,Time(ms)%s" ,lastValue, tot));

49	System.out.println( "Waiting formessage" );

50	while ( true )

52	long newSize=fc.size();

53	if (newSize>oldSize)

55	oldSize =newSize;

56	xx =xx+mem.position();

57	mem =fc.map(FileChannel.MapMode.READ_ONLY,xx,oldSize-xx);

58	System.out.println( "Got somedata" );

break

Observation

UsingamemorymappedfilecanbeaverygoodoptionfordevelopingInterProcesscommunication,throughputisalsoreasonablywellforbothproduce&consumer.Performancestatsbyrunproducerandconsumertogether:

Eachmessageisonelongnumber

Produce–10Millionmessage–16(s)

Consumer–10Millionmessage0.6(s)

Averysimplemessageisusedtoshowyoutheidea,butitcanbeanytypeofcomplexmessage,butwhenthereiscomplexdatastructurethenserializationcanaddtooverhead.Therearemanytechniquestogetoverthatoverhead.Moreinnextblog.

JavaNIO-ChannelsandMemorymapping

ChannelClasses

Channel-Achannelrepresentsaconnectiontoafile,socketoranyothercomponentthatcanperformIOoperations.Achannelreadsorwritesdatatoabytebuffer.Channelsaregenerallythreadsafe.Achannelmaybespecified
tobereadonlyorbothreadableandwritable.
ReadableByteChannel-AReadableByteChannelpermitsreadoperationonabuffer,allowingonlyonethreadtoreadatatime.
WritableByteChannel-AWritableByteChannelpermitswriteoperationonabuffer,allowingonlyonethreadtoreadatatime.
ByteChannel-AByteChannelextendsbothReadableByteChannelandWritableByteChannelandallowsbothreadandwrite.
SeekableByteChannel-ASeekableByteChannelextendsByteChannelandallowstomaintainandmodifythecurrentpositionontheunderlyingentitytowhichitisconnected.Ithasmethodstogetthesizeoftheunderlyingentity
ortruncateittoagivensize,ifpermitted.
GatheringByteChannel-AGatheringByteChannelisusedtowritedatafrommultiplebuffersintoasinglechannel.
ScatteringByteChannel-AScatteringByteChannelisusedtoreaddatafromachannelintomultiplebuffers.

FileChannel

AFileChannelisusedtoreadandwritedatatoafile.ItimplementsseekableByteChannel,ScatteringByteChannelandGatheringByteChannel.Itispossibletomaparegionoffiledirectlyintomemory.Datacanbetransferredtoanotherchannelorfromanother
channelusingthetransferTo(..)andtransferFrom(..)methods.Thesemethodsusetheunderlyingoptimizationoftheoperatingsystem.Filelockingcanbeappliedtomanageaccessbetweenmultipleprocesses.Themethodsarethreadsafeandathreadthatwishes
tomodifythepositionmaybeblockeduntilanotherthreadisactinguponthefile.

ScatterReadusingFileChannel

DatafromaFileChannelcanbereadintomultiplebuffers.Thisisknownasascatterread.Here'sandexampledemonstratingascatterread.

?

ScatterWriteusingFileChannel

wecanwritethebytesbacktothefileusingascatteringwrite.Thefilechannelwascreatedontheinputstreamsothechannelisonlyreadablebutnotwritable.wecreateafilechannelfromanoutputstream

?

ReadingHugeFilesusingMemoryMappedBuffer

JavaNIOallowsreadinghugefilesdirectlyfromthememory.i.e.ThefileneednotbeloadedintotheJVM.Allreadsandwritesonthebytebufferhappendirecltyonthefile.

?

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签：

相关文章推荐

新的分享

章节导航