terrier建立索引遇到的问题，以及解决方案

Issuse1：java.lang.OutOfMemoryError: Java heap space

具体错误如下：

java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

at gnu.trove.TObjectIntHashMap.rehash(TObjectIntHashMap.java:170)

at gnu.trove.THash.postInsertHook(THash.java:359)

at gnu.trove.TObjectIntHashMap.put(TObjectIntHashMap.java:155)

at org.terrier.utility.TermCodes.getCode(TermCodes.java:100)

at org.terrier.structures.indexing.DocumentPostingList.getTermId(DocumentPostingList.java:133)

at org.terrier.structures.indexing.DocumentPostingList$2.execute(DocumentPostingList.java:168)

at org.terrier.structures.indexing.DocumentPostingList$2.execute(DocumentPostingList.java:166)

at gnu.trove.TObjectIntHashMap.forEachEntry(TObjectIntHashMap.java:426)

at org.terrier.structures.indexing.DocumentPostingList.getPostings2(DocumentPostingList.java:165)

at org.terrier.indexing.BasicIndexer.indexDocument(BasicIndexer.java:368)

at org.terrier.indexing.BasicIndexer.createDirectIndex(BasicIndexer.java:261)

at org.terrier.indexing.Indexer.index(Indexer.java:344)

at org.terrier.applications.TRECIndexing.index(TRECIndexing.java:123)

at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:390)

at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:573)

at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:237)

21877.18user 916.34system 6:01:37elapsed 105%CPU (0avgtext+0avgdata 0maxresident)k

45946520inputs+21416016outputs (1major+1978833minor)pagefaults 0swaps

解决方案：

increased the maximum Java Heap Space to 2GB, by setting TERRIER_HEAP_MEM to 2048M in bin/terrier-env.sh.

And It seems to be running smoothly.

Issuse2:（可以概括为Key is not unique）

ERROR - This index (Index(/users/ishanic/terrier-3.0/var/index,data_1)) doesnt have an index structure called lexicon-keyfactory: property index.lexicon-keyfactory.class not found

ERROR - Valid structures are: [document-inputstream, meta-inputstream, document-factory, meta, document]

具体错误如下：

NFO - Collection #0 took 183780 seconds to build the runs for 20000000 documents

ERROR - Problem finishing index

java.io.IOException: Key is not unique: 38131,3514

at org.terrier.structures.collections.FSOrderedMapFile$MultiFSOMapWriter.mergeTwo(FSOrderedMapFile.java:908)

at org.terrier.structures.collections.FSOrderedMapFile$MultiFSOMapWriter.close(FSOrderedMapFile.java:861)

at org.terrier.structures.indexing.CompressingMetaIndexBuilder.close(CompressingMetaIndexBuilder.java:259)

at org.terrier.indexing.BasicSinglePassIndexer.createInvertedIndex(BasicSinglePassIndexer.java:274)

at org.terrier.indexing.BasicSinglePassIndexer.createDirectIndex(BasicSinglePassIndexer.java:147)

at org.terrier.indexing.Indexer.index(Indexer.java:344)

at org.terrier.applications.TRECIndexing.createSinglePass(TRECIndexing.java:221)

at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:384)

at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:573)

at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:237)

INFO - Optimising structure lexicon

ERROR - This index (Index(/users/ishanic/terrier-3.0/var/index,data_1)) doesnt have an index structure called lexicon-keyfactory: property index.lexicon-keyfactory.class not found

ERROR - Valid structures are: [document-inputstream, meta-inputstream, document-factory, meta, document]

ERROR - This index (Index(/users/ishanic/terrier-3.0/var/index,data_1)) doesnt have an index structure called lexicon-valuefactory: property index.lexicon-valuefactory.class not found

ERROR - Valid structures are: [document-inputstream, meta-inputstream, document-factory, meta, document]

A problem occurred: java.lang.NullPointerException

java.lang.NullPointerException

at org.terrier.structures.collections.FSOrderedMapFile.numberOfEntries(FSOrderedMapFile.java:490)

at org.terrier.structures.FSOMapFileLexicon.optimise(FSOMapFileLexicon.java:389)

at org.terrier.structures.indexing.LexiconBuilder.optimise(LexiconBuilder.java:790)

at org.terrier.indexing.BasicIndexer.finishedInvertedIndexBuild(BasicIndexer.java:438)

at org.terrier.indexing.BasicSinglePassIndexer.createInvertedIndex(BasicSinglePassIndexer.java:292)

at org.terrier.indexing.BasicSinglePassIndexer.createDirectIndex(BasicSinglePassIndexer.java:147)

at org.terrier.indexing.Indexer.index(Indexer.java:344)

at org.terrier.applications.TRECIndexing.createSinglePass(TRECIndexing.java:221)

at org.terrier.applications.TrecTerrier.run(TrecTerrier.java:384)

at org.terrier.applications.TrecTerrier.applyOptions(TrecTerrier.java:573)

at org.terrier.applications.TrecTerrier.main(TrecTerrier.java:237)

解决方案：

这些问题是建立meta index 造成的，以下为解决方案：

Give this issue some thought.

* My initial idea was that your indexer.meta.forward.keylens was too small, but this is not the case.

* The error is occurring when building the reverse lookup table (docno -> docid). Will you need this functionality? If not, then you can disable it using indexer.meta.reverse.keys= during indexing.

* Otherwise, can you alter the exception being raised in FSOrderedMapFile to print the value of the key that is causing the collision?

我采用的是第二种，问题迎刃而解。