基因数据处理100之bwamem算法处理100万条paired-reads数据GRCH38chr1L3556522N1000000L100paired12
2018-01-11 00:44
375 查看
运行记录：
flagstat：
hadoop@Master:~/xubo/ref/GRCH38L1Index/pe$ bwa mem ../GRCH38chr1L3556522.fasta GRCH38chr1L3556522N1000000L100paired1.fastq GRCH38chr1L3556522N1000000L100paired2.fastq >GRCH38chr1L3556522N1000000L100paired12.sam [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 100000 sequences (10000000 bp)... [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44602, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.79, 49.84) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.011 CPU sec, 37.988 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44446, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666) [M::mem_pestat] mean and std.dev: (498.57, 49.70) [M::mem_pestat] low and high boundaries for proper pairs: (264, 733) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.287 CPU sec, 38.187 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44553, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (499.09, 49.89) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.447 CPU sec, 37.335 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44490, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.78, 49.86) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.639 CPU sec, 37.529 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44537, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (499.18, 49.73) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.181 CPU sec, 37.063 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44493, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667) [M::mem_pestat] mean and std.dev: (499.19, 49.67) [M::mem_pestat] low and high boundaries for proper pairs: (265, 734) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.731 CPU sec, 37.645 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44395, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (499.19, 49.91) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.868 CPU sec, 38.783 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44428, 1, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.75, 49.92) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.643 CPU sec, 39.199 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44455, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.90, 49.75) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.059 CPU sec, 38.147 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44497, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.75, 49.95) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.032 CPU sec, 38.342 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44452, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667) [M::mem_pestat] mean and std.dev: (499.25, 49.76) [M::mem_pestat] low and high boundaries for proper pairs: (265, 734) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.383 CPU sec, 41.257 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44624, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.90, 49.83) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.706 CPU sec, 37.771 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44548, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 498, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666) [M::mem_pestat] mean and std.dev: (498.50, 49.73) [M::mem_pestat] low and high boundaries for proper pairs: (264, 733) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.009 CPU sec, 38.306 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44369, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667) [M::mem_pestat] mean and std.dev: (499.29, 49.65) [M::mem_pestat] low and high boundaries for proper pairs: (265, 734) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.027 CPU sec, 38.429 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44525, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666) [M::mem_pestat] mean and std.dev: (498.71, 49.67) [M::mem_pestat] low and high boundaries for proper pairs: (264, 733) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.977 CPU sec, 39.461 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44586, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666) [M::mem_pestat] mean and std.dev: (498.80, 49.59) [M::mem_pestat] low and high boundaries for proper pairs: (264, 733) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.294 CPU sec, 40.004 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44519, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (464, 498, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (328, 668) [M::mem_pestat] mean and std.dev: (498.21, 50.11) [M::mem_pestat] low and high boundaries for proper pairs: (260, 736) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.618 CPU sec, 43.286 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44469, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (329, 669) [M::mem_pestat] mean and std.dev: (498.90, 49.89) [M::mem_pestat] low and high boundaries for proper pairs: (261, 737) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.862 CPU sec, 39.607 real sec [M::process] read 100000 sequences (10000000 bp)... [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44503, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (466, 499, 533) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (332, 667) [M::mem_pestat] mean and std.dev: (498.94, 49.63) [M::mem_pestat] low and high boundaries for proper pairs: (265, 734) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 38.381 CPU sec, 41.691 real sec [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 44510, 0, 0) [M::mem_pestat] skip orientation FF as there are not enough pairs [M::mem_pestat] analyzing insert size distribution for orientation FR... [M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532) [M::mem_pestat] low and high boundaries for computing mean and std.dev: (331, 666) [M::mem_pestat] mean and std.dev: (498.66, 49.88) [M::mem_pestat] low and high boundaries for proper pairs: (264, 733) [M::mem_pestat] skip orientation RF as there are not enough pairs [M::mem_pestat] skip orientation RR as there are not enough pairs [M::mem_process_seqs] Processed 100000 reads in 37.654 CPU sec, 41.754 real sec [main] Version: 0.7.13-r1126 [main] CMD: bwa mem ../GRCH38chr1L3556522.fasta GRCH38chr1L3556522N1000000L100paired1.fastq GRCH38chr1L3556522N1000000L100paired2.fastq [main] Real time: 790.344 sec; CPU: 760.405 sec
flagstat：
hadoop@Master:~/xubo/ref/GRCH38L1Index/pe$ samtools flagstat GRCH38chr1L3556522N1000000L100paired12.sam 2000000 + 0 in total (QC-passed reads + QC-failed reads) 0 + 0 secondary 0 + 0 supplementary 0 + 0 duplicates 2000000 + 0 mapped (100.00% : N/A) 2000000 + 0 paired in sequencing 1000000 + 0 read1 1000000 + 0 read2 1999960 + 0 properly paired (100.00% : N/A) 2000000 + 0 with itself and mate mapped 0 + 0 singletons (0.00% : N/A) 0 + 0 with mate mapped to a different chr 0 + 0 with mate mapped to a different chr (mapQ>=5)
参考
【1】https://github.com/xubo245/AdamLearning 【2】https://github.com/bigdatagenomics/adam/ 【3】https://github.com/xubo245/SparkLearning 【4】http://spark.apache.org 【5】http://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job 【6】http://stackoverflow.com/questions/28840438/how-to-override-sparks-log4j-properties-per-driver
研究成果：
【1】 [BIBM] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Chao Wang, and Xuehai Zhou, "Distributed Gene Clinical Decision Support System Based on Cloud Computing", in IEEE International Conference on Bioinformatics and Biomedicine. (BIBM 2017, CCF B) 【2】 [IEEE CLOUD] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Xuehai Zhou. Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark (CLOUD 2017, CCF-C). 【3】 [CCGrid] Bo Xu, Changlong Li, Hang Zhuang, Jiali Wang, Qingfeng Wang, Jinhong Zhou, Xuehai Zhou. DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions. (CCGrid 2017, CCF-C). 【4】more: https://github.com/xubo245/Publications
Help
If you have any questions or suggestions, please write it in the issue of this project or send an e-mail to me: xubo245@mail.ustc.edu.cn Wechat: xu601450868 QQ: 601450868
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- 基因数据处理102之SparkBWA本地运行100万条paired-reads实例
- 基因数据处理58之snap运行paired-end（1千万条100bp的reads对）
- 基因数据处理53之cs-bwamem集群版运行paird-end（1千万条100bp的reads）
- 基因数据处理103之SparkBWAYarn模式运行100万条paired-reads实例
- 基因数据处理61之idea运行cs-bwamem处理single-end（1条100bp的reads）
- 基因数据处理105之SparkBWAYarn模式运行1000万条paired-reads实例g38L100c10000000Nhs20Paired12YarnPartition
- 基因数据处理84之cs-bwamem处理小数据集
- 基因数据处理57之BWA-MEM运行single-end(1千万条100bp的reads)
- 基因数据处理82之cs-bwamem处理SRR003161（参考基因组为GRCH38chr1）
- 基因数据处理44之cloud-scale-bwamem安装
- 基因数据处理59之snap运行single-end（1千万条100bp的reads）
- 基因数据处理82之cs-bwamem处理SRR003161（参考基因组为GRCH38chr1）
- 基因数据处理60之bwa运行single-end（1千万条100bp的reads）
- 基因数据处理54之bwa-mem运行paird-end（1千万条100bp的reads）
- 基因数据处理55之cs-bwamem安装记录（idea maven ，没有通过pl）
- 基因数据处理56之bwa运行paird-end（1千万条100bp的reads）.md
- 基因数据处理62之snap默认无法处理大于400bp的reads
- 基因数据处理45之cloud-scale-bwamem安装（compile.pl安装有问题）
- 基因数据处理49之cloud-scale-bwamem运行成功
- 基因数据处理50之cs-bwamem、bwa、snap、bwa-mem与art比较