Quantcast
Channel: Recent Discussions — GATK-Forum
Viewing all articles
Browse latest Browse all 12345
↧

SVCNVDiscovery Error "java.lang.OutOfMemoryError: Java heap space" AFTER adding "-memLimit 100" flag

March 2, 2019, 1:36 pm
$
0
0

Dear Genome STRiP users,

I completed the SVPreprocess to the 10686 samples successfully. When I run SVCNVDiscovery pipeline with the following script,

classpath="${SV_DIR}/lib/SVToolkit.jar:${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar:${SV_DIR}/lib/gatk/Queue.jar"
gs_dir="/proj/yunligrp/users/minzhi/gs"
svpreprocess_dir="${gs_dir}/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success"
rundir="${gs_dir}/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000"

java -Xmx4g -cp ${classpath} \
    org.broadinstitute.gatk.queue.QCommandLine \
    -S ${SV_DIR}/qscript/discovery/cnv/CNVDiscoveryPipeline.q \
    -S ${SV_DIR}/qscript/SVQScript.q \
    -cp ${classpath} \
    -gatk ${SV_DIR}/lib/gatk/GenomeAnalysisTK.jar \
    -configFile ${SV_DIR}/conf/genstrip_parameters.txt \
    -R ${gs_dir}/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta \
    -I ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_sample.list \
    -genderMapFile ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map \
    -ploidyMapFile ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map \
    -md ${svpreprocess_dir}/md_tempdir \
    -tempDir ${gs_dir}/gs_tempdir/svcnv_tmp \
    -runDirectory ${rundir} \
    -jobLogDir ${rundir}/logs \
    -intervalList ${gs_dir}/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_1-500000_interval.list \
    -tilingWindowSize 1000 \
    -tilingWindowOverlap 500 \
    -maximumReferenceGapLength 1000 \
    -boundaryPrecision 100 \
    -minimumRefinedLength 500 \
    -memLimit 100 \
    -jobRunner Drmaa \
    -gatkJobRunner Drmaa \
    -jobNative "--mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8" \
    -jobQueue general \
    -run \
    || exit 1

The modification I made to the QScripts are
1. SVQScript (line 1284&1285)

        this.memoryLimit = Some(85)
        this.javaMemoryLimit = Some(85)
  1. Under "CallSampleGender()" and "ComputeDepthProfile(profilesDir: File, sequenceName: String, intervalList: List[GenomeInterval])" functions, I added the following lines to allow more memory
        this.memoryLimit = Some(10)
        this.javaMemoryLimit = Some(10)

Then I got the following errors:

  1. Non-specific error
ERROR 16:06:20,037 FunctionEdge - Error:  'java'  '-Xmx102400m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/SVQScript.q'  '-gatk' '/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs'  '-memLimit' '100.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '--mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8'  '-jobQueue' 'general'  -run  '-sequenceName' 'chr16'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-sentinelFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_sentinel_files/stage_2_seq_chr16.sent'  --disableJobReport  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-copyNumberMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gcmask.fasta'  '-readDepthMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.rdmask.bed'  '-genderMaskBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gendermask.bed'  '-vdjBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.vdjregions.bed'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  -disableGATKTraversal  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-genderGenotypeFilterFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter.txt'  '-filterDescriptionFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter_descr.txt'  '-genotypingParallelRecords' '1000'  
ERROR 16:06:20,069 FunctionEdge - Contents of /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/logs/CNVDiscoveryPipeline-5.out:
INFO  16:05:27,250 QScriptManager - Compiling 4 QScripts 
INFO  16:05:32,094 QScriptManager - Compilation complete 
INFO  16:05:32,226 HelpFormatter - ---------------------------------------------------------------------- 
INFO  16:05:32,227 HelpFormatter - Queue v3.7.GS-r1748-0-g74bfe0b, Compiled 2018/04/10 10:30:23 
INFO  16:05:32,227 HelpFormatter - Copyright (c) 2012 The Broad Institute 
INFO  16:05:32,227 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
INFO  16:05:32,227 HelpFormatter - Program Args: -cp /proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q -S /proj/yunligrp/users/minzhi/svtoolkit/qscript/SVQScript.q -gatk /proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar -jobLogDir /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs -memLimit 100.0 -jobRunner Drmaa -gatkJobRunner Drmaa -jobNative --mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8 -jobQueue general -run -sequenceName chr16 -runDirectory /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16 -sentinelFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_sentinel_files/stage_2_seq_chr16.sent --disableJobReport -configFile /proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt -R /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta -ploidyMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta -copyNumberMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gcmask.fasta -readDepthMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.rdmask.bed -genderMaskBedFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gendermask.bed -vdjBedFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.vdjregions.bed -genderMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map -md /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir -disableGATKTraversal -I /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam -vcf /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz -genderGenotypeFilterFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter.txt -filterDescriptionFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter_descr.txt -genotypingParallelRecords 1000 
INFO  16:05:32,228 HelpFormatter - Executing as minzhi@c0318.ll.unc.edu on Linux 3.10.0-957.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_191-b12. 
INFO  16:05:32,228 HelpFormatter - Date/Time: 2019/03/02 16:05:32 
INFO  16:05:32,228 HelpFormatter -
  1. java.lang.OutOfMemoryError: Java heap space
ERROR 16:06:03,437 FunctionEdge - Error:  'java'  '-Xmx3072m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar' org.broadinstitute.sv.main.SVGenotyper '-T' 'SVGenotyperWalker'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-O' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/P0001.genotypes.vcf.gz'  '-disableGATKTraversal' 'true'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt' '-configFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gsparams.txt'  '-P' 'genotyping.modules:depth' '-P' 'depth.readCountCacheIgnoreGenomeMask:true'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-partitionName' 'P0001'  '-partition' 'records:1-922'  
ERROR 16:06:03,450 FunctionEdge - Contents of /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs/CNVDiscoveryStage2-1.out:
INFO  16:05:46,373 HelpFormatter - ----------------------------------------------------------------------------------------- 
INFO  16:05:46,375 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7.GS-r1748-0-g74bfe0b, Compiled 2018/04/10 10:30:23 
INFO  16:05:46,376 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute 
INFO  16:05:46,376 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk 
INFO  16:05:46,376 HelpFormatter - [Sat Mar 02 16:05:46 EST 2019] Executing on Linux 3.10.0-957.el7.x86_64 amd64 
INFO  16:05:46,376 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_191-b12 
INFO  16:05:46,380 HelpFormatter - Program Args: -T SVGenotyperWalker -R /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta -O /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/P0001.genotypes.vcf.gz -disableGATKTraversal true -md /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir -configFile /proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt -configFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gsparams.txt -P genotyping.modules:depth -P depth.readCountCacheIgnoreGenomeMask:true -runDirectory /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16 -genderMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map -ploidyMapFile /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta -genomeMaskFile /proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta -vcf /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz -partitionName P0001 -partition records:1-922 -L chr1:1-1 
INFO  16:05:46,385 HelpFormatter - Executing as minzhi@c0320.ll.unc.edu on Linux 3.10.0-957.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_191-b12. 
INFO  16:05:46,385 HelpFormatter - Date/Time: 2019/03/02 16:05:46 
INFO  16:05:46,386 HelpFormatter - ----------------------------------------------------------------------------------------- 
INFO  16:05:46,386 HelpFormatter - ----------------------------------------------------------------------------------------- 
INFO  16:05:46,396 02-Mar-2019 GenomeAnalysisEngine - Strictness is SILENT
INFO  16:05:47,594 02-Mar-2019 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000
INFO  16:05:48,474 02-Mar-2019 IntervalUtils - Processing 1 bp from intervals
INFO  16:05:48,521 02-Mar-2019 GenomeAnalysisEngine - Preparing for traversal
INFO  16:05:48,522 02-Mar-2019 GenomeAnalysisEngine - Done preparing for traversal
INFO  16:05:48,522 02-Mar-2019 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO  16:05:48,522 02-Mar-2019 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining
INFO  16:05:48,522 02-Mar-2019 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime
INFO  16:05:48,528 02-Mar-2019 SVGenotyper - Opening reference sequence ...
INFO  16:05:48,528 02-Mar-2019 SVGenotyper - Opened reference sequence.
INFO  16:05:48,529 02-Mar-2019 SVGenotyper - Opening genome mask ...
INFO  16:05:48,530 02-Mar-2019 SVGenotyper - Opened genome mask.
INFO  16:05:48,532 02-Mar-2019 MetaData - Opening metadata ... 
INFO  16:05:48,532 02-Mar-2019 MetaData - Adding metadata location /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir ...
INFO  16:05:48,535 02-Mar-2019 MetaData - Opened metadata.
INFO  16:05:48,569 02-Mar-2019 SVGenotyper - Initializing input data set ...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
    at java.lang.StringBuilder.append(StringBuilder.java:136)
    at htsjdk.samtools.SAMTextHeaderCodec.advanceLine(SAMTextHeaderCodec.java:139)
    at htsjdk.samtools.SAMTextHeaderCodec.decode(SAMTextHeaderCodec.java:94)
    at htsjdk.samtools.BAMFileReader.readHeader(BAMFileReader.java:655)
    at htsjdk.samtools.BAMFileReader.<init>(BAMFileReader.java:298)
    at htsjdk.samtools.BAMFileReader.<init>(BAMFileReader.java:176)
    at htsjdk.samtools.SamReaderFactory$SamReaderFactoryImpl.open(SamReaderFactory.java:376)
    at htsjdk.samtools.SamReaderFactory$SamReaderFactoryImpl.open(SamReaderFactory.java:202)
    at org.broadinstitute.sv.dataset.SAMFileLocation.createSamFileReader(SAMFileLocation.java:97)
    at org.broadinstitute.sv.dataset.SAMLocation.createSamFileReader(SAMLocation.java:41)
    at org.broadinstitute.sv.dataset.DataSet.initInputFile(DataSet.java:138)
    at org.broadinstitute.sv.dataset.DataSet.initialize(DataSet.java:128)
    at org.broadinstitute.sv.genotyping.SVGenotyperWalker.initDataSet(SVGenotyperWalker.java:355)
    at org.broadinstitute.sv.genotyping.SVGenotyperWalker.initialize(SVGenotyperWalker.java:197)
    at org.broadinstitute.gatk.engine.executive.LinearMicroScheduler.execute(LinearMicroScheduler.java:83)
    at org.broadinstitute.gatk.engine.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:316)
    at org.broadinstitute.gatk.engine.CommandLineExecutable.execute(CommandLineExecutable.java:123)
    at org.broadinstitute.sv.main.SVCommandLine.execute(SVCommandLine.java:141)
    at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:256)
    at org.broadinstitute.gatk.utils.commandline.CommandLineProgram.start(CommandLineProgram.java:158)
    at org.broadinstitute.sv.main.SVCommandLine.main(SVCommandLine.java:91)
    at org.broadinstitute.sv.main.SVGenotyper.main(SVGenotyper.java:21) 
INFO  16:06:03,451 QGraph - Writing incremental jobs reports... 
INFO  16:06:03,452 QGraph - 8 Pend, 0 Run, 1 Fail, 0 Done 
INFO  16:06:03,453 QCommandLine - Writing final jobs report... 
INFO  16:06:03,454 QCommandLine - Done with errors

You can find that even though I added the flag "-memLimit 100", at top of the second error, it still shows '-Xmx3072m'.

Besides, there are also two failure messages here:

The first also have shows '-Xmx3072m':

INFO  16:06:03,458 QGraph - Failed:   'java'  '-Xmx3072m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar' org.broadinstitute.sv.main.SVGenotyper '-T' 'SVGenotyperWalker'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-O' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/P0001.genotypes.vcf.gz'  '-disableGATKTraversal' 'true'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt' '-configFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gsparams.txt'  '-P' 'genotyping.modules:depth' '-P' 'depth.readCountCacheIgnoreGenomeMask:true'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-partitionName' 'P0001'  '-partition' 'records:1-922'  
INFO  16:06:03,458 QGraph - Log:     /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs/CNVDiscoveryStage2-1.out 
INFO  16:06:03,459 QCommandLine - Script failed: 8 Pend, 0 Run, 1 Fail, 0 Done 
------------------------------------------------------------------------------------------
Done. There were no warn messages.

But the second shows '-Xmx102400m':

INFO  16:06:22,771 QGraph - Failed:   'java'  '-Xmx102400m'  '-XX:+UseParallelOldGC'  '-XX:ParallelGCThreads=4'  '-XX:GCTimeLimit=50'  '-XX:GCHeapFreeLimit=10'  '-Djava.io.tmpdir=/proj/yunligrp/users/minzhi/gs/gs_tempdir/svcnv_tmp'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  'org.broadinstitute.gatk.queue.QCommandLine'  '-cp' '/proj/yunligrp/users/minzhi/svtoolkit/lib/SVToolkit.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar:/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/Queue.jar'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStage2.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryStageBase.q' '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/discovery/cnv/CNVDiscoveryGenotyper.q'  '-S' '/proj/yunligrp/users/minzhi/svtoolkit/qscript/SVQScript.q'  '-gatk' '/proj/yunligrp/users/minzhi/svtoolkit/lib/gatk/GenomeAnalysisTK.jar'  '-jobLogDir' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16/logs'  '-memLimit' '100.0'  '-jobRunner' 'Drmaa'  '-gatkJobRunner' 'Drmaa'  '-jobNative' '--mem=100000 --time=08:00:00 --nodes=1 --ntasks-per-node=8'  '-jobQueue' 'general'  -run  '-sequenceName' 'chr16'  '-runDirectory' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage2/seq_chr16'  '-sentinelFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_sentinel_files/stage_2_seq_chr16.sent'  --disableJobReport  '-configFile' '/proj/yunligrp/users/minzhi/svtoolkit/conf/genstrip_parameters.txt'  '-R' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.fasta'  '-ploidyMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_standard_ploidy.map'  '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.svmask.fasta' '-genomeMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.lcmask.fasta'  '-copyNumberMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gcmask.fasta'  '-readDepthMaskFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.rdmask.bed'  '-genderMaskBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.gendermask.bed'  '-vdjBedFile' '/proj/yunligrp/users/minzhi/gs/Homo_sapiens_assembly38/Homo_sapiens_assembly38.vdjregions.bed'  '-genderMapFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/supporting_freeze6-AA_chr16/freeze6-AA_chr16_full_all-male_gender.map'  '-md' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svpre_freeze6-AA_chr16_standard_full_single_1-500000over1-500000_parallel_success/md_tempdir'  -disableGATKTraversal  '-I' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/bam_headers/merged_headers.bam'  '-vcf' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/cnv_stage1/seq_chr16/seq_chr16.sites.vcf.gz'  '-genderGenotypeFilterFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter.txt'  '-filterDescriptionFile' '/proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/gender_gt_filters/gender_gt_filter_descr.txt'  '-genotypingParallelRecords' '1000'  
INFO  16:06:22,771 QGraph - Log:     /proj/yunligrp/users/minzhi/gs/freeze6-AA_chr16/svcnv_freeze6-AA_chr16_standard_full_single_1-500000over1-500000/logs/CNVDiscoveryPipeline-5.out 
INFO  16:06:22,772 QCommandLine - Script failed: 10 Pend, 0 Run, 1 Fail, 4 Done

Does it mean that I need more deeply edit somewhere in the Qscript to allow more memory? Because I indeed find that the first subjob (successful) trigged by the SVCNVDiscovery takes about 80 GB memory, but the other subjobs (failed) only takes about 2.9 GB memory. And especially, some errors and failure coming from QGraph steps which I cannot find in the QScript. May I have your suggestions about these errors? Thank you very much.

Best regards,
Wusheng

↧
Search
Viewing all articles
Browse latest Browse all 12345

Trending Articles

Practice Sheet of Right form of verbs for HSC Students

September 22, 2019, 11:40 pm

Download: FK ft Shenky – Nakuyewa ”Prod by: Shenky”

February 16, 2017, 4:24 pm

How to win at Markstrat (Markstrat Tips and Tricks) – Vodites

January 5, 2014, 10:34 pm

Ominde Commission Report and Recommendations – Ominde Report of 1964

March 16, 2015, 5:14 am

Bureau of Internal Revenue: Regional Offices (Directory)

January 9, 2014, 11:06 pm

GO 53 on Enhancement of Ex-gratia upto 5 Lakhs Toddy Tappers in Telangana

March 26, 2017, 11:23 pm

Cakewalk CA-2A Leveling Amplifier v2.0.1.97 WiN, v2.0.1.96 OSX Incl Keygen

October 17, 2016, 7:20 am

Mp3 Download: Mdu - Kunjenjenjena

December 7, 2017, 8:16 am

How the kill the job , when DTP request running for long hours.

July 26, 2013, 2:41 am

Microsoft Intune から展開しているアプリのアップデートについて

October 17, 2016, 4:11 am

18-year-old girl was beaten for half an hour by two Northampton men in 'an...

September 1, 2017, 10:00 pm

Car crash in Dunton Bassett leaves driver in critical condition

October 7, 2014, 7:51 am

Macky 2, Two Others In Road Accident

March 29, 2015, 5:34 am

Application log 00000000000000089514: Could not convert queue DLVST90CLNT

May 14, 2015, 11:27 pm

Detroit mafia: D’Anna Brothers agree to plea deal

April 21, 2016, 6:56 am

Delivery block field greyed out using VA02

January 26, 2016, 2:52 pm

Muloraki Au

June 22, 2016, 1:44 am

【個人撮影】スマホのプライベート映像♪「中に出さないで///」カラオケ屋での生ハメ撮りが流出ｗ【リベンジポルノ】＠PornHub

October 12, 2017, 2:23 pm

BREAKING NEWS: Diamond Platnumz Is Reported Dead After Ghastly Car Accident

February 9, 2018, 4:56 am

FIAT 500 B0111 B0112

July 5, 2018, 10:31 am
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>