Hello,
I ran ExtractIlluminaBarcodes and came out with the following metrics file:
METRICS CLASS picard.illumina.ExtractIlluminaBarcodes$BarcodeMetric
BARCODE BARCODE_NAME LIBRARY_NAME READS PF_READS PERFECT_MATCHES PF_PERFECT_MATCHES ONE_MISMATCH_MATCHES PF_ONE_MISMATCH_MATCHES PCT_MATCHES RATIO_THIS_BARCODE_TO_BEST_BARCODE_PCT PF_PCT_MATCHES PF_RATIO_THIS_BARCODE_TO_BEST_BARCODE_PCT PF_NORMALIZED_MATCHES
TAAGGC 3787945 3787945 3283555 3283555 504390 504390 0.0245 1 0.024741 1 1.283624
AGGCAG 2902034 2902034 2473189 2473189 428845 428845 0.01877 0.766124 0.018955 0.766124 0.983415
CGTACT 2162953 2162953 1869231 1869231 293722 293722 0.01399 0.57101 0.014127 0.57101 0.732962
NNNNNN 145758436 144249478 0 0 0 0 0.942741 38.479554 0.942176 38.081197 0
now I need to run IlluminaBasecallsToSam, with
java -jar 3rdParty/picard/Old_picard.jar IlluminaBasecallsToSam BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/Basecalls LANE=1 READ_STRUCTURE=20T6B55T RUN_BARCODE=run15 LIBRARY_PARAMS=library_params.tsv
where library_params.tsv looks like
BARCODE OUTPUT SAMPLE_ALIAS LIBRARY_NAME
TAAGGC TAAGGC.bam TAAGGC TAAGGC
AGGCAG AGGCAG.bam AGGCAG AGGCAG
CGTACT CGTACT.bam CGTACT CGTACT
NNNNNN NNNNNN.bam NNNNNN NNNNNN
but this generates a huge list of errors, how can I get IlluminaBasecallsToSam to run correctly?
con@e:~/Alex$ java -jar 3rdParty/picard/Old_picard.jar IlluminaBasecallsToSam BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/Basecalls LANE=1 READ_STRUCTURE=20T6B55T RUN_BARCODE=TAAGGC LIBRARY_PARAMS=library_params.tsv
[Thu Dec 22 08:32:30 EST 2016] picard.illumina.IlluminaBasecallsToSam BASECALLS_DIR=/home/con/Alex/Files/Data/Intensities/Basecalls LANE=1 RUN_BARCODE=TAAGGC READ_STRUCTURE=20T6B55T LIBRARY_PARAMS=library_params.tsv SEQUENCING_CENTER=BI PLATFORM=illumina ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2, FLUIDIGM] NUM_PROCESSORS=0 FORCE_GC=true APPLY_EAMSS_FILTER=true MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 INCLUDE_NON_PF_READS=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
[Thu Dec 22 08:32:30 EST 2016] Executing as con@e on Linux 4.4.0-57-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2ubuntu0.16.04.2-b14; Picard version: 1.138(89618e408692ff6288c7c880658f32f16fcbec53_1441135673) IntelDeflater
INFO 2016-12-22 08:32:30 IlluminaBasecallsToSam DONE_READING STRUCTURE IS 20T6B55T
ERROR 2016-12-22 08:32:31 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:695)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:311)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:213)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:206)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-1" picard.PicardException: Read records with barcode null, but this barcode was not expected. (Is it referenced in the parameters file?)
at picard.illumina.IlluminaBasecallsConverter$TileProcessingRecord.addRecord(IlluminaBasecallsConverter.java:430)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:546)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-1-thread-7" Exception in thread "pool-1-thread-2" picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0055.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
Exception in thread "pool-1-thread-6" Exception in thread "pool-1-thread-5" Exception in thread "pool-1-thread-8" Exception in thread "pool-1-thread-3" picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0060.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:971)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:114)
... 18 more
Exception in thread "pool-1-thread-4" picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0063.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:971)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:114)
... 18 more
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0046.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0065.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0049.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
[Thu Dec 22 08:32:31 EST 2016] picard.illumina.IlluminaBasecallsToSam done. Elapsed time: 0.01 minutes.
picard.PicardException: IOException opening cluster binary file /home/con/Alex/Files/Data/Intensities/Basecalls/L001/0051.bcl.bgzf.bci
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getLongIterator(MMapBackedIteratorFactory.java:82)
at picard.illumina.parser.readers.BclIndexReader.(BclIndexReader.java:47)
at picard.illumina.parser.readers.BclReader.seek(BclReader.java:336)
at picard.illumina.parser.MultiTileBclParser.makeReader(MultiTileBclParser.java:60)
at picard.illumina.parser.MultiTileBclParser.access$000(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser$MultiTileBclDataCycleFileParser.(MultiTileBclParser.java:128)
at picard.illumina.parser.MultiTileBclParser.makeCycleFileParser(MultiTileBclParser.java:71)
at picard.illumina.parser.PerTileCycleParser.seekToTile(PerTileCycleParser.java:131)
at picard.illumina.parser.MultiTileBclParser.seekToTile(MultiTileBclParser.java:38)
at picard.illumina.parser.MultiTileBclParser.initialize(MultiTileBclParser.java:53)
at picard.illumina.parser.MultiTileBclParser.(MultiTileBclParser.java:47)
at picard.illumina.parser.IlluminaDataProviderFactory.makeParser(IlluminaDataProviderFactory.java:378)
at picard.illumina.parser.IlluminaDataProviderFactory.makeDataProvider(IlluminaDataProviderFactory.java:234)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:537)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:643)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:315)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 18 more
Runtime.totalMemory()=438304768
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:314)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:213)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:206)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)