I am attempting to demultiplex a lane of Illumina HiSeq2500 data using Picard IlluminaBasecallsToSam (v2.5, Java(TM) SE Runtime Environment (build 1.8.0_20-b26)). This tool fails to complete and states that
picard.PicardException: Read the number of expected bins( 65600) but still had more elements in file( /Illumina/Basecalls/170808_SN218_0895_AHN3VWBCXY/Data/Intensities/L001/s_1_1107.clocs)
I've also run Picard CheckIlluminaDirectory and everything turns out to be fine. Also, I can successfully demultiplex lane 2 from this run that has the same read-structure. I suspected that a file was corrupted during network transfer so I re-ran RTA but Picard IlluminaBasecallsToSam still has the same result.
I'd appreciate any idea about what could be causing this error.
Complete error below:
[Tue Aug 15 11:01:59 EDT 2017] picard.illumina.IlluminaBasecallsToSam BASECALLS_DIR=/Illumina/Basecalls/170808_SN218_0895_AHN3VWBCXY/Data/Intensities/BaseCalls BARCODES_DIR=/Project/Capture LANE=1 RUN_BARCODE=HN3VWBCXY170810 READ_GROUP_ID=HN3VWBCXY170810 SEQUENCING_CENTER=ABC READ_STRUCTURE=98T8B6M8B98T LIBRARY_PARAMS=/Project/DemultCapture.txt NUM_PROCESSORS=6 IGNORE_UNEXPECTED_BARCODES=true TMP_DIR=[/Project/TMP] PLATFORM=illumina ADAPTERS_TO_CHECK=[INDEXED, DUAL_INDEXED, NEXTERA_V2, FLUIDIGM] FORCE_GC=true APPLY_EAMSS_FILTER=true MAX_READS_IN_RAM_PER_TILE=1200000 MINIMUM_QUALITY=2 INCLUDE_NON_PF_READS=true MOLECULAR_INDEX_TAG=RX MOLECULAR_INDEX_BASE_QUALITY_TAG=QX VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
[Tue Aug 15 11:01:59 EDT 2017] Executing as rb@rcsgc22 on Linux 2.6.32-358.2.1.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_20-b26; Picard version: 2.5.0(2c370988aefe41f579920c8a6a678a201c5261c1_1466708365)
INFO 2017-08-15 11:04:47 IlluminaBasecallsToSam DONE_READING STRUCTURE IS 98T8B6M8B98T
INFO 2017-08-15 11:05:36 IlluminaBasecallsConverter Read 1,000,000 records. Elapsed time: 00:03:36s. Time for last 1,000,000: 35s. Last read position: */*
INFO 2017-08-15 11:06:14 IlluminaBasecallsConverter Read 2,000,000 records. Elapsed time: 00:04:14s. Time for last 1,000,000: 37s. Last read position: */*
INFO 2017-08-15 11:06:32 IlluminaBasecallsConverter Read 3,000,000 records. Elapsed time: 00:04:32s. Time for last 1,000,000: 18s. Last read position: */*
INFO 2017-08-15 11:07:00 IlluminaBasecallsConverter Read 4,000,000 records. Elapsed time: 00:04:59s. Time for last 1,000,000: 27s. Last read position: */*
INFO 2017-08-15 11:07:00 IlluminaBasecallsConverter Before explicit GC, Runtime.totalMemory()=6693060608
INFO 2017-08-15 11:07:01 IlluminaBasecallsConverter After explicit GC, Runtime.totalMemory()=6512705536
INFO 2017-08-15 11:07:22 IlluminaBasecallsConverter Read 5,000,000 records. Elapsed time: 00:05:22s. Time for last 1,000,000: 22s. Last read position: */*
INFO 2017-08-15 11:07:33 IlluminaBasecallsConverter Read 6,000,000 records. Elapsed time: 00:05:33s. Time for last 1,000,000: 11s. Last read position: */*
INFO 2017-08-15 11:07:41 IlluminaBasecallsConverter Read 7,000,000 records. Elapsed time: 00:05:41s. Time for last 1,000,000: 8s. Last read position: */*
INFO 2017-08-15 11:07:56 IlluminaBasecallsConverter Read 8,000,000 records. Elapsed time: 00:05:56s. Time for last 1,000,000: 14s. Last read position: */*
INFO 2017-08-15 11:08:09 IlluminaBasecallsConverter Read 9,000,000 records. Elapsed time: 00:06:09s. Time for last 1,000,000: 12s. Last read position: */*
INFO 2017-08-15 11:08:24 IlluminaBasecallsConverter Read 10,000,000 records. Elapsed time: 00:06:24s. Time for last 1,000,000: 15s. Last read position: */*
INFO 2017-08-15 11:08:35 IlluminaBasecallsConverter Read 11,000,000 records. Elapsed time: 00:06:35s. Time for last 1,000,000: 10s. Last read position: */*
INFO 2017-08-15 11:08:48 IlluminaBasecallsConverter Read 12,000,000 records. Elapsed time: 00:06:48s. Time for last 1,000,000: 12s. Last read position: */*
INFO 2017-08-15 11:08:58 IlluminaBasecallsConverter Read 13,000,000 records. Elapsed time: 00:06:58s. Time for last 1,000,000: 10s. Last read position: */*
INFO 2017-08-15 11:09:06 IlluminaBasecallsConverter Read 14,000,000 records. Elapsed time: 00:07:06s. Time for last 1,000,000: 8s. Last read position: */*
INFO 2017-08-15 11:09:16 IlluminaBasecallsConverter Read 15,000,000 records. Elapsed time: 00:07:16s. Time for last 1,000,000: 10s. Last read position: */*
INFO 2017-08-15 11:09:25 IlluminaBasecallsConverter Read 16,000,000 records. Elapsed time: 00:07:24s. Time for last 1,000,000: 8s. Last read position: */*
INFO 2017-08-15 11:09:38 IlluminaBasecallsConverter Read 17,000,000 records. Elapsed time: 00:07:38s. Time for last 1,000,000: 13s. Last read position: */*
INFO 2017-08-15 11:09:57 IlluminaBasecallsConverter Write 1,000,000 records. Elapsed time: 00:07:57s. Time for last 1,000,000: 13s. Last read position: */*
INFO 2017-08-15 11:10:03 IlluminaBasecallsConverter Read 18,000,000 records. Elapsed time: 00:08:03s. Time for last 1,000,000: 25s. Last read position: */*
INFO 2017-08-15 11:10:08 IlluminaBasecallsConverter Write 2,000,000 records. Elapsed time: 00:08:07s. Time for last 1,000,000: 10s. Last read position: */*
INFO 2017-08-15 11:10:17 IlluminaBasecallsConverter Write 3,000,000 records. Elapsed time: 00:08:17s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:10:27 IlluminaBasecallsConverter Read 19,000,000 records. Elapsed time: 00:08:26s. Time for last 1,000,000: 23s. Last read position: */*
INFO 2017-08-15 11:10:29 IlluminaBasecallsConverter Write 4,000,000 records. Elapsed time: 00:08:29s. Time for last 1,000,000: 12s. Last read position: */*
INFO 2017-08-15 11:10:39 IlluminaBasecallsConverter Write 5,000,000 records. Elapsed time: 00:08:38s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:10:48 IlluminaBasecallsConverter Write 6,000,000 records. Elapsed time: 00:08:48s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:10:48 IlluminaBasecallsConverter Read 20,000,000 records. Elapsed time: 00:08:48s. Time for last 1,000,000: 21s. Last read position: */*
INFO 2017-08-15 11:10:59 IlluminaBasecallsConverter Write 7,000,000 records. Elapsed time: 00:08:59s. Time for last 1,000,000: 10s. Last read position: */*
INFO 2017-08-15 11:11:08 IlluminaBasecallsConverter Write 8,000,000 records. Elapsed time: 00:09:08s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:11:14 IlluminaBasecallsConverter Read 21,000,000 records. Elapsed time: 00:09:14s. Time for last 1,000,000: 25s. Last read position: */*
INFO 2017-08-15 11:11:18 IlluminaBasecallsConverter Write 9,000,000 records. Elapsed time: 00:09:17s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:11:27 IlluminaBasecallsConverter Write 10,000,000 records. Elapsed time: 00:09:27s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:11:36 IlluminaBasecallsConverter Read 22,000,000 records. Elapsed time: 00:09:36s. Time for last 1,000,000: 22s. Last read position: */*
INFO 2017-08-15 11:11:37 IlluminaBasecallsConverter Write 11,000,000 records. Elapsed time: 00:09:36s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:11:46 IlluminaBasecallsConverter Write 12,000,000 records. Elapsed time: 00:09:45s. Time for last 1,000,000: 8s. Last read position: */*
INFO 2017-08-15 11:11:55 IlluminaBasecallsConverter Write 13,000,000 records. Elapsed time: 00:09:55s. Time for last 1,000,000: 9s. Last read position: */*
INFO 2017-08-15 11:11:56 IlluminaBasecallsConverter Read 23,000,000 records. Elapsed time: 00:09:56s. Time for last 1,000,000: 20s. Last read position: */*
INFO 2017-08-15 11:12:00 IlluminaBasecallsConverter Before explicit GC, Runtime.totalMemory()=7281311744
INFO 2017-08-15 11:12:00 IlluminaBasecallsConverter After explicit GC, Runtime.totalMemory()=7281311744
Exception in thread "pool-1-thread-6" ERROR 2017-08-15 11:12:04 IlluminaBasecallsConverter Failure encountered in worker thread; attempting to shut down remaining worker threads and terminate ...
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator.awaitWorkComplete(IlluminaBasecallsConverter.java:709)
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:318)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:230)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
picard.PicardException: Read the number of expected bins( 65600) but still had more elements in file( /Illumina/Basecalls/170808_SN218_0895_AHN3VWBCXY/Data/Intensities/L001/s_1_1107.clocs)
at picard.illumina.parser.readers.ClocsFileReader.hasNext(ClocsFileReader.java:150)
at picard.illumina.parser.PosParser$1.hasNext(PosParser.java:98)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:120)
at picard.illumina.parser.PerTileParser.maybeAdvance(PerTileParser.java:99)
at picard.illumina.parser.PerTileParser.next(PerTileParser.java:109)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-1-thread-1" java.lang.ArrayIndexOutOfBoundsException
at htsjdk.samtools.util.BlockCompressedOutputStream.deflateBlock(BlockCompressedOutputStream.java:357)
at htsjdk.samtools.util.BlockCompressedOutputStream.write(BlockCompressedOutputStream.java:250)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at htsjdk.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:206)
at htsjdk.samtools.util.BinaryCodec.writeByteBuffer(BinaryCodec.java:174)
at htsjdk.samtools.util.BinaryCodec.writeInt(BinaryCodec.java:220)
at htsjdk.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:132)
at htsjdk.samtools.BAMFileWriter.writeAlignment(BAMFileWriter.java:134)
at htsjdk.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:190)
at picard.illumina.IlluminaBasecallsToSam$SAMFileWriterWrapper.write(IlluminaBasecallsToSam.java:483)
at picard.illumina.IlluminaBasecallsToSam$SAMFileWriterWrapper.write(IlluminaBasecallsToSam.java:472)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$3.run(IlluminaBasecallsConverter.java:831)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-1-thread-5" java.lang.ArrayIndexOutOfBoundsException
at htsjdk.samtools.util.BlockCompressedOutputStream.deflateBlock(BlockCompressedOutputStream.java:357)
at htsjdk.samtools.util.BlockCompressedOutputStream.write(BlockCompressedOutputStream.java:250)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at htsjdk.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:206)
at htsjdk.samtools.util.BinaryCodec.writeByteBuffer(BinaryCodec.java:174)
at htsjdk.samtools.util.BinaryCodec.writeInt(BinaryCodec.java:220)
at htsjdk.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:131)
at htsjdk.samtools.BAMFileWriter.writeAlignment(BAMFileWriter.java:134)
at htsjdk.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:190)
at picard.illumina.IlluminaBasecallsToSam$SAMFileWriterWrapper.write(IlluminaBasecallsToSam.java:483)
at picard.illumina.IlluminaBasecallsToSam$SAMFileWriterWrapper.write(IlluminaBasecallsToSam.java:472)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$3.run(IlluminaBasecallsConverter.java:831)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[Tue Aug 15 11:12:06 EDT 2017] picard.illumina.IlluminaBasecallsToSam done. Elapsed time: 10.12 minutes.
Runtime.totalMemory()=7301234688
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" picard.PicardException: Failure encountered in worker thread; see log for details.
at picard.illumina.IlluminaBasecallsConverter.doTileProcessing(IlluminaBasecallsConverter.java:321)
at picard.illumina.IlluminaBasecallsToSam.doWork(IlluminaBasecallsToSam.java:230)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)
Exception in thread "pool-1-thread-3" picard.PicardException: IOException opening cluster binary file /Illumina/Basecalls/170808_SN218_0895_AHN3VWBCXY/Data/Intensities/BaseCalls/L001/s_1_1109.filter
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getByteIterator(MMapBackedIteratorFactory.java:66)
at picard.illumina.parser.readers.FilterFileReader.<init>(FilterFileReader.java:68)
at picard.illumina.parser.FilterParser$1.<init>(FilterParser.java:55)
at picard.illumina.parser.FilterParser.makeTileIterator(FilterParser.java:54)
at picard.illumina.parser.PerTileParser.advanceTile(PerTileParser.java:80)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:121)
at picard.illumina.parser.IlluminaDataProvider.hasNext(IlluminaDataProvider.java:104)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:554)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:314)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 12 more
Exception in thread "pool-1-thread-2" picard.PicardException: IOException opening cluster binary file /Illumina/Basecalls/170808_SN218_0895_AHN3VWBCXY/Data/Intensities/L001/s_1_1110.clocs
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:119)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getByteIterator(MMapBackedIteratorFactory.java:66)
at picard.illumina.parser.readers.ClocsFileReader.<init>(ClocsFileReader.java:85)
at picard.illumina.parser.PosParser.makeTileIterator(PosParser.java:83)
at picard.illumina.parser.PerTileParser.advanceTile(PerTileParser.java:80)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:121)
at picard.illumina.parser.PerTileParser.maybeAdvance(PerTileParser.java:99)
at picard.illumina.parser.PerTileParser.next(PerTileParser.java:109)
at picard.illumina.parser.IlluminaDataProvider.next(IlluminaDataProvider.java:133)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:555)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:314)
at picard.illumina.parser.readers.MMapBackedIteratorFactory.getBuffer(MMapBackedIteratorFactory.java:113)
... 13 more
Exception in thread "pool-1-thread-7" picard.PicardException: Error reading from file /Project/CorrectCapture/s_1_1111_barcode.txt
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:120)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:85)
at picard.util.AbstractInputParser.advance(AbstractInputParser.java:44)
at htsjdk.samtools.util.AbstractIterator.hasNext(AbstractIterator.java:44)
at picard.illumina.parser.readers.BarcodeFileReader.hasNext(BarcodeFileReader.java:42)
at picard.illumina.parser.BarcodeParser$BarcodeDataIterator.hasNext(BarcodeParser.java:69)
at picard.illumina.parser.PerTileParser.hasNext(PerTileParser.java:120)
at picard.illumina.parser.IlluminaDataProvider.hasNext(IlluminaDataProvider.java:104)
at picard.illumina.IlluminaBasecallsConverter$TileReader.process(IlluminaBasecallsConverter.java:554)
at picard.illumina.IlluminaBasecallsConverter$TileReadAggregator$2.run(IlluminaBasecallsConverter.java:657)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: htsjdk.samtools.util.RuntimeIOException: java.nio.channels.ClosedByInterruptException
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:74)
at picard.util.BasicInputParser.readNextLine(BasicInputParser.java:103)
... 12 more
Caused by: java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:163)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at htsjdk.samtools.util.BufferedLineReader.readLine(BufferedLineReader.java:70)
... 13 more