I am trying to run GenotypeGVCFs over a couple of hundred whole genomes. Besides the fact that it uses obscene amounts of RAM, I noticed that it was failing with the message:
ERROR MESSAGE: Timeout of 30000 milliseconds was reached while trying to acquire a lock on file blah.gvcf.idx. Since the GATK uses non-blocking lock acquisition calls that are not supposed to wait, this implies a problem with the file locking support in your operating system.
Now, this is Linux, and the file is on a local XFS filesystem, so there is clearly no problem with file locking support. Rather, what is causing the problem is the garbage collector, which is causing pauses of 500 seconds or more when it tries to collect 150GB worth of heap. Now, adding the --disable_auto_index_creation_and_locking_when_reading_rods option helps, but it isn't elegant. I would suggest that in order to cope with GC pauses like this, GATK should retry acquiring the lock a few times, rather than bombing out immediately.