Hello,
Been trying to run Malt build to generate a all-bacteria reference database using roughly 50Gb of reference sequences. Initially it gave me a lot of memory issues, even needing to go up to a >1TB node on our server to allow it to run without a ‘memory head dump’ crash.
But now that it’s finally running it abruptly crashes with a non-descriptive error “killed” after trying to procure ~440Gb of memory
Version MALT (version 0.5.1, built 19 Jan 2021)
Author(s) Daniel H. Huson
Copyright (C) 2020 Daniel H. Huson. This program comes with ABSOLUTELY NO WARRANTY.
Classifications to use: Taxonomy
Reference sequence type set to: DNA
Seed shape(s): 111110111011110110111111
Deleting index files: 3
Number input files: 1
Loading FastA files:
10% 100% (862.2s)
Number of sequences: 739,440
Number of letters:58,050,232,143
BUILDING table (0)...
Seeds found: 58,033,225,023
tableSize= 2,147,483,639
hashMask.length=31
maxHitsPerHash set to: 1000
Initializing arrays...
100% (0.0s)
Analysing seeds...
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (3103.9s)
Number of low-complexity seeds skipped: 563,788,066
Allocating hash table...
10% 20% 30% 40% 50% 60% ^A70% 80% 90% 100% (967.8s)
Total keys used: 2,147,442,655
Total seeds matched:57,292,023,863
Total seeds dropped: 41,019,979
Opening file: /mnt/archgen/users/lesley_sitter/Test_dir/Malt_output/refDB/malt_index/table0.db
Allocating: 434.9 GB
Killed
Can anyone help clarify what causes "killed’ as an error message? I’m assume it’s still related to the memory, but it’s been running on a 2TB node, so seems unlikely that it’s unable to allocate 440GB on a 2TB node. I’m running Jdk 15.0.2 so also can’t be a 32bit java issue (i.e. only being able to allocate a max of 4Gb by default)… so i’m really at a loss what’s even going on.
Any help is appreciated.
Cheers,
Lesley