SQLite error when building MALT database with v0.50

Bug Reports
1

Hello,

I have been trying to build a MALT database using the latest version available from the website (v0.5; downloaded in Sept. 2020) and the SQLite based mapping files from the MEGAN homepage. However, MALT always finishes with error message regarding the SQLite database being locked.

Version MALT (version 0.5.0, built 5 Aug 2020)
Author(s) Daniel H. Huson
Copyright (C) 2020 Daniel H. Huson. This program comes with ABSOLUTELY NO WARRANTY.
Reference sequence type set to: DNA
Seed shape(s): 111110111011110110111111
Deleting index files: 3
Number input files: 332
Loading FastA files:
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (110.1s)
Number of sequences: 55,546
Number of letters: 9,917,988,634
BUILDING table (0)…
Seeds found: 9,916,711,076
tableSize= 2,147,483,639
hashMask.length=31
maxHitsPerHash set to: 1000
Initializing arrays…
100% (0.0s)
Analysing seeds…
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (469.9s)
Number of low-complexity seeds skipped: 134,132,060
Allocating hash table…
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (464.2s)
Total keys used: 2,108,615,361
Total seeds matched: 9,780,048,046
Total seeds dropped: 1,172,171
Opening file: …/…/built/refseq/fungi_20200929/table0.db
Allocating: 79.1 GB
Filling hash table…
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (994.6s)
Randomizing rows…
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (213.6s)
Writing file: …/…/built/refseq/fungi_20200929/table0.idx
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (348.6s)
Writing file: …/…/built/refseq/fungi_20200929/table0.db
Size: 79.1 GB
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (1905.5s)
Writing file: …/…/built/refseq/fungi_20200929/index0.idx
100% (0.7s)
SQLiteException: [SQLITE_BUSY] The database file is locked (database is locked)

My starting parameters are the following:

malt-build -J-Xmx200G -i <fasta files> -igff <gff files> --sequenceType DNA --index maltv050 -t 8 -mdb megan-map-Jul2020-2.db

Is this a problem with MALT and how it access the SQLite files or does it have something to do with how I call malt-build?

Thank you very much!

2

I am looking into this. I may have found the problem (the mapping file was produced using the option ``write ahead logging’’ turned on, which created a database that cannot be opened read-only, causing problems when opening the database in multiple threads). I will try to fix this by creating new mapping databases without the WAL option, as soon as possible.

3

Actually, you can fix this yourself, use the commandline program sqlite3 to open the file and turn off WAL:

linux$ sqlite3 megan-nucl-map-Jul2020.db
sqlite> PRAGMA journal_mode=DELETE;
sqlite> .quit

4

Thank you for the reply. I tried to turn off WAL like you suggested in your post and modified the mapping files using sqlite3, however, the malt-build still fails at the very some point with the same error message: “SQLiteException: [SQLITE_BUSY] The database file is locked (database is locked)”

The error message is independent whether I use single-threading or multi-threading to build the database.

Might there be any SQLite3 related issue?