Create-accession-db failure

Bug Reports
1

Megan UE shipped with the "create-accession-db” script (I just want to create a taxonomic database only), but the following SQL error message jams the process at the merging step:

org.sqlite.SQLiteException: [SQLITE_TOOBIG] String or BLOB exceeds size limit (statement too long)

Detailed log:

./create-accession-db -c Taxonomy -i accessionmap202309.map -o accessionmap202309.db --nr nr.faa -v --threads 32
CreateMappingDB - Create MappingDB files for MEGAN
Version MEGAN Ultimate Edition (version 6.25.2, built 13 Sep 2023)

Java version: 20.0.2; max memory: 1000G
Parsing input files: accessionmap202309.map
Processing file: accessionmap202309.map
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (559.7s)
Taxonomy:1,193,208,055 from file: accessionmap202309.map
Loading ncbi.map: 2,396,736
Loading ncbi.tre: 2,396,740
Using LCA for Taxonomy
Processing file: nr.faa
10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (4,569.9s)
Taxonomy: 380,349,598
Creating mappings table: init
Merging all:
Caught:
org.sqlite.SQLiteException: [SQLITE_TOOBIG] String or BLOB exceeds size limit (statement too long)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.core.DB.newSQLException(DB.java:1179)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.core.DB.newSQLException(DB.java:1190)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.core.DB.throwex(DB.java:1150)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.core.NativeDB.prepare_utf8(Native Method)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.core.NativeDB.prepare(NativeDB.java:126)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.core.DB.prepare(DB.java:264)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.jdbc3.JDBC3Statement.lambda$execute$0(JDBC3Statement.java:51)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:454)
at org.xerial.sqlitejdbc@3.42.0.0/org.sqlite.jdbc3.JDBC3Statement.execute(JDBC3Statement.java:40)
at megan/megan.accessiondb.CreateAccessionMappingDatabase.execute(CreateAccessionMappingDatabase.java:355)
at megan6u/megan6u.tools.utils.CreateMappingDB.A(Unknown Source)
at megan6u/megan6u.tools.utils.CreateMappingDB.A(Unknown Source)
at megan6u/megan6u.tools.utils.CreateMappingDB.main(Unknown Source

2

I am working on this. We are putting together a set of SOPs for creating a MEGAN mapping database. I am rewriting the code for creating the DB to deal with the issue that you raised.

3

I’m going to write a new program for this. It will merge a given set of classifications and write them out to a file as a table. Then the sqlite3 program will be used to import the mapping table from the file. This will avoid any issues that Java is having with writing to a sqlite file.
I will aim to make it available at the end of this week or early next week.

4

Thank you in advance! It would be a great help to keep the mapping DB up to date!