MSA Compression

[lrauschning]

Tracking issue for the MSA compression in this pipeline.

Tasks

Preview Give feedback

1. Implement PIGZ modules
2. Check which MSA tools can automagically produce compressed output
3. Change the MSA modules to produce compressed output:
4. MUSCLE5
5. MTMalign
6. TCOFFEE
7. FAMSA
8. MAFFT
9. MAGUS
10. Test behaviour with --resume

[lrauschning]

when compressing using a separate module, most compression tools will automatically delete the input file.
This behaviour makes sense, as otherwise there would be no space savings when keeping the uncompressed and compressed file simultaneously.
However, it breaks nextflows paradigm by having the running of one module affect the output of an earlier step, potentially breaking the resume support.
For this reason, the MSA modules should natively produce compressed output that is then uncompressed as needed.

[alessiovignoli]

The compression step is actually happening on the alignment outputs files. So it happens before the evaluate. It is a skipable step. And if it is run the files are actually saved as compressed. It should create a copy during compression because the un-compress version is actually passed to the evaluation step as that can not read compressed files.

[alessiovignoli]

From the documentations i could scavenge FAMSA seems the only one able to compress the outptut using a flag.

FAMSA: Option for compressing output aligment to gzip (-gz switch).

Might need to look at the help page for each program while implementing.

NB. could not find any documentation for MTMalign stand alone version.

[lrauschning]

Yes, MTMalign has neither documentation nor a publicly accessible source code repository. On the plus side, at least it compiles on my machine ^^.
I think just calling pigz in the module after the call to the MSA tool is fine, the only annoying thing is that for the singularity & biocontainers backend that requires us to construct a joint container of the MSA tool and pigz, unless anyone knows a way to use docker compose in nfcore modules.

[luisas]

I would not overcomplicate things now.
If the tools do not provide the compression, maybe we can just have the zip module as we currently have. At least for the first release and then see who uses the pipeline and if this is even necessary.

In any case, multipackage containers are doable https://github.com/BioContainers/multi-package-containers

But for now, I would build a clean and functional pipeline first. Then we can polish these things once we have everything else in place :)

[lrauschning]

Branch where I am updating the modules to produce compressed output:
https://github.com/lrauschning/modules/tree/msa-compression
Made a pull request to create mulled containers for some tools:
BioContainers/multi-package-containers#2867
Will do PR to nfcore once they are merged.

[luisas]

Closed by #124