A vast amount of biological multi-omics data is generated worldwide at any given point; this data has enormous potential for discovery and reusability for various R&D projects. However, the increasing volume and complexity of handling omics data go hand-in-hand. One of the challenges with analyzing and interpreting omics data is extracting meaningful insights from large-scale and high-dimensional data sets from multiple sources. Making sense of omics data requires advanced methods of data reduction and visualization techniques that can only be achieved with powerful data analytics tools, methods, and software. There is a lack of good practice pertaining to software and database usage which is the main source of computational irreproducibility, especially while analyzing very large datasets. Even the smallest variations across computational platforms contribute to irreproducibility. Handling pipelines, a large number of software packages, and dealing with hundreds of intermediate files produced by individual tools are some of the many difficulties. Hardware fluctuations in these types of pipelines, combined with poor error handling could result in considerable readout instability.

What Is Nextflow?

A solution designed to address numerical instability, efficient parallel execution, error tolerance, execution provenance, and traceability is Nextflow.  Nextflow is a workflow management system that uses Docker technology for the multi-scale handling of containerized computation. It is a domain-specific language that enables reproducible and parallel processing of pipelines.

Interesting, right? Continue to read this blog to know how we use it on Polly to handle pipelines effectively, saving both cost and time.

What is Nextflow Used For?

1. Nextflow allows you to code in any language like R, Python, Bash, etc.
2. It allows you to run your processes in parallel with channels, thus saving the total runtime to a great extent.
3. It has its own version control system. Whatever you do with a Nextflow script will be logged into separate log files in separate folders per session. This helps when you have to spend a lot of time debugging your code.
4. It has its own execution report generation system with tracing and visualizations.
5. It provides flexibility with containers and deployment. You can run your script using customized docker containers or conda environments, etc. To deploy, Nextflow supports a wide array of platforms like AWS, Google Cloud, and many more!
6. It allows you to resume a workflow for additions/corrections in the code.
7. The Nextflow process can be launched either in a local computer or an EC2 instance. The latter is suggested for heavy or long-running workloads.
8. Nextflow encourages workflow containerization i.e. each compute task is executed in its own Docker container.

Nextflow on Polly:

• Multi-processing jobs on Polly that have diverse machine needs and demand high computing time can be converted into the Nextflow pipeline, for an effective run.
• As the code uses modularized processes, it is more adaptable since individual modules may be plugged into and used in other pipelines, increasing its reproducibility and flexibility.
• For computationally intensive analyses to process enormous numbers of data and metadata, resource optimization is a major bonus, which in turn helps save costs.
• With parallel processing of pipeline/sub-processes within a pipeline with the support of Polly's computational resources, there is a significant reduction in the execution time.
• It offers users interactive reports, timelines, DAGs, and trace documents—this will help with monitoring and boost pipeline efficiency through better planning.
• Dockerisation of codes is not required. This reduces effort and enables better portability.

Comparison of Polly Cli Jobs with and Without Nextflow

‍

We hope you found this blog useful! If you have any questions on how we use Nextflow in Polly or would like to know more about Polly, reach out to us and we will be happy to answer them!

‍