Demystifying Bioinformatics Pipelines

What is a Bioinformatics Pipeline?

Bioinformatics—used extensively in genomics, pathology, and drug discovery—combines mathematical and computational methods to collect, classify, store, and analyze large and complex biological data. The set of biological data analysis operations executed in a predefined order is commonly referred to as a “bioinformatics pipeline”. In other words, a bioinformatics pipeline is an analysis workflow that takes input data files in unprocessed raw form through a series of transformations to produce output data in a human-interpretable form.

Typically, a bioinformatics pipeline consists of four components: 1) a user interface; 2) a core workflow framework; 3) input and output data; and 4) downstream scientific insights.

The core framework contains a variety of third-party software tools and in-house scripts wrapped into specific workflow steps. The steps are executed in a particular environment via a user interface, taking raw experimental data, reference files, and metadata as inputs. The resulting output data is then used to drive scientific insights through downstream advanced results, visualization, and interpretation.

Bioinformatics Pipelines in R&D

Despite the existence of highly sophisticated data pipelines, there is a frequent need in R&D to create ad-hoc bioinformatics pipelines either for prototyping and proof-of-concept purposes or to integrate newly published tools and methods since existing pipelines are not easily customizable. As the pipelines grow with additional steps, managing and maintaining the necessary tools becomes more difficult. Moreover, the complex and rapidly changing nature of biological databases, experimental techniques, and analysis tools makes reproducing, extending, and scaling pipelines a significant challenge.

Scientists, bioinformaticians, and lab managers are tasked with designing their pipelines and identifying the gaps within their frameworks. The best approach to prioritizing efforts depends highly on the operational need, the scientific scope, and the state of the bioinformatic pipeline. The very first step, however, is to understand the evolution of a bioinformatics pipeline.

The 5 Phases of a Bioinformatics Pipeline

A bioinformatics pipeline evolves through five phases. Pipeline stakeholders first seek to explore and collect the essential components, including raw data, tools, and references (Conception Phase). Then, they automate the analysis steps and investigate pipeline results (Survival Phase). Once satisfied, they move on to seek reproducibility and robustness (Stability Phase), extensibility (Success Phase), and finally scalability (Significance Phase). Below is a figure of the evolution at-a-glance as well as a description of each phase.

1. Conception Phase: Exploration of Requirements and Science |  The Conception Phase focuses on the basic needs of a new bioinformatics pipeline. It consists of exploring all the dependencies (software tools and reference databases), assessing adequate input data and metadata, and running the different steps individually. Building a minimum viable pipeline requires selecting the proper processing steps to run in the correct order and get the appropriate outputs.
2. Survival Phase: Automated Minimum Viable Analysis Workflow | The Survival Phase focuses on having an automated minimum viable analysis workflow. The process of analysis automation consists of streamlining the data processing steps to build a consistent analysis pipeline after setting the needed dependencies and reference databases. It ensures that the pipeline can be run as one transaction using a single entry point, which simplifies the analysis execution and the pipeline results investigation.
3. Stability Phase: Reproducibility of Pipeline Results |  The Stability Phase focuses on the reproducibility of the pipeline results, which is a recognized concern in biological research and is even more critical in a clinical setting. To ensure a full reproducibility of pipeline execution, the software dependencies should be managed automatically and the data should be indexed and searchable. Ideally, a pipeline user should only define the input data, execute the pipeline, and control the downstream analysis without worrying about the code versioning, the dependencies management, and the data organization.
4. Success Phase: Extensibility of Pipeline Composition |  The Success Phase focuses on the extensibility of pipeline composition, which is a significant concern for research labs that need to keep up-to-date and adapt to continuously evolving analysis practices. Extensible pipelines can be easily modified and extended to define new custom pipelines including new tools or a custom selection of steps.
5. Significance Phase: Scalability for Multiple Users and Large Datasets | The Significance Phase focuses on pipeline scalability. To be scalable, pipelines should be organized and optimized to facilitate multi-user interactions and support running resource-consuming processes and storing large-scale data.