Usage

PhenomeDB is a database, data processing, and analysis and visualisation platform for metabolomics data. The general usage of which is outlined below, where users import data, harmonise the data, build queries to integrate and stratify the data, scale, normalise, and batch correct the data, and run analyses and reports to analyse, visualise, and interpret the data. PhenomeDB provides python APIs and web-based UIs for these steps, including a novel QueryFactory for building and executing queries, and a novel PipelineFactory for building and executing pipelines via Apache-Airflow.

Tasks

PhenomeDB is structured around the concept of a phenomedb.task. Tasks are units of work that execute a specific function, such as importing annotations, harmonising metadata, or generating datasets on the basis of defined Queries. Tasks can be executed independently, or chained together into Pipelines using the PipelineFactory python API or UI. A full list of included Tasks is shown below, and guides to implementing new tasks can be in Development.

The inbuilt tasks for PhenomeDB are shown below. To view their parameters, follow the links to the relevant documentation page:

Title
Task class	Task description
`phenomedb.imports.ImportMetadata()`	Import a CSV sample metadata file
`phenomedb.imports.ImportSampleManifest()`	Import a Sample Manifest file
`phenomedb.imports.ImportDataLocations()`	Import a data locations file (deprecated)
`phenomedb.imports.ImportBrukerIVDRAnnotations()`	Import Bruker IVDr annotations
`phenomedb.imports.ImportPeakPantherAnnotations()`	Import PeakPantheR annotations
`phenomedb.imports.ImportTargetLynxAnnotations()`	Import TargetLynx annotations
`phenomedb.imports.XCMSFeatureImportTaskUnifiedCSV()`	Import XCMS features
`phenomedb.imports.ImportMetabolightsStudy()`	Import a Metabolights study
`phenomedb.imports.DownloadMetabolightsStudy()`	Download a Metabolights study
`phenomedb.metadata.HarmoniseMetadataField()`	Harmonise/curate a metadata field
`phenomedb.analysis.RunPCA()`	Run a PCA analysis
`phenomedb.analysis.RunPCPR2()`	Run a PCPR2 analysis
`phenomedb.analysis.RunMWAS()`	Run an MWAS analysis
`phenomedb.analysis.RunNPYCReport()`	Run an nPYc report
`phenomedb.analysis.RunXCMS()`	Run XCMS
`phenomedb.batch_correction.RunCombatCorrection()`	Run COMBAT correction
`phenomedb.batch_correction.RunNormResidualsMM()`	Run NormResidualsMixedModel correction
`phenomedb.batch_correction.RunNPYCBatchCorrection()`	Run LOWESS correction
`phenomedb.batch_correction.SaveBatchCorrection()`	Save a LOWESS corrected dataset
`phenomedb.compounds.ParseKEGGtoPubchemCIDTask()`	Parse KEGG to a PubChem CID lookup CSV file
`phenomedb.compounds.ParseHMDBXMLtoCSV()`	Parse HMDB XML to a lookup CSV file
`phenomedb.compounds.UpdateCompoundRefs()`	Look for and update the external database refs for existing compounds
`phenomedb.compounds.AddMissingClassyFireClasses()`	Look for and update the ClassyFire classes for existing compounds
`phenomedb.compounds.CleanROIFile()`	Clean an ROI file by checking the data matches online databases
`phenomedb.compounds.ImportROICompounds()`	Import compounds from a PeakPantheR ROI file
`phenomedb.compounds.ImportBrukerBILISACompounds()`	Import Bruker BI-LISA lipoprotein and lipid fractions from a source file
`phenomedb.compounds.ImportBrukerBiQuantCompounds()`	Import Bruker Bi-Quant-P compounds from a source file
`phenomedb.compounds.ExportCompoundsToCSV()`	Export all compounds to CSV
`phenomedb.pipelines.RebuildPipelinesFromDB()`	Rebuild the Airflow pipelines based on the DB entries
`phenomedb.pipelines.GenerateSingleTaskPipelines()`	Build the single-task pipelines for single-task execution
`phenomedb.pipelines.BasicSetup()`	Run the BasicSetup to populate the database with assays, projects, annotation methods etc
`phenomedb.pipelines.BatchCorrectionAssessmentPipelineGenerator()`	Build the BatchCorrectionAssessmentPipeline
`phenomedb.pipelines.RunBatchCorrectionAssessmentPipeline()`	Run the BatchCorrectionAssessmentPipeline
`phenomedb.pipelines.RunMWASMulti()`	Run multiple MWAS
`phenomedb.pipelines.ImportAllMetabolightsPipelineGenerator()`	Build a pipeline to import all data from Metabolights
`phenomedb.task.ManualSQL()`	Execute manual SQL
`phenomedb.cache.CreateSavedQueryDataframeCache()`	Create a SavedQuery Combined dataframe cache
`phenomedb.cache.CreateSavedQuerySummaryStatsCache()`	Create a SavedQuery summary stats cache
`phenomedb.cache.CreateTaskViewCache()`	Create the task-view cache (deprecated)
`phenomedb.cache.RemoveUntransformedDataFromCache()`	Remove untransformed data from the cache (clean up task)
`phenomedb.cache.MoveTaskOutputToCache()`	Move the task output from the db to cache (clean up task)

For more information on tasks, including implementing your own, please head over to the Development page.

The Apache-Airflow interface

To access Apache-Airflow, once the system is running, open your web browser and navigate to http://localhost:8080/. The default username and password are admin and testpass.

From here, pipelines (‘DAGs’ in Airflow) for individual tasks can be parameterised, executed, and monitored, and the various PhenomeDB views can be accessed.

Apache-Airflow is structured around the concepts of pipelines and pipeline runs (executions). You parameterise a pipeline run and then Airflow manages the execution. Output logs for each task in the pipeline can be inspected via the interface.

For more information regarding the usage of Apache-Airflow, please see the Apache-Airflow documentation.

Airflow UI home — Airflow home page showing registered pipelines (DAGs)

Airflow Pipeline Overview — Graphical view of the ImportPeakPantherAnnotations pipeline

Airflow Run Pipeline — View for running a pipeline, with example JSON for parameterising the import task.

Airflow Logs example — Example output of the TaskRun logs, viewed from within the Airflow interface

Tasks and Pipelines

Tasks can be executed individually using command line interface (cli.py), or via Airflow as part of a Pipeline.

Pipelines can be created, registered with Airflow, and executed via the PipelineFactory. Using this approach removes the requirements for manually writing Airflow DAG files.

The PipelineFactory UI can be used to created parameterised, hard-coded Pipelines/DAGs made of many tasks (executed sequentially).

PhenomeB PipelineFactory UI Example — Example of using a the phenomedb.modules.PipelineFactory UI to create a parameterised pipeline

See the phenomedb.pipeline_factory for more information, including how to build and execute pipelines via the python API.

Command line interface

The cli.py provides a way of running individual tasks. To use it either install a local Python installation or inside the scheduler docker container.

From the project root:

$ cd phenomedb
$ python cli.py -h

Importing analytical data and sample metadata

Two main analytical data import sources are supported - Metabolights format, and the nPYc-toolbox 3-file format, consisting of 3 separate sources of information:

Sample manifests: CSV files containing sample metadata subject as clinical factors, outcomes-of-interest, or covariates.
Feature metadata: CSV files containing feature metadata such as RT, m/z, and other feature-specific analytical metadata.
Study data files: CSV files containing analytical features (measurements) relating to the samples and features/annotated compounds.

Title
Task class	Task description
`phenomedb.imports.ImportMetadata()`	Import a CSV sample metadata file
`phenomedb.imports.ImportSampleManifest()`	Import a Sample Manifest file
`phenomedb.imports.ImportDataLocations()`	Import a data locations file (deprecated)
`phenomedb.imports.ImportBrukerIVDRAnnotations()`	Import Bruker IVDr annotations
`phenomedb.imports.ImportPeakPantherAnnotations()`	Import PeakPantheR annotations
`phenomedb.imports.ImportTargetLynxAnnotations()`	Import TargetLynx annotations
`phenomedb.imports.XCMSFeatureImportTaskUnifiedCSV()`	Import XCMS features
`phenomedb.imports.ImportMetabolightsStudy()`	Import a Metabolights study
`phenomedb.imports.DownloadMetabolightsStudy()`	Download a Metabolights study

See the phenomedb.imports for more information, and use the CLI or phenomedb.pipeline_factory to run them.

Harmonising sample metadata

In order to compare, integrate, and stratify data across multiple cohorts, the sample metadata must be harmonised. To do this, it is recommended to use the phenomedb.metadata.HarmoniseMetadataField, which enables the curation of unharmonised ‘raw’ metadata fields and values into harmonised ‘curated’ metadata fields and values. Please see the phenomedb.metadata module for more information.

Title
Task class	Task description
`phenomedb.metadata.HarmoniseMetadataField`	Harmonise/curate a metadata field

Importing compound metadata

PhenomeDB enables the storage of annotation metadata such as chemical references and classes, and has a data model and import processes capable of harmonising annotations to their analytical specificity.

phenomedb.models.Compound metadata can be imported from PeakPantheR region-of-interest files (ROI) files for LC-MS annotations. Recent versions for these can be found in ./phenomedb/data/compounds/.

To import the ROI compound data use the tasks phenomedb.compounds.ImportROICompounds and phenomedb.compounds.ImportROILipids.

IVDr annotation metadata can be imported using phenomedb.compounds.ImportBrukerBiLISACompounds and phenomedb.compounds.ImportBrukerBiQuantCompounds. The source data are available in ./phenomedb/data/compounds/

PhenomeDB Compound View — The Compound View, showing the imported information for one compound, with links to external databases

Title
Task class	Task description
`phenomedb.compounds.ParseKEGGtoPubchemCIDTask()`	Parse KEGG to a PubChem CID lookup CSV file
`phenomedb.compounds.ParseHMDBXMLtoCSV()`	Parse HMDB XML to a lookup CSV file
`phenomedb.compounds.UpdateCompoundRefs()`	Look for and update the external database refs for existing compounds
`phenomedb.compounds.AddMissingClassyFireClasses()`	Look for and update the ClassyFire classes for existing compounds
`phenomedb.compounds.CleanROIFile()`	Clean an ROI file by checking the data matches online databases
`phenomedb.compounds.ImportROICompounds()`	Import compounds from a PeakPantheR ROI file
`phenomedb.compounds.ImportBrukerBILISACompounds()`	Import Bruker BI-LISA lipoprotein and lipid fractions from a source file
`phenomedb.compounds.ImportBrukerBiQuantCompounds()`	Import Bruker Bi-Quant-P compounds from a source file
`phenomedb.compounds.ExportCompoundsToCSV()`	Export all compounds to CSV

Harmonising annotation metadata

In order to integrate annotations across projects, the annotations must be harmonised. PhenomeDB will attempt to do this automatically where possible, however in some cases it is necessary to manually harmonise annotations. To do this use the ‘Harmonise Annotations’ view.

PhenomeDB manual annotation harmonisation — The Harmonise Annotations View, where unharmonised annotations can be harmonised manually to enable cross-project comparisons

Creating and executing queries

PhenomeDB has a complex and rich querying system that enables users to define queries as a collection of filters and the conversion of the results of those filters to a dataset, enabling cross-project integration and stratification. For more information on the QueryFactory, including its Python API and UI, please head over to phenomedb.imports.

In short, users define queries, build the dataframe cache, and then that cache can be used in downstream analyses/tasks.

Creating queries can be done either via the Query Factory view or the QueryFactory Python API. In PhenomeDB Queries are created by chaining QueryFilter objects containing boolean operators and QueryMatches, which specifying the fields and comparison operators and values. An overview of this can be seen below. With the collection of QueryFilters and QueryMatches, the QueryFactory then calculates/transpiles the query definition into an SQLAlchemy query, and executes the query. The QueryFactory can then construct a combined-format and 3-file format dataset of the results, and store them in the PhenomeDB Cache, an extended version of Redis that enables file-system persistency of objects. Generating the dataframes can currently take a long time depending on the number of records the query returns, for this reason once the query has been defined the user should run the CreateSavedQueryDataframeCache task to execute the query and set it into the cache. This can be run manually via the interface or via the QueryFactory UI.

PhenomeDB QueryFactory QueryFilters and QueryMatches — The QueryFilter and QueryMatch architecture. Multiple QueryFilters can be added, each with AND or OR boolean operators. Each QueryFilter can have multiple QueryMatches, targeting a specific Model.property, with a specific comparison operator and value.

See the phenomedb.query_factory.QueryFactory for more information, including using the Python API.

Scaling, normalisation, and batch correction

In order to compare metabolite levels across different batches, projects, or assays, scaling/normalisation, transformation, and batch correction must be undertaken. The aim of these methods is to minimise inter-batch technical variation while maintaining inter-sample biological variation.

Two kinds of intensity values are stored in the database, raw, and SR-corrected. Raw are the ‘raw’, peak-picked intensities from the instruments. ‘SR-corrected’ is data that has been run-order corrected using study-reference QC-based LOWESS correction. Typically LC-MS data will need to be run-order corrected so it is advisable to use this SR-corrected data for LC-MS datasets.

Batch correction can be undertaken using the following tasks:

Title
Task class	Task description
`phenomedb.batch_correction.RunCombatCorrection()`	Run COMBAT correction
`phenomedb.batch_correction.RunNormResidualsMM()`	Run NormResidualsMixedModel correction
`phenomedb.batch_correction.RunNPYCBatchCorrection()`	Run LOWESS correction
`phenomedb.batch_correction.SaveBatchCorrection()`	Save a LOWESS corrected dataset

When running analyses, the following scaling and transformations can be executed as options to the analysis methods.

Title
Scaling methods	Transform methods
Mean-centred (mc)	Log10 (log)
Unit-variance (uv)	Square root (sqrt)
Median

Running analyses

Implemented analysis functions include:

Title
Task class	Task description
`phenomedb.analysis.RunPCA()`	Run a PCA analysis
`phenomedb.analysis.RunPCPR2()`	Run a PCPR2 analysis
`phenomedb.analysis.RunMWAS()`	Run an MWAS analysis
`phenomedb.analysis.RunNPYCReport()`	Run an nPYc report
`phenomedb.analysis.RunXCMS()`	Run XCMS

Individual analyses can be run via the AnalysisView page, where task runs can be parameterised and scheduled, and the results can be explored.

PhenomeDB AnalysisView list — Analyses can be executed against queries (and upstream task runs) using the AnalysisView. Parameters for the task run can be specified using the html form, including scaling and transformation steps and task-specific options. Previous task runs can be explored via a table.

The results of each analysis can be explored via a dedicated UI, with panels common to all analysis tasks with options to rerun the task, and options to download the input and output datasets.

PhenomeDB AnalysisView common — Each task run output view has the ability to re-run the task with new parameters, and explore and download the input and output datasets.

Each AnalysisTask also has specific charts and figures available to explore the results.

PhenomeDB RunPCA visualisation — Interactive visualisation of PCA outputs, including A: Scree plot, B: control panel to control the chart options, C: 2D scores plots, D, E, F: loadings plots.

PhenomeDB RunPCPr2 visualisation — Visualisation of PCPR2 results

PhenomeDB RunMWAS visualisation — Interactive visualisation of 1D MWAS outputs

PhenomeDB RunMWAS compare visualisation — Interactive visualisation of MWAS comparison heatmaps, where the results of two MWAS analyses can be compared, in this case comparing the age-associated metabolites of males and females