Manifest Format Specification

The manifest is a YAML configuration file that controls the biallelic inactivation analysis pipeline. It specifies input data files, reference datasets, and which discovery analyses to run.

File Structure

A manifest file has three main sections:

  1. Metadata: Project title and date

  2. References: Annotation and metadata files (genes, sample donors)

  3. Input: Genomic data files to analyze (SNVs, indels, copy number, etc.)

  4. Analyses: Discovery algorithms to execute

Basic Format

title: Human-readable project title
date: MM/DD/YYYY

ref:
  reference_name:
    path: path/to/file
    format_driver: driver_name

input:
  - path: path/to/file
    type: data_type
    format_driver: driver_name
    extra_driver_args: {}

analyses:
  - name: analysis_name

Metadata Section

title: Test biallelic with some cBio portal data
date: 07/03/2022

Fields:

  • title (string, required): Human-readable name for the analysis project

  • date (string, required): Date of manifest creation (MM/DD/YYYY format recommended)

These fields appear in output files and logs to identify the analysis run.

Reference Section (ref)

Reference datasets must be loaded before input data. These typically include gene annotations and sample/donor metadata that are used to annotate aberrations and match samples to donors.

ref:
  genes:
    path: ../ref/gencode_sort.v19.bed.gz
    format_driver: bed
  sample_donors:
    path: data_mutations_mskcc_17.txt.gz
    format_driver: maf

Reference Types and Required Drivers:

genes (required)

Gene annotations with genomic coordinates and names.

  • Supported driver: bed

  • Input format: BED file with gene coordinates

  • Output: DataFrame with columns: chrom, start, end, gene, strand

sample_donors (required)

Sample-to-donor mapping and sample metadata (gender, cellularity, ploidy).

  • Supported drivers: maf, (custom drivers for other formats)

  • Must include columns for: sample_id, donor_id, gender (optional: cellularity, ploidy)

  • Example from MAF: extracts TUMOR_SAMPLE_BARCODE from file headers

Reference Specification:

Each reference entry contains:

  • path (string, required): Path to reference file (relative or absolute)

    • Relative paths: resolved relative to manifest directory

    • Absolute paths: used as-is

  • format_driver (string, required): Name of the driver module that can read this file

    • Available drivers: bed, maf, and custom drivers in biallelic/drivers/

    • Driver must implement method matching reference name (e.g., genes(), sample_donors())

Important:

  • sample_donors reference is mandatory - analysis fails without it

  • Relative file paths are resolved relative to the manifest directory

  • Missing references cause informative error messages

Input Section (input)

Input files contain the genomic aberrations to analyze. Multiple files can be specified, and the same file can be loaded as different aberration types.

Each input file is processed by a format driver that reads the file and converts it to a standardized Aberration DataFrame. This allows discovery analyses to work uniformly with data from diverse sources (MAF, VCF, BED, custom formats) without needing to understand each format individually. See Architecture Guide for details on how data harmonization enables this flexibility, and Development Guide for instructions on creating custom drivers.

input:
  - path: data_cna_hg19_17.seg.gz
    type: scna
    format_driver: simple_segments
    extra_driver_args: {}

  - path: data_mutations_mskcc_17.txt.gz
    type: snv
    format_driver: maf
    extra_driver_args: {}

  - path: data_mutations_mskcc_17.txt.gz
    type: indel
    format_driver: maf
    extra_driver_args: {}

Supported Aberration Types:

  • snv: Single nucleotide variants (point mutations)

  • indel: Insertions and deletions

  • sv: Structural variants (translocations, inversions, etc.)

  • scna: Somatic copy number alterations (segmentation format)

  • germ_snv: Germline variants

  • methyl: DNA methylation data

Input Specification:

Each input entry contains:

  • path (string, required): Path to input data file

    • Relative paths: resolved relative to manifest directory

    • Supports gzip-compressed files (.gz)

  • type (string, required): Category of aberrations in this file

    • Determines which driver method is called

    • One file can be loaded as multiple types (see example above)

  • format_driver (string, required): Name of driver module

    • Examples: maf, simple_segments, vcf_vep_ppcg

    • Driver must implement method matching type (e.g., snv(), scna())

  • extra_driver_args (dictionary, optional): Additional arguments for driver

    • Format-specific parameters (e.g., column indices, filters)

    • Driver documentation specifies available options

    • Default: empty dictionary {}

Order Matters:

  • Input files are processed in the order specified

  • All reference files must be loaded before input files

  • This affects logging and output organization

Analyses Section (analyses)

Analyses are discovery algorithms that process loaded aberrations to identify biallelic inactivation patterns.

analyses:
  - name: write_aberrations
  - name: write_sample_donor
  - name: annotate_snv
  - name: annotate_double_snv
  - name: annotate_indel
  - name: summary_oncoprint_png

Supported Analyses:

Core Discovery:

  • annotate_snv: Identify SNV + loss biallelic pairs

  • annotate_double_snv: Identify SNV + SNV biallelic pairs

  • annotate_indel: Identify indel-based biallelic hits

  • annotate_germ_snv: Handle germline SNVs in analysis

  • annotate_sv: Analyze structural variants

  • annotate_meth: Methylation-based inactivations

  • annotate_subclonal_snv: Subclonal SNV detection

  • homozygous_inactivations: Homozygous loss detection

Output Generation:

  • write_aberrations: Output annotated aberrations table

  • write_sample_donor: Output sample/donor metadata

  • summary_biallelic: Merge and summarize biallelic hits

  • summary_biallelic_ppcg: PPCG-specific summary with cleanup

  • summary_oncoprint_png: Generate oncoprint visualization

Utility:

  • load_aberrations: Load pre-computed aberrations from file

Analysis Specification:

Each analysis entry contains:

  • name (string, required): Name of the analysis module

    • Module must exist in biallelic/discovery/

    • Module must implement main() function

Execution Order:

Analyses execute in the order specified. Typical order:

  1. Annotation analyses (detect biallelic hits)

  2. Summary analyses (aggregate results)

  3. Output analyses (write files and visualizations)

Complete Example

Complete manifest example with TCGA BRCA data
title: Test biallelic with some cBio portal data
date: 07/03/2022
ref:
  genes:
    path: ../ref/gencode_sort.v19.bed.gz
    format_driver: bed
  sample_donors:
    path:  data_mutations_mskcc_17.txt.gz
    format_driver: maf
input:
  - path: data_cna_hg19_17.seg.gz
    type: scna
    format_driver: simple_segments
    extra_driver_args: {}
  - path: data_mutations_mskcc_17.txt.gz
    type: snv
    format_driver: maf
    extra_driver_args: {}
  - path: data_mutations_mskcc_17.txt.gz
    type: indel
    format_driver: maf
    extra_driver_args: {}

analyses:
  - name: write_aberrations
  - name: write_sample_donor
  - name: annotate_snv
  - name: annotate_double_snv
  - name: annotate_indel
  - name: summary_oncoprint_png

Key Points:

  • All paths are relative to the manifest directory (except those starting with /)

  • File compression (.gz) is detected automatically

  • Analyses receive all loaded aberrations as input

  • Output files are written to results/ subdirectory

Running Analysis

Once manifest is created, run analysis:

biallelic_inactivation /path/to/manifest.yaml

Output:

  • results/: Biallelic hits and summary tables

  • logs/: Detailed execution logs for debugging

Common Issues

“sample_donors reference not found”

  • Ensure sample_donors entry exists in ref section

  • Verify the referenced file contains sample/donor information

  • Check file path is correct and file exists

“Driver XXX doesn’t implement YYY”

  • Verify driver name matches available driver

  • Verify aberration type is supported by driver

  • Check driver module documentation

“File not found” errors

  • Relative paths are resolved from manifest directory

  • Use absolute paths if files are elsewhere

  • Use correct path separators for your OS

“Empty reference_map”

  • Ensure gene annotations are loaded successfully

  • Verify genes reference path is correct

  • Check gene file format matches driver

Advanced Topics

Custom Drivers

To add support for a new file format, create a custom driver module in biallelic/drivers/ with functions matching the aberration types needed:

# biallelic/drivers/my_format.py
def genes(file_path, logger):
    """Load gene annotations from custom format."""
    # Parse file and return DataFrame with:
    # chrom, start, end, gene, strand
    return gene_df

def snv(file_path, logger, reference_map):
    """Load SNVs from custom format."""
    # Parse file and return DataFrame with aberrations
    return snv_df

Then reference in manifest:

ref:
  genes:
    path: genes.my_format
    format_driver: my_format

Extra Driver Arguments

Some drivers accept additional parameters via extra_driver_args:

input:
  - path: variants.vcf.gz
    type: snv
    format_driver: vcf_vep_ppcg
    extra_driver_args:
      vep_column_index: 8
      min_consequence_impact: 2

Check driver documentation for available options.

Validation

Manifest files are validated on load:

✓ Required sections present (title, ref, input, analyses) ✓ Required fields in each entry ✓ Referenced files exist (relative paths checked from manifest directory) ✓ Referenced drivers available in biallelic/drivers/ and biallelic/discovery/ ✓ sample_donors reference exists and can be loaded

Invalid manifests fail fast with descriptive error messages.

Best Practices

  1. Use relative paths for portability across machines

  2. Keep manifest with data in same directory or subdirectory

  3. Document custom parameters in comments

  4. Include date for audit trail

  5. Test with subset of data before full analysis (e.g., single chromosome)

  6. Preserve manifest file alongside results for reproducibility

  7. Use meaningful title that identifies cohort/experiment

Manifest Template

# Template for new biallelic analysis
title: My Cancer Cohort - Biallelic Analysis
date: 10/29/2025

ref:
  genes:
    path: gene_annotations.bed.gz
    format_driver: bed
  sample_donors:
    path: sample_metadata.tsv
    format_driver: maf  # Or custom driver

input:
  - path: somatic_variants.maf.gz
    type: snv
    format_driver: maf
    extra_driver_args: {}

  - path: somatic_variants.maf.gz
    type: indel
    format_driver: maf
    extra_driver_args: {}

  - path: copy_number.seg
    type: scna
    format_driver: simple_segments
    extra_driver_args: {}

analyses:
  - name: write_aberrations
  - name: write_sample_donor
  - name: annotate_snv
  - name: annotate_double_snv
  - name: annotate_indel
  - name: summary_biallelic
  - name: summary_oncoprint_png