.. index:: Manifest

.. _manifest:

Manifest Format Specification
=============================

The manifest is a YAML configuration file that controls the biallelic inactivation
analysis pipeline. It specifies input data files, reference datasets, and which
discovery analyses to run.

File Structure
--------------

A manifest file has three main sections:

1. **Metadata**: Project title and date
2. **References**: Annotation and metadata files (genes, sample donors)
3. **Input**: Genomic data files to analyze (SNVs, indels, copy number, etc.)
4. **Analyses**: Discovery algorithms to execute

Basic Format
~~~~~~~~~~~~

.. code-block:: yaml

    title: Human-readable project title
    date: MM/DD/YYYY

    ref:
      reference_name:
        path: path/to/file
        format_driver: driver_name

    input:
      - path: path/to/file
        type: data_type
        format_driver: driver_name
        extra_driver_args: {}

    analyses:
      - name: analysis_name

Metadata Section
----------------

.. code-block:: yaml

    title: Test biallelic with some cBio portal data
    date: 07/03/2022

**Fields:**

- ``title`` (string, required): Human-readable name for the analysis project
- ``date`` (string, required): Date of manifest creation (MM/DD/YYYY format recommended)

These fields appear in output files and logs to identify the analysis run.

Reference Section (ref)
------------------------

Reference datasets must be loaded before input data. These typically include
gene annotations and sample/donor metadata that are used to annotate aberrations
and match samples to donors.

.. code-block:: yaml

    ref:
      genes:
        path: ../ref/gencode_sort.v19.bed.gz
        format_driver: bed
      sample_donors:
        path: data_mutations_mskcc_17.txt.gz
        format_driver: maf

**Reference Types and Required Drivers:**

``genes`` (required)
    Gene annotations with genomic coordinates and names.

    - Supported driver: ``bed``
    - Input format: BED file with gene coordinates
    - Output: DataFrame with columns: chrom, start, end, gene, strand

``sample_donors`` (required)
    Sample-to-donor mapping and sample metadata (gender, cellularity, ploidy).

    - Supported drivers: ``maf``, (custom drivers for other formats)
    - Must include columns for: sample_id, donor_id, gender (optional: cellularity, ploidy)
    - Example from MAF: extracts TUMOR_SAMPLE_BARCODE from file headers

**Reference Specification:**

Each reference entry contains:

- ``path`` (string, required): Path to reference file (relative or absolute)

  - Relative paths: resolved relative to manifest directory
  - Absolute paths: used as-is

- ``format_driver`` (string, required): Name of the driver module that can read this file

  - Available drivers: ``bed``, ``maf``, and custom drivers in ``biallelic/drivers/``
  - Driver must implement method matching reference name (e.g., ``genes()``, ``sample_donors()``)

**Important:**

- ``sample_donors`` reference is **mandatory** - analysis fails without it
- Relative file paths are resolved relative to the manifest directory
- Missing references cause informative error messages

Input Section (input)
----------------------

Input files contain the genomic aberrations to analyze. Multiple files can be
specified, and the same file can be loaded as different aberration types.

Each input file is processed by a **format driver** that reads the file and converts it
to a standardized Aberration DataFrame. This allows discovery analyses to work uniformly
with data from diverse sources (MAF, VCF, BED, custom formats) without needing to
understand each format individually. See :ref:`architecture` for details on how data
harmonization enables this flexibility, and :ref:`developing` for instructions on
creating custom drivers.

.. code-block:: yaml

    input:
      - path: data_cna_hg19_17.seg.gz
        type: scna
        format_driver: simple_segments
        extra_driver_args: {}

      - path: data_mutations_mskcc_17.txt.gz
        type: snv
        format_driver: maf
        extra_driver_args: {}

      - path: data_mutations_mskcc_17.txt.gz
        type: indel
        format_driver: maf
        extra_driver_args: {}

**Supported Aberration Types:**

- ``snv``: Single nucleotide variants (point mutations)
- ``indel``: Insertions and deletions
- ``sv``: Structural variants (translocations, inversions, etc.)
- ``scna``: Somatic copy number alterations (segmentation format)
- ``germ_snv``: Germline variants
- ``methyl``: DNA methylation data

**Input Specification:**

Each input entry contains:

- ``path`` (string, required): Path to input data file

  - Relative paths: resolved relative to manifest directory
  - Supports gzip-compressed files (.gz)

- ``type`` (string, required): Category of aberrations in this file

  - Determines which driver method is called
  - One file can be loaded as multiple types (see example above)

- ``format_driver`` (string, required): Name of driver module

  - Examples: ``maf``, ``simple_segments``, ``vcf_vep_ppcg``
  - Driver must implement method matching type (e.g., ``snv()``, ``scna()``)

- ``extra_driver_args`` (dictionary, optional): Additional arguments for driver

  - Format-specific parameters (e.g., column indices, filters)
  - Driver documentation specifies available options
  - Default: empty dictionary ``{}``

**Order Matters:**

- Input files are processed in the order specified
- All reference files must be loaded before input files
- This affects logging and output organization

Analyses Section (analyses)
-----------------------------

Analyses are discovery algorithms that process loaded aberrations to identify
biallelic inactivation patterns.

.. code-block:: yaml

    analyses:
      - name: write_aberrations
      - name: write_sample_donor
      - name: annotate_snv
      - name: annotate_double_snv
      - name: annotate_indel
      - name: summary_oncoprint_png

**Supported Analyses:**

Core Discovery:

- ``annotate_snv``: Identify SNV + loss biallelic pairs
- ``annotate_double_snv``: Identify SNV + SNV biallelic pairs
- ``annotate_indel``: Identify indel-based biallelic hits
- ``annotate_germ_snv``: Handle germline SNVs in analysis
- ``annotate_sv``: Analyze structural variants
- ``annotate_meth``: Methylation-based inactivations
- ``annotate_subclonal_snv``: Subclonal SNV detection
- ``homozygous_inactivations``: Homozygous loss detection

Output Generation:

- ``write_aberrations``: Output annotated aberrations table
- ``write_sample_donor``: Output sample/donor metadata
- ``summary_biallelic``: Merge and summarize biallelic hits
- ``summary_biallelic_ppcg``: PPCG-specific summary with cleanup
- ``summary_oncoprint_png``: Generate oncoprint visualization

Utility:

- ``load_aberrations``: Load pre-computed aberrations from file

**Analysis Specification:**

Each analysis entry contains:

- ``name`` (string, required): Name of the analysis module

  - Module must exist in ``biallelic/discovery/``
  - Module must implement ``main()`` function

**Execution Order:**

Analyses execute in the order specified. Typical order:

1. Annotation analyses (detect biallelic hits)
2. Summary analyses (aggregate results)
3. Output analyses (write files and visualizations)

Complete Example
----------------

.. literalinclude:: ../test/data/test_cbio/manifest.yaml
    :language: yaml
    :caption: Complete manifest example with TCGA BRCA data

Key Points:

- All paths are relative to the manifest directory (except those starting with ``/``)
- File compression (.gz) is detected automatically
- Analyses receive all loaded aberrations as input
- Output files are written to ``results/`` subdirectory

Running Analysis
----------------

Once manifest is created, run analysis:

.. code-block:: bash

    biallelic_inactivation /path/to/manifest.yaml

Output:

- ``results/``: Biallelic hits and summary tables
- ``logs/``: Detailed execution logs for debugging

Common Issues
~~~~~~~~~~~~~

**"sample_donors reference not found"**
    - Ensure ``sample_donors`` entry exists in ``ref`` section
    - Verify the referenced file contains sample/donor information
    - Check file path is correct and file exists

**"Driver XXX doesn't implement YYY"**
    - Verify driver name matches available driver
    - Verify aberration type is supported by driver
    - Check driver module documentation

**"File not found" errors**
    - Relative paths are resolved from manifest directory
    - Use absolute paths if files are elsewhere
    - Use correct path separators for your OS

**"Empty reference_map"**
    - Ensure gene annotations are loaded successfully
    - Verify ``genes`` reference path is correct
    - Check gene file format matches driver

Advanced Topics
---------------

Custom Drivers
~~~~~~~~~~~~~~

To add support for a new file format, create a custom driver module in
``biallelic/drivers/`` with functions matching the aberration types needed:

.. code-block:: python

    # biallelic/drivers/my_format.py
    def genes(file_path, logger):
        """Load gene annotations from custom format."""
        # Parse file and return DataFrame with:
        # chrom, start, end, gene, strand
        return gene_df

    def snv(file_path, logger, reference_map):
        """Load SNVs from custom format."""
        # Parse file and return DataFrame with aberrations
        return snv_df

Then reference in manifest:

.. code-block:: yaml

    ref:
      genes:
        path: genes.my_format
        format_driver: my_format

Extra Driver Arguments
~~~~~~~~~~~~~~~~~~~~~~

Some drivers accept additional parameters via ``extra_driver_args``:

.. code-block:: yaml

    input:
      - path: variants.vcf.gz
        type: snv
        format_driver: vcf_vep_ppcg
        extra_driver_args:
          vep_column_index: 8
          min_consequence_impact: 2

Check driver documentation for available options.

Validation
----------

Manifest files are validated on load:

✓ Required sections present (title, ref, input, analyses)
✓ Required fields in each entry
✓ Referenced files exist (relative paths checked from manifest directory)
✓ Referenced drivers available in biallelic/drivers/ and biallelic/discovery/
✓ sample_donors reference exists and can be loaded

Invalid manifests fail fast with descriptive error messages.

Best Practices
--------------

1. **Use relative paths** for portability across machines
2. **Keep manifest with data** in same directory or subdirectory
3. **Document custom parameters** in comments
4. **Include date** for audit trail
5. **Test with subset** of data before full analysis (e.g., single chromosome)
6. **Preserve manifest file** alongside results for reproducibility
7. **Use meaningful title** that identifies cohort/experiment

Manifest Template
-----------------

.. code-block:: yaml

    # Template for new biallelic analysis
    title: My Cancer Cohort - Biallelic Analysis
    date: 10/29/2025

    ref:
      genes:
        path: gene_annotations.bed.gz
        format_driver: bed
      sample_donors:
        path: sample_metadata.tsv
        format_driver: maf  # Or custom driver

    input:
      - path: somatic_variants.maf.gz
        type: snv
        format_driver: maf
        extra_driver_args: {}

      - path: somatic_variants.maf.gz
        type: indel
        format_driver: maf
        extra_driver_args: {}

      - path: copy_number.seg
        type: scna
        format_driver: simple_segments
        extra_driver_args: {}

    analyses:
      - name: write_aberrations
      - name: write_sample_donor
      - name: annotate_snv
      - name: annotate_double_snv
      - name: annotate_indel
      - name: summary_biallelic
      - name: summary_oncoprint_png