.. index:: Architecture

.. _architecture:

Architecture Guide
==================

This document describes the overall architecture and design of the biallelic_py package.

System Overview
===============

The biallelic_py package is a modular, plugin-based pipeline for discovering biallelic
inactivation patterns in genomic data. It uses a manifest-driven configuration approach
to control the analysis workflow.

**System Overview**:

::

    Manifest YAML
        ↓
    Aberrations Orchestrator
        ↓
    Reference Data (genes, samples)
    Input Drivers (MAF, BED, VCF)
        ↓
    Discovery Analyses
        ↓
    Output Files & Visualizations

Core Components
===============

1. **Manifest Parser** (biallelic.bi)
   - Reads YAML configuration files
   - Validates manifest structure and references
   - Orchestrates entire workflow

2. **Input Drivers** (biallelic/drivers/)
   - Load data from various file formats
   - Convert to standard DataFrame representation
   - Examples: MAF, BED, VCF, segmentation files

3. **Data Models** (biallelic.models)
   - Enums for data types (Gender, AberrationType, etc.)
   - Classes for genomic events (Aberration, DoubleHit)
   - Classes for metadata (SampleDonor)

4. **Discovery Analyses** (biallelic/discovery/)
   - Detect biallelic inactivation patterns
   - Generate summary statistics
   - Create visualizations
   - Examples: SNV + loss, double SNV, indel pairing

5. **Logging System** (biallelic.logging)
   - Hierarchical logging to file and console
   - Progress tracking for long analyses
   - Debug information capture

6. **Utilities** (biallelic.misc)
   - File I/O (transparently handles gzip, BGZF)
   - Module discovery and dynamic loading
   - String utilities for genomic data
   - Visualization color palettes

Data Flow
=========

The typical analysis workflow:

.. code-block:: text

    Manifest YAML
        ↓
    Aberrations.load_refs()
        ↓ (Load genes, sample_donors from ref section)
    Reference Data loaded
        ↓
    Aberrations.load_contents()
        ↓ (Call input drivers for each file)
    Input Drivers (MAF, BED, VCF, etc.)
        ↓ (Convert to standard format)
    Aberration DataFrames
        ↓
    Aberrations.biallelic_inactivations()
        ↓ (Execute discovery analyses in order)
    Discovery Analyses
        ↓ (Each analysis processes all loaded data)
    Annotated Aberrations + Hits
        ↓
    Output Files + Visualizations

Data Harmonization: The Core Design Pattern
=============================================

One of the key architectural insights of biallelic_py is its **data harmonization**
mechanism. This pattern is fundamental to how the framework supports arbitrary input
data formats while maintaining simple, uniform downstream analyses.

**The Problem It Solves**

Genomic data comes in many formats:

- **MAF files**: Tab-separated mutation tables from cancer sequencing projects
- **VCF files**: Variant Call Format with flexible INFO fields
- **BED files**: Simple genomic coordinates with metadata
- **Custom formats**: Organization-specific data structures
- **Segmentation files**: Copy number variation data

Without harmonization, each discovery analysis would need to understand every possible
input format. This would lead to code duplication, maintenance nightmares, and brittle
analyses.

**How It Works**

The solution is elegant: **All input data is converted to a standardized Aberration
DataFrame structure before any analysis begins.** This happens in the Input Drivers layer:

1. **Diversity**: Files come in different formats
2. **Parsing**: Each driver reads its specific format
3. **Transformation**: Driver creates Aberration objects from the parsed data
4. **Standardization**: Aberration objects are converted to a DataFrame with consistent columns
5. **Analysis**: All discovery algorithms work with this standardized DataFrame

**Why This Matters**

Once data is harmonized to the Aberration DataFrame format:

- Discovery analyses don't care where data came from
- You can combine SNVs from a MAF file with copy numbers from a BED segmentation
- New data formats can be supported by adding a single driver
- Complex analyses work uniformly on all data types
- The framework is extensible without touching core code

**The Aberration Data Model**

The standard structure is defined by the :class:`biallelic.models.Aberration` class,
which includes fields like:

- Genomic coordinates: ``chrom``, ``start``, ``end``
- Event type: ``aberration_type``, ``aberration_subtype``
- Samples: ``sample_id``, ``gene``
- Optional metrics: ``vaf`` (variant allele frequency), ``n_copy`` (copy number)

Any driver, regardless of input format, produces output with these columns. The result
is a DataFrame where each row represents one aberration event, and all rows have the
same schema regardless of their source format.

**For Developers**

This architecture is crucial when extending biallelic_py:

- **Adding a new input format?** Create a driver that returns Aberration DataFrames
- **Adding a new analysis?** Work with the standardized Aberration DataFrame
- **No need to change analyses** when adding support for new data formats

See :ref:`developing` for detailed guides on implementing custom drivers and analyses.

Module Organization
====================

Core Package Structure
~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: text

    biallelic/
    ├── __init__.py              # Package initialization
    ├── __version__.py           # Version and metadata
    ├── models.py                # Data classes and enums
    ├── bi.py                    # Main orchestrator
    ├── commands.py              # CLI entry point
    ├── logging.py               # Logging utilities
    ├── misc.py                  # Utility functions
    ├── bgzf.py                  # BGZF compression
    ├── drivers/                 # Input format readers
    │   ├── __init__.py
    │   ├── maf.py              # MAF file driver
    │   ├── bed.py              # BED file driver
    │   └── ...                 # Other drivers
    └── discovery/              # Discovery analyses
        ├── __init__.py
        ├── annotate_snv.py      # SNV analysis
        ├── annotate_*.py        # Other analyses
        └── ...

Core Classes
~~~~~~~~~~~~

**Aberrations (bi.py)**
    Main orchestrator class. Responsibilities:
    - Parse manifest YAML
    - Manage reference data loading
    - Coordinate input driver execution
    - Execute discovery analyses
    - Handle error conditions

**SimpleLogger (logging.py)**
    Hierarchical logger. Responsibilities:
    - Output to file and console
    - Support sub-loggers for analysis stages
    - Manage log levels
    - Format log messages

**Data Classes (models.py)**
    Represent genomic information:
    - Gender, OmicsType, AberrationType, DoubleHitType (enums)
    - SampleDonor (sample metadata)
    - Aberration (genomic variant)
    - DoubleHit (biallelic event)

Plugin Architecture
===================

The package uses dynamic plugin discovery and loading for drivers and analyses.

Input Drivers
~~~~~~~~~~~~~

**Location**: ``biallelic/drivers/``

**Discovery**: Modules discovered by scanning directory for .py files

**Interface**: Each driver module implements methods matching aberration types:

.. code-block:: python

    def snv(file_path: str, logger, reference_map) -> pd.DataFrame:
        """Load SNVs from file."""
        # Return DataFrame with columns:
        # chrom, start, end, aberration_type, aberration_subtype,
        # sample_id, gene, vaf, ...
        pass

    def genes(file_path: str, logger) -> pd.DataFrame:
        """Load gene annotations."""
        # Return DataFrame with columns:
        # chrom, start, end, gene, strand
        pass

**Execution**: Called via ``get_module_method(drivers, "maf", "snv")``

Discovery Analyses
~~~~~~~~~~~~~~~~~~~

**Location**: ``biallelic/discovery/``

**Discovery**: Modules discovered by scanning directory for .py files

**Interface**: Each analysis module implements:

.. code-block:: python

    def main(
        aberration_list: List[pd.DataFrame],
        output_path: str,
        reference_map: Dict,
        title: str,
        logger: SimpleLogger
    ) -> None:
        """Run discovery analysis."""
        # Process aberration_list
        # Write results to output_path
        # Log progress to logger
        pass

**Execution**: Called in order from manifest analyses section

**Execution Context**:
    - All previous input files already loaded into aberration_list
    - Previous analyses have already executed
    - Results available for downstream analyses

Configuration Format
====================

Manifest YAML Structure
~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: yaml

    # Metadata (human-readable identifiers)
    title: Project Name
    date: MM/DD/YYYY

    # References (data loaded before inputs)
    ref:
      genes:
        path: relative/path/genes.bed.gz
        format_driver: bed
      sample_donors:
        path: relative/path/samples.txt
        format_driver: maf

    # Input (genomic data files)
    input:
      - path: relative/path/file.maf.gz
        type: snv                          # aberration type
        format_driver: maf                 # which driver loads this
        extra_driver_args: {}              # driver-specific options

    # Analyses (discovery algorithms)
    analyses:
      - name: annotate_snv                 # module in biallelic/discovery/
      - name: summary_biallelic

Path Resolution
~~~~~~~~~~~~~~~

- **Relative paths**: Resolved relative to manifest directory
- **Absolute paths**: Used as-is
- **Path checking**: Happens at load time with informative errors

Error Handling Strategy
=======================

The package uses specific exception types and informative messages:

**Validation Errors**
    Caught during manifest loading:
    - Missing required fields
    - Invalid file paths
    - Unknown drivers/analyses

**Runtime Errors**
    Caught during execution:
    - File I/O failures
    - Missing required references
    - Data format issues
    - Driver failures

**Error Messages**
    All errors include:
    - What went wrong (specific, not generic)
    - Where it happened (file, location)
    - How to fix it (actionable guidance)

Extensibility Points
====================

Users can extend the platform in these ways:

1. **Add Custom Input Driver**
    - Create ``biallelic/drivers/my_format.py``
    - Implement methods for data types needed
    - Reference in manifest ``format_driver: my_format``

2. **Add Custom Discovery Analysis**
    - Create ``biallelic/discovery/my_analysis.py``
    - Implement ``main()`` function
    - Reference in manifest ``analyses: - name: my_analysis``

3. **Custom Data Processing**
    - Subclass SimpleLogger for custom logging
    - Subclass Aberrations for custom workflows
    - Import utilities (xopen, color_palettes, etc.) for your analyses

Backward Compatibility
======================

All improvements maintain backward compatibility:

- ✓ Existing manifests work unchanged
- ✓ Existing driver/analysis modules work unchanged
- ✓ Existing output formats preserved
- ✓ API additions are purely additive
- ✓ No breaking changes to public classes

Performance Considerations
==========================

**Memory Usage**
    - All aberrations loaded into DataFrames
    - Large files may require significant RAM
    - Consider processing by chromosome for very large datasets

**Execution Time**
    - Input drivers dominate execution time
    - Discovery analyses are typically fast
    - Logging is minimal overhead

**File I/O**
    - Automatic gzip detection for compression
    - BGZF support for random access
    - xopen() provides transparent file handling

Design Patterns
===============

**Plugin Architecture**
    Dynamic loading of drivers and analyses allows extensibility
    without modifying core code.

**Orchestrator Pattern**
    Aberrations class orchestrates workflow, coordinating multiple
    components (drivers, analyses, logging).

**Strategy Pattern**
    Different drivers implement common interface for loading
    different file formats.

**Factory Pattern**
    Dynamic module discovery creates driver/analysis instances
    at runtime based on configuration.

**Builder Pattern**
    Manifest gradually builds analysis configuration as sections
    are parsed and validated.

Testing Strategy
================

Tests are organized by module:

- ``test/test_models.py`` - Data model tests
- ``test/test_logging.py`` - Logging functionality
- ``test/test_misc.py`` - Utility function tests
- ``test/test_drivers.py`` - Driver functionality
- ``test/test_bgzf.py`` - File I/O tests

Integration tests verify:
- Manifest parsing and validation
- Reference data loading
- Input data processing
- Discovery analysis execution

See Also
~~~~~~~~

- :ref:`developing`: Step-by-step guide for extending the package
- :ref:`manifest`: Complete manifest specification
- :ref:`api`: API reference for all classes and functions