Architecture Guide¶

This document describes the overall architecture and design of the biallelic_py package.

System Overview¶

The biallelic_py package is a modular, plugin-based pipeline for discovering biallelic inactivation patterns in genomic data. It uses a manifest-driven configuration approach to control the analysis workflow.

System Overview:

Manifest YAML
    ↓
Aberrations Orchestrator
    ↓
Reference Data (genes, samples)
Input Drivers (MAF, BED, VCF)
    ↓
Discovery Analyses
    ↓
Output Files & Visualizations

Core Components¶

Manifest Parser (biallelic.bi) - Reads YAML configuration files - Validates manifest structure and references - Orchestrates entire workflow
Input Drivers (biallelic/drivers/) - Load data from various file formats - Convert to standard DataFrame representation - Examples: MAF, BED, VCF, segmentation files
Data Models (biallelic.models) - Enums for data types (Gender, AberrationType, etc.) - Classes for genomic events (Aberration, DoubleHit) - Classes for metadata (SampleDonor)
Discovery Analyses (biallelic/discovery/) - Detect biallelic inactivation patterns - Generate summary statistics - Create visualizations - Examples: SNV + loss, double SNV, indel pairing
Logging System (biallelic.logging) - Hierarchical logging to file and console - Progress tracking for long analyses - Debug information capture
Utilities (biallelic.misc) - File I/O (transparently handles gzip, BGZF) - Module discovery and dynamic loading - String utilities for genomic data - Visualization color palettes

Data Flow¶

The typical analysis workflow:

Manifest YAML
    ↓
Aberrations.load_refs()
    ↓ (Load genes, sample_donors from ref section)
Reference Data loaded
    ↓
Aberrations.load_contents()
    ↓ (Call input drivers for each file)
Input Drivers (MAF, BED, VCF, etc.)
    ↓ (Convert to standard format)
Aberration DataFrames
    ↓
Aberrations.biallelic_inactivations()
    ↓ (Execute discovery analyses in order)
Discovery Analyses
    ↓ (Each analysis processes all loaded data)
Annotated Aberrations + Hits
    ↓
Output Files + Visualizations

Data Harmonization: The Core Design Pattern¶

One of the key architectural insights of biallelic_py is its data harmonization mechanism. This pattern is fundamental to how the framework supports arbitrary input data formats while maintaining simple, uniform downstream analyses.

The Problem It Solves

Genomic data comes in many formats:

MAF files: Tab-separated mutation tables from cancer sequencing projects
VCF files: Variant Call Format with flexible INFO fields
BED files: Simple genomic coordinates with metadata
Custom formats: Organization-specific data structures
Segmentation files: Copy number variation data

Without harmonization, each discovery analysis would need to understand every possible input format. This would lead to code duplication, maintenance nightmares, and brittle analyses.

How It Works

The solution is elegant: All input data is converted to a standardized Aberration DataFrame structure before any analysis begins. This happens in the Input Drivers layer:

Diversity: Files come in different formats
Parsing: Each driver reads its specific format
Transformation: Driver creates Aberration objects from the parsed data
Standardization: Aberration objects are converted to a DataFrame with consistent columns
Analysis: All discovery algorithms work with this standardized DataFrame

Why This Matters

Once data is harmonized to the Aberration DataFrame format:

Discovery analyses don’t care where data came from
You can combine SNVs from a MAF file with copy numbers from a BED segmentation
New data formats can be supported by adding a single driver
Complex analyses work uniformly on all data types
The framework is extensible without touching core code

The Aberration Data Model

The standard structure is defined by the biallelic.models.Aberration class, which includes fields like:

Genomic coordinates: chrom, start, end
Event type: aberration_type, aberration_subtype
Samples: sample_id, gene
Optional metrics: vaf (variant allele frequency), n_copy (copy number)

Any driver, regardless of input format, produces output with these columns. The result is a DataFrame where each row represents one aberration event, and all rows have the same schema regardless of their source format.

For Developers

This architecture is crucial when extending biallelic_py:

Adding a new input format? Create a driver that returns Aberration DataFrames
Adding a new analysis? Work with the standardized Aberration DataFrame
No need to change analyses when adding support for new data formats

See Development Guide for detailed guides on implementing custom drivers and analyses.

Module Organization¶

Core Package Structure¶

biallelic/
├── __init__.py              # Package initialization
├── __version__.py           # Version and metadata
├── models.py                # Data classes and enums
├── bi.py                    # Main orchestrator
├── commands.py              # CLI entry point
├── logging.py               # Logging utilities
├── misc.py                  # Utility functions
├── bgzf.py                  # BGZF compression
├── drivers/                 # Input format readers
│   ├── __init__.py
│   ├── maf.py              # MAF file driver
│   ├── bed.py              # BED file driver
│   └── ...                 # Other drivers
└── discovery/              # Discovery analyses
    ├── __init__.py
    ├── annotate_snv.py      # SNV analysis
    ├── annotate_*.py        # Other analyses
    └── ...

Core Classes¶

Aberrations (bi.py): Main orchestrator class. Responsibilities: - Parse manifest YAML - Manage reference data loading - Coordinate input driver execution - Execute discovery analyses - Handle error conditions
SimpleLogger (logging.py): Hierarchical logger. Responsibilities: - Output to file and console - Support sub-loggers for analysis stages - Manage log levels - Format log messages
Data Classes (models.py): Represent genomic information: - Gender, OmicsType, AberrationType, DoubleHitType (enums) - SampleDonor (sample metadata) - Aberration (genomic variant) - DoubleHit (biallelic event)

Plugin Architecture¶

The package uses dynamic plugin discovery and loading for drivers and analyses.

Input Drivers¶

Location: biallelic/drivers/

Discovery: Modules discovered by scanning directory for .py files

Interface: Each driver module implements methods matching aberration types:

def snv(file_path: str, logger, reference_map) -> pd.DataFrame:
    """Load SNVs from file."""
    # Return DataFrame with columns:
    # chrom, start, end, aberration_type, aberration_subtype,
    # sample_id, gene, vaf, ...
    pass

def genes(file_path: str, logger) -> pd.DataFrame:
    """Load gene annotations."""
    # Return DataFrame with columns:
    # chrom, start, end, gene, strand
    pass

Execution: Called via get_module_method(drivers, "maf", "snv")

Discovery Analyses¶

Location: biallelic/discovery/

Discovery: Modules discovered by scanning directory for .py files

Interface: Each analysis module implements:

def main(
    aberration_list: List[pd.DataFrame],
    output_path: str,
    reference_map: Dict,
    title: str,
    logger: SimpleLogger
) -> None:
    """Run discovery analysis."""
    # Process aberration_list
    # Write results to output_path
    # Log progress to logger
    pass

Execution: Called in order from manifest analyses section

Execution Context:

All previous input files already loaded into aberration_list
Previous analyses have already executed
Results available for downstream analyses

Configuration Format¶

Manifest YAML Structure¶

# Metadata (human-readable identifiers)
title: Project Name
date: MM/DD/YYYY

# References (data loaded before inputs)
ref:
  genes:
    path: relative/path/genes.bed.gz
    format_driver: bed
  sample_donors:
    path: relative/path/samples.txt
    format_driver: maf

# Input (genomic data files)
input:
  - path: relative/path/file.maf.gz
    type: snv                          # aberration type
    format_driver: maf                 # which driver loads this
    extra_driver_args: {}              # driver-specific options

# Analyses (discovery algorithms)
analyses:
  - name: annotate_snv                 # module in biallelic/discovery/
  - name: summary_biallelic

Path Resolution¶

Relative paths: Resolved relative to manifest directory
Absolute paths: Used as-is
Path checking: Happens at load time with informative errors

Error Handling Strategy¶

The package uses specific exception types and informative messages:

Validation Errors: Caught during manifest loading: - Missing required fields - Invalid file paths - Unknown drivers/analyses
Runtime Errors: Caught during execution: - File I/O failures - Missing required references - Data format issues - Driver failures
Error Messages: All errors include: - What went wrong (specific, not generic) - Where it happened (file, location) - How to fix it (actionable guidance)

Extensibility Points¶

Users can extend the platform in these ways:

Add Custom Input Driver
- Create biallelic/drivers/my_format.py
- Implement methods for data types needed
- Reference in manifest format_driver: my_format
Add Custom Discovery Analysis
- Create biallelic/discovery/my_analysis.py
- Implement main() function
- Reference in manifest analyses: - name: my_analysis
Custom Data Processing
- Subclass SimpleLogger for custom logging
- Subclass Aberrations for custom workflows
- Import utilities (xopen, color_palettes, etc.) for your analyses

Backward Compatibility¶

All improvements maintain backward compatibility:

✓ Existing manifests work unchanged
✓ Existing driver/analysis modules work unchanged
✓ Existing output formats preserved
✓ API additions are purely additive
✓ No breaking changes to public classes

Performance Considerations¶

Memory Usage

All aberrations loaded into DataFrames
Large files may require significant RAM
Consider processing by chromosome for very large datasets

Execution Time

Input drivers dominate execution time
Discovery analyses are typically fast
Logging is minimal overhead

File I/O

Automatic gzip detection for compression
BGZF support for random access
xopen() provides transparent file handling

Design Patterns¶

Plugin Architecture: Dynamic loading of drivers and analyses allows extensibility without modifying core code.
Orchestrator Pattern: Aberrations class orchestrates workflow, coordinating multiple components (drivers, analyses, logging).
Strategy Pattern: Different drivers implement common interface for loading different file formats.
Factory Pattern: Dynamic module discovery creates driver/analysis instances at runtime based on configuration.
Builder Pattern: Manifest gradually builds analysis configuration as sections are parsed and validated.

Testing Strategy¶

Tests are organized by module:

test/test_models.py - Data model tests
test/test_logging.py - Logging functionality
test/test_misc.py - Utility function tests
test/test_drivers.py - Driver functionality
test/test_bgzf.py - File I/O tests

Integration tests verify: - Manifest parsing and validation - Reference data loading - Input data processing - Discovery analysis execution