Aladynoulli Documentation

📋 How to Use This Documentation

This documentation is organized into five main sections for reviewers:

1. Model Architecture - Understand how the model works: core components, mathematical framework, and key concepts
2. Reviewer Response Analyses - Interactive analyses addressing all reviewer questions, organized by referee
3. Main Figure Reproduction - Per-figure notebooks (Figures 1–5) with rendered HTML versions; cross-references to Zenodo data archive
4. Non-Centered Parameterization - Reparameterization for identifiable genetic effects, with simulation evidence
5. Reparam Results - AUC, calibration, ROC curves, and Delphi state-of-the-art comparison (same v1 model, improved γ estimation)
6. Complete Workflow - Step-by-step guide to running the model: preprocessing, training, and prediction
7. Performance & Scalability - Computational requirements and scaling characteristics

Note: Installation instructions are not required for reviewers. A pre-configured environment will be provided for running the code.

📖 Table of Contents

• How to Use This Documentation
• Overview
• Model Architecture
• Reviewer Response Analyses
• Main Figure Reproduction
• Non-Centered Parameterization
• Reparam Results
• Complete Workflow
• Performance & Scalability
• Citation

🔬 Overview

Aladynoulli is a comprehensive Bayesian survival model that predicts disease trajectories by integrating genetic and clinical data. The model captures:

• Disease Signatures: Latent disease states that capture shared patterns across diseases
• Genetic Effects: Individual-specific genetic contributions to disease risk
  
• Temporal Dynamics: Time-varying disease probabilities using Gaussian processes
• Censoring: Proper handling of incomplete follow-up data

Key Features

🏗️ Model Architecture

Core Components

1. Signature States (K): Latent disease signatures that capture shared patterns
2. Genetic Effects (γ): Individual-specific genetic contributions
3. Temporal Dynamics (λ): Time-varying signature proportions using GPs
4. Disease Probabilities (φ): Signature-specific disease probabilities
5. Censoring Matrix (E): Event times and censoring information

Mathematical Framework

The model predicts disease probability at time t as:

π_i,d,t = κ × Σ_k θ_i,k,t × φ_k,d,t

Where: - θ_i,k,t = softmax(λ_i,k,t) (signature proportions) - λ_i,k,t ~ GP(μ_k + G_i γ_k, K_λ) (temporal dynamics) - φ_k,d,t = sigmoid(μ_d+ψ_k,d, K_φ) (disease probabilities)

📊 Reviewer Response Analyses

Comprehensive interactive analyses addressing reviewer questions and model validation.

📚 Navigation Hub

• Reviewer Response README - Complete guide to all interactive analyses

🔬 Analysis Categories

Referee #1 Analyses

Referee #2 Analyses

Referee #3 Analyses

🎨 Main Figure Reproduction

Each main-text figure in the published manuscript has a dedicated Jupyter notebook. The rendered HTML versions are linked below; for full details on inputs, the Zenodo data archive, and the suggested reproduction workflow, see figure_reproduction.html.

🧬 Non-Centered Parameterization (Reparameterization)

In the standard (“centered”) formulation, genetic effects (γ) enter only through the GP prior mean for λ. Because λ is a free parameter that directly enters the likelihood, the optimizer fits the data by adjusting λ and γ receives weaker gradient signal (only the W = 10⁻⁴ scaled GP prior). This can limit the accuracy of γ recovery — the individual-specific genetic effects that are essential for biological interpretation and out-of-sample prediction.

The non-centered formulation addresses this by decomposing λ into a genetic mean and a residual:

λ_i,k,t = μ_k + G_i γ_k + δ_i,k,t

where δ (not λ) carries the GP prior. Now γ flows through the forward pass into the NLL via the chain rule, receiving full gradient signal. Additionally, κ is fixed at 1 rather than learned, since κ and γ are not jointly identifiable (only κ·γ enters the likelihood).

Core Model Files (Non-Centered)

📊 Reparam Results

The reparameterization above is a methodological refinement of the same v1 generative model — same disease topology, same survival likelihood, same parameter counts. Only γ estimation changes. The results below are from the reparam (nokappa-v3 LOO) pipeline applied to the published v1 cohort and metrics.

Headline: Across 28 major diseases, reparam improves AUC on 24–25 of 28 at every horizon (1-year baseline, 1-year median rolling, 10-year static/dynamic, 10-year rolling). Mean Δ AUC ranges from +0.023 (10-yr static) to +0.093 (1-yr baseline). The improvement reflects better γ identifiability, not a model change.

State-of-the-art comparison: Against the Delphi-2M transformer on the same disease–ICD-10 mappings, Aladynoulli reparam wins 27/28 phecode groups on rolling 1-year prediction, mean advantage +0.10 AUC.

Side-by-Side Comparison Tables

Figures (reparam Figure 5)

Delphi-2M State-of-the-Art Comparison

Predictions used in all of the above: the reparam LOO pi tensor at Dropbox/enrollment_predictions_nokappa_v3_loo_all40/pi_enroll_fixedphi_sex_FULL.pt (400K × 348 × 52). Per-batch checkpoints in Dropbox/censor_e_batchrun_vectorized_REPARAM_v3_nokappa/.

PRS-Signature γ Magnitude (centered vs reparam)

The reparam parameterization recovers larger γ magnitudes than the centered fit (centered γ is pulled toward zero by the joint MAP gradient; reparam isolates γ identifiability). The notebooks below quantify this gain and visualize it across three pools (original v1 centered, 400K LOO reparam, and centered-vs-reparam scatter).

💻 Complete Workflow

The Aladynoulli workflow consists of 5 main steps:

1. Preprocessing: Create smoothed prevalence, initial clusters, and reference trajectories
2. Batch Training: Train models on data batches with full E matrix
3. Master Checkpoint: Generate pooled checkpoint (phi and psi)
4. Pool Gamma & Kappa: Pool gamma (genetic effects) and kappa (calibration) from training batches
5. Prediction: Run predictions using master checkpoint with fixed gamma and kappa (only lambda is learned)

Essential Resources

Core Model Files

Note: The prediction model uses fixed gamma (genetic effects) and kappa (calibration parameter) from pooled training batches. This ensures complete separation between training and testing data in each validation fold. Only lambda (individual-specific signature loadings) is learned during prediction.

Workflow Scripts

📈 Performance & Scalability

Computational Requirements

For 10K individuals, 348 diseases, 52 timepoints: - Training Time: ~8-10 minutes per batch (converges after ~200 epochs) - Prediction Time: ~8 minutes per batch - Memory: ~8GB RAM (peak usage during training) - CPU: Multi-core recommended (4+ cores); PyTorch uses BLAS for parallel matrix operations

Scaling: - Full UK Biobank (400K individuals): Processed in 39 batches of ~10K each - Total training time: ~5-7 hours for all batches (can be parallelized) - Memory scales linearly: ~8GB per 10K batch

Why it’s fast: - Vectorized PyTorch operations (batched matrix decompositions) - BLAS Level 3 operations for efficient linear algebra - ~100-fold speedup compared to loop-based implementation

Data Privacy & Access

This repository contains no patient-identifying information. No individual-level identifiers (EIDs, MRNs, or other participant IDs) from UK Biobank, All of Us, or Mass General Brigham are included in any files or git history. All analyses use de-identified, aggregate-level results only.

Access to the underlying individual-level data from UK Biobank, All of Us, and Mass General Brigham requires separate approval from each institution’s data access committee and is available only with PI permission and institutional authorization.

📝 Citation

If you use Aladynoulli in your research, please cite:

@article{aladynoulli2024,
  title={Aladynoulli: A Bayesian Survival Model for Disease Trajectory Prediction},
  author={Sur, P. and others},
  journal={medRxiv},
  year={2024},
  doi={10.1101/2024.09.29.24314557}
}

📧 Contact

For questions or issues, please open an issue on GitHub.