Annotating genomic variants
against public references
The system links a person's genomic variants (VCF, chip arrays) to public references and reports disease and trait risk, population frequencies, and related drugs and herbal medicines. The analysis algorithm was designed at the company with my team lead; I implemented and deployed the pipeline, backend and frontend. It was an international collaboration with Kyung Hee University's College of Korean Medicine and a Canadian bio research lab.
Architecture
The flow from input to the report API. Click a node to see the path through it and who did what.
- 01Input
-
- Input
- Raw sequencing data (FASTQ), the input to the variant-calling pipeline.
-
- Input
- A VCF file or a genotyping-array (chip) result file uploaded by the customer.
-
- Data
- dbSNP (GRCh37, GRCh38), 1000 Genomes, the GWAS Catalog and UniProt.
-
- 02Processing
-
- What I did
- Implemented the FASTQ → BWA → samtools → Picard → GATK → VCF pipeline.
- Others
- The pipeline algorithm was designed at the company together with my team lead.
- Stack
- BWA
- samtools
- Picard
- GATK
-
- What I did
- Read customer VCF and chip (array) input with PyVCF and shaped it into the input table used for annotation. Chip input finds its position key in dbSNP by rsID.
- Stack
- PyVCF
- Python
-
- 03Storage
-
- What I did
- Built dbSNP, 1000 Genomes, the GWAS Catalog and UniProt into MySQL. Split the files by chromosome (sed, split) and loaded them in pandas chunks through SQLAlchemy.
- Stack
- MySQL
- pandas
- SQLAlchemy
- sed · split
-
- 04Annotation
-
- What I did
- A four-stage SQL join: dbSNP and 1000 Genomes on a key of chromosome, position, REF and ALT, the GWAS Catalog on rsID and risk allele, and UniProt on variant ID and allele. A rule-based risk score from GWAS odds ratios and betas, allele frequencies compared across the 26 populations of 1000 Genomes, and drug and herbal-medicine information per disease (MedlinePlus, MeSH, ATC, PharmDB-K, OASIS).
- Note
- The LDA and 1D-CNN models linking diseases to herbal medicines stayed prototype experiments.
- Stack
- SQL
- MySQL
- Python
-
- 05Serving
-
- Others
- The screens were designed by a designer at the Canadian bio research lab.
- What I did
- Worked out the requirements with the designer and handled technical communication between the Korean and Canadian teams.
-
- What I did
- Served the reports through a FastAPI JSON API and also built the user-facing screens (frontend). Later I removed the web screens from the service and turned it into a JSON API for the frontend.
- Stack
- FastAPI
- JSON API
-
- What I did
- Set up a Harbor private registry myself on an Ubuntu production server at Korea IDC (certificate, domain, Let's Encrypt) and deployed with Docker images. The production server was kept reachable from internal IPs only.
- Stack
- Docker
- Harbor
- Ubuntu
- Let's Encrypt
-
References
The public data used for annotation.
- 1000 Genomes
- Allele frequencies in 26 populations
- dbSNP
- GRCh37 · GRCh38, IDs and positions
- GWAS Catalog
- Disease and trait associations, odds ratios
- UniProt
- Protein variants and clinical significance
Loading and annotation
Too large to load in one go, so it went in piece by piece.
The dbSNP and 1000 Genomes source VCFs were too large to load in one go, so I split the files by chromosome (sed, split) and loaded them a little at a time, in pandas chunks through SQLAlchemy.
Annotation is a four-stage SQL join that attaches the reference tables one after another: dbSNP and 1000 Genomes on a key of chromosome, position, REF and ALT, the GWAS Catalog on rsID and risk allele, and UniProt on variant ID and allele. Chip input finds its position key in dbSNP by rsID.
Scope
What was designed together and what I implemented, listed separately.
What I implemented and deployed
- Implementing the variant-calling pipeline (FASTQ → BWA → samtools → Picard → GATK → VCF)
- Handling of customer VCF and genotyping-array input (PyVCF)
- Loading public references and four-stage SQL annotation
- Disease and trait risk scores from GWAS odds ratios, population frequencies, and drug and herbal-medicine information per disease
- The FastAPI JSON API and the report frontend (user-facing screens)
- Docker deployment, a Harbor private registry, domain and HTTPS
- Technical communication between the Korean and Canadian teams, and aligning requirements with the Canadian lab's designer
Done together or by others
- With the team lead · designing the pipeline algorithm
- The Canadian lab's designer · UI design
- Kyung Hee University's College of Korean Medicine · a Canadian bio research lab · partners in the international collaboration
Deployment and limits
What ran in production, and what stayed an experiment.
Deployed with Harbor and Docker
- How
- I set up a Harbor private registry myself on an Ubuntu production server at Korea IDC (certificate, domain, Let's Encrypt), pushed the images there and deployed with Docker.
- Harbor
- Deployed to production through the Harbor registry I built, and used the same registry to set up a Kubernetes test environment with the team lead.
- Cleanup
- I removed the web screens and turned the service into a JSON API for the frontend.
Limits
- Score
- Risk is a rule-based score from GWAS odds ratios and betas, not a machine-learning prediction.
- Models
- The LDA and 1D-CNN models linking diseases to herbal medicines stayed prototype experiments.
- Access
- After deployment to the production server it stayed reachable only from internal IPs. It was never launched as a public service.
- Python
- SQL
- FastAPI
- MySQL
- pandas
- SQLAlchemy
- PyVCF
- BWA
- samtools
- Picard
- GATK
- Docker
- Harbor
- Ubuntu