# V2.1 MC HPRC Graphs These graphs have some minor improvements over v2.0: - minigraph construction and ordering determined by chromosome: reduces underalingments and improves Giraffe-DV results - assemblies patched so that centromeres align with rCRS - indexes generated with vg v1.70.0 - missing genotypes bug resolved in vcfwave oputput (.wave.vcf.gz files) - typo in sample names in minigraph output (.sv.gfa.gz) fixed For questions: HPRC Slack: glennhickey email: glenn.hickey@gmail.com ## Samples All graphs have HG00272 held out (due to likely large-scale misassembly in chrX) The benchmark graphs (-eval suffix) also have the following samples held out: HG002,HG005,NA19240 Different versions of CHM13 are used in the different graphs: hprc-v2.1-mc-chm13: CHM13v2.1 (chrY from HG002) hprc-v2.1-mc-chm13-eval: CHM13v1.1 + chrY from HG03017 hprc-v2.1-mc-grch38: CHM13v1.1 (no chrY) hprc-v2.1-mc-grch38-eval: CHM13v1.1 (no chrY) ## Which Graph to Use? If you plan on working with GRCh38 coordinates, the grch38-based graphs will be slightly better. Otherwise the CHM13 graphs will be more complete and accurate. You should only use the benchmark (-eval) graphs if you are benchmarking with the held-out samples, ex: using the GIAB HG002 truth VCFs. ## Which files do I need? (I'm using hprc-v2.1-mc-chm13 as an example, but all graphs have the same files / formats) * Just the Minigraph output (suitable for gfatools): hprc-v2.1-mc-chm13.sv.gfa.gz * Mapping with vg giraffe (using personal pangenome approach): hprc-v2.1-mc-chm13.gbz, hprc-v2.1-mc-chm13.hapl * Mapping with vg giraffe (old AF filtering): hprc-v2.1-mc-chm13.d46.gbz, hprc-v2.1-mc-chm13.d46.dist, hprc-v2.1-mc-chm13.d46.min * vg tools: hprc-v2.1-mc-chm13.gbz, hprc-v2.1-mc-chm13.chroms/chr*.vg * odgi tools: hprc-v2.1-mc-chm13.chroms/chr*.full.og * other tools: hprc-v2.1-mc-chm13.gfa.gz * hal tools: hprc-v2.1-mc-chm13.full.hal * VCF: hprc-v2.1-mc-chm13.wave.vcf.gz, hprc-v2.1-mc-chm13.vcf.gz.tbi Note that the graphs with ".full" in the names contained unaligned centromere sequences, whereas the others don't (see the Minigraph-Cactus documentation https://github.com/ComparativeGenomicsToolkit/cactus/blob/master/doc/pangenome.md for more information on all this) ## More On VCF There are three VCFs for each graph: hprc-v2.1-mc-chm13.raw.vcf.gz, hprc-v2.1-mc-chm13.vcf.gz, hprc-v2.1-mc-chm13.wave.vcf.gz. These are much like the VCFs from the v1.1 graph as described in https://github.com/human-pangenomics/hpp_pangenome_resources?tab=readme-ov-file#vcf-decomposition and https://github.com/ComparativeGenomicsToolkit/cactus/blob/master/doc/mc-pangenomes/hprc-v1.1-mc.md#vcf-postprocessing. But the .wave (decomposed) VCF provided here is a little bit different than the v1.1 version in that it has undergone additional normalization. The steps used to produce it (built into MC via the --vcfwave option) were as follows: vcfbub -l 0 -a 100000 bcftools annotate -x INFO/AT bcftools norm -m -any vcfwave -I 1000 bcftools norm -f sort -k1,1d -k2,2n -s https://github.com/Han-Cao/collapse-bubble/blob/76070b8abb009b38972f97bb21ec2dd3ac4a7037/scripts/merge_duplicates.py bcftools norm -m +any vcffixup (from vcflib) The decomposed graphs from v1.1 used only vcfbub and vcfwave. The additional steps here are used to 1) pass biallelic input to vcfwave, which has had issues with multiallelics. 2) left-shift variants after vcfwave 3) merge the left-shifted variants back into multiallelic sites efficiently Note that there is additional machinery used in Cactus to parallelize this process (especially vcfwave) on 100000-line VCF chunks. ## Pangenie VCF Update (Jul 28, 2026) I added four VCFs (in addition to those described below). They are normalized using PanGenie's pipeline. hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.pgbi.vcf.gz hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.pgin.vcf.gz hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.pgbi.vcf.gz hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.pgin.vcf.gz The ".pgbi" files are the decomposed biallelic (_filtered_ids_biallelic) which can be used as alternatives to the .wave VCFs below. The primary difference is that instead of realigned large bubbles, this approach uses the paths in the graph to split them up. The ".pgin" files are the Pangenie input VCFs (_filtered_ids) and can be used to genotype other samples using PanGenie. They were created using this snakemake: https://github.com/eblerjana/genotyping-pipelines/tree/f77d927f626e8749a98d1ce5bbcb7cdfc833b107/prepare-vcf-MC Using this config.yaml. Note that hprc_y2_sex.tsv was derived from https://github.com/human-pangenomics/hprc_intermediate_assembly/blob/main/data_tables/sample/hprc_release2_sample_metadata.csv ``` # name of the folder to write the results to results: "results-hprc-v2.1" # callsets generated by Minigraph-Cactus pipeline (already filtered with vcfbub) callsets: hprc-v2.1-mc-chm13: # Minigraph-Cactus VCF #vcf: "/private/home/ghickey/dev/work/genotyping-pipelines/prepare-vcf-MC/hprc-v2.1-mc-chm13.vcf" vcf: "/private/groups/cgl/hprc-graphs/hprc-v2.1-dec23/hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.vcf.gz" # Minigraph-Cactus graph GFA gfa: "/private/groups/cgl/hprc-graphs/hprc-v2.1-dec23/hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.gfa.gz" # TSV file specifying sex (1=male, 2=female) of each sample in input VCF. Format: . sample_info: "/private/home/ghickey/dev/work/genotyping-pipelines/prepare-vcf-MC/hprc_y2_sex.tsv" exclude: - GRCh38 # prefix used for chromosome names in FASTA file. reference_prefix: "chr" # Choose between: CHM13 | GRCh38 reference_version: "CHM13" hprc-v2.1-mc-grch38: # Minigraph-Cactus VCF #vcf: "/private/home/ghickey/dev/work/genotyping-pipelines/prepare-vcf-MC/hprc-v2.1-mc-grch38.vcf" vcf: "/private/groups/cgl/hprc-graphs/hprc-v2.1-dec23/hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.vcf.gz" # Minigraph-Cactus graph GFA gfa: "/private/groups/cgl/hprc-graphs/hprc-v2.1-dec23/hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.gfa.gz" # TSV file specifying sex (1=male, 2=female) of each sample in input VCF. Format: . sample_info: "/private/home/ghickey/dev/work/genotyping-pipelines/prepare-vcf-MC/hprc_y2_sex.tsv" # prefix used for chromosome names in FASTA file. reference_prefix: "chr" # Choose between: CHM13 | GRCh38 reference_version: "GRCh38" exclude: - CHM13 ``` ## Reference Gap Update (Feb 10, 2026) Reference genomes are never clipped. But it can be interesting to know about regions that don't align to anything and where, presumably, *other* haplotypes get clipped. I added ".refgap.bed" files for both graphs to reflect this. They were created with ``` wget https://raw.githubusercontent.com/glennhickey/pg-stuff/8f197691a6fc795ba45a0b146446b0fb6f00c16e/refgaps.sh chmod +x refgaps.sh ./refgaps.sh -p CHM13 -j 16 hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.chroms/*.vg > hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.refgaps.bed ./refgaps.sh -p CHM13 -j 16 hprc-v2.1-mc-chm13-eval/hprc-v2.1-mc-chm13-eval.chroms/*.vg > hprc-v2.1-mc-chm13-eval/hprc-v2.1-mc-chm13-eval.refgaps.bed ./refgaps.sh -p GRCh38 -j 16 hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.chroms/*.vg > hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.refgaps.bed ./refgaps.sh -p GRCh38 -j 16 hprc-v2.1-mc-grch38-eval/hprc-v2.1-mc-grch38-eval.chroms/*.vg > hprc-v2.1-mc-grch38-eval/hprc-v2.1-mc-grch38-eval.refgaps.bed ``` ## Clipped Regions Update (Jul 26, 2026) The MC pipeline removes contigs or piecies of contigs that don't align well enough into the graph. BED files for each sample for each graph showing removed intervals are provided. Even the .full graph will be missing contigs that could not be confidently assigned to a reference chromosome. These are reporded in the ".missing.tar.gz" files. The BED files of missing regions were determined with https://github.com/glennhickey/pg-stuff/blob/47ae5e855b9eab441d572dc0a26eb6d1d712e711/clip_beds.py (note newer MC version produce a similar report automatically) ``` python3 ~/dev/pg-stuff/clip_beds.py --seqfile hprc-v2.1-mc-grch37.seqfile --graph hprc-v2.1-mc-grch37/hprc-v2.1-mc-grch37.gfa.gz --graph hprc-v2.1-mc-grch37/hprc-v2.1-mc-grch37.full.gfa.gz --graph hprc-v2.1-mc-grch37/hprc-v2.1-mc-grch37.d46.gbz --outdir hprc-v2.1-mc-grch37/hprc-v2.1-mc-grch37.stats ~/dev/pg-stuff/clip_beds.py --seqfile hprc-v2.1-mc-grch38.seqfile --graph hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.gfa.gz --graph hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.full.gfa.gz --graph hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.d46.gbz --outdir hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.stats ~/dev/pg-stuff/clip_beds.py --seqfile hprc-v2.1-mc-chm13.seqfile --graph hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.gfa.gz --graph hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.full.gfa.gz --graph hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.d46.gbz --outdir hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.stats ``` ## MAF files (Jul 27, 2026) MAF files were added. They were created with cactus v3.2.1 ``` cactus-hal2maf ./js-chm13-maf ./hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.full.hal ./hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.full.maf.gz --outType norm single --index --chunkSize 100000 --batchCores 96 --batchCount 32 --noAncestors --batchParallelTaf 32 --batchSystem slurm --doubleMem true --slurmTime 100:00:00 --retryCount 5 --maxMemory 1Ti --disableProgress --logFile ./hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.full.maf.log --refGenome CHM13 cactus-hal2maf ./js-grch38-maf ./hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.full.hal ./hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.full.maf.gz --outType norm single --index --chunkSize 100000 --batchCores 96 --batchCount 32 --noAncestors --batchParallelTaf 32 --batchSystem slurm --doubleMem true --slurmTime 100:00:00 --retryCount 5 --maxMemory 1Ti --disableProgress --logFile ./hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.full.maf.log --refGenome GRCh38 ``` ## GBZ-Base .db files (Aug 11, 2026) These files were made with https://github.com/jltsiren/gbz-base/commit/9515f4759c89aed817ff6a4526ad6c42c050e447. But you should get identical results with the 0.5.0 and 0.5.1 releases. (courtesy Joini Siren) ## Fixed .sv.gfa.gz Update (Apr 28, 2026) I replaced these files ``` hprc-v2.1-mc-grch38/hprc-v2.1-mc-grch38.sv.gfz.gz hprc-v2.1-mc-grch38-eval/hprc-v2.1-mc-grch38-eval.sv.gfz.gz hprc-v2.1-mc-chm13/hprc-v2.1-mc-chm13.sv.gfz.gz hprc-v2.1-mc-chm13-eval/hprc-v2.1-mc-chm13-eval.sv.gfz.gz ``` with patched versions. There was a bug in cactus-pangenome's --mgSplit option that resulted in corrupt L-lines in the minigraph GFAs. The old ones are renamed with .bad extensions. I used `fix_merged_sv_gfa.py` to fix them. ## Reprodocuing the hprc-v2.1-mc Graphs Note: The input file list (hprc-v2.1-mc-raw.seqfile) was obtained directly from Julian. But these files should end up on genbank at some point. The only difference between this input and v2.0 is the mitos. ### remove HG00272 grep -v ^HG00272\. hprc-v2.1-mc-raw.seqfile > hprc-v2.1-mc-raw.seqfile.1 && mv hprc-v2.1-mc-raw.seqfile.1 hprc-v2.1-mc-raw.seqfile ### hg38 only chromosomes aws s3 cp --no-sign-request $(grep ^GRCh38 hprc-v2.1-mc-raw.seqfile | awk '{print $2}') hg38.fa.gz rm -f hg38.chroms.fa for i in `seq 22` M X Y; do samtools faidx hg38.fa.gz GRCh38#0#chr${i} >> hg38.chroms.fa; done bgzip -f --threads 4 hg38.chroms.fa ### chm13 with no Y (ie v1.1) rm -f chm13.fa.gz aws s3 cp --no-sign-request $(grep ^CHM13 hprc-v2.1-mc-raw.seqfile | awk '{print $2}') chm13.fa.gz rm -f chm13.noy.fa for i in `seq 22` M X; do samtools faidx chm13.fa.gz CHM13#0#chr${i} >> chm13.noy.fa; done bgzip -f --threads 4 chm13.noy.fa ### make a version of chm13 that has chrY from HG03017 (chrom names are in annotation/chrom_assignment in the bucket) cp $(grep ^HG03017.1 hprc-v2.1-mc-raw.seqfile | awk '{print $2}') HG03017.1.fa.gz zcat chm13.noy.fa.gz > chm13.hg03017y.fa samtools faidx HG03017.1.fa.gz HG03017#1#CM088923.1 | sed -e 's/HG03017#1#CM088923.1/CHM13#0#chrY/g' >> chm13.hg03017y.fa bgzip -f --threads 4 chm13.hg03017y.fa # note: we put references at the top, but by default mc will use mash dist for order # release chm13: CHM13v2 + GRCh38-chrom-only grep ^CHM13 hprc-v2.1-mc-raw.seqfile > hprc-v2.1-mc-chm13.seqfile printf "GRCh38\thg38.chroms.fa.gz\n" >> hprc-v2.1-mc-chm13.seqfile grep -v ^GRC hprc-v2.1-mc-raw.seqfile | grep -v ^CHM13 >> hprc-v2.1-mc-chm13.seqfile ### release GRCh38: GRCh38 + chm13v1.1 (no Y) grep ^GRCh38 hprc-v2.1-mc-raw.seqfile > hprc-v2.1-mc-grch38.seqfile printf "CHM13\tchm13.noy.fa.gz\n" >> hprc-v2.1-mc-grch38.seqfile grep -v ^CHM hprc-v2.1-mc-raw.seqfile | grep -v ^GRCh38 >> hprc-v2.1-mc-grch38.seqfile ### benchmark chm13: CHM13 with HG03017-Y + GRCh38-chrom-only - HG002,HG005,NA19240 printf "CHM13\tchm13.hg03017y.fa.gz\n" > hprc-v2.1-mc-chm13-eval.seqfile printf "GRCh38\thg38.chroms.fa.gz\n" >> hprc-v2.1-mc-chm13-eval.seqfile grep -v '^HG002\.' hprc-v2.1-mc-raw.seqfile | grep -v '^HG005\.' | grep -v '^NA19240' | grep -v ^GRC | grep -v ^CHM >> hprc-v2.1-mc-chm13-eval.seqfile ### benchmark GRCh38: GRCh38 + chm13v1.1 (no Y) - HG002,HG005,NA19240 grep -v '^HG002\.' hprc-v2.1-mc-grch38.seqfile | grep -v '^HG005\.' | grep -v '^NA19240' > hprc-v2.1-mc-grch38-eval.seqfile ### Cactus commands (v3.1.2, local binary install with vcfwave built using build-tools/downloadVCFWave) cactus-pangenome ./js-chm13 hprc-v2.1-mc-chm13.seqfile --outDir hprc-v2.1-mc-chm13 --outName hprc-v2.1-mc-chm13 --logFile hprc-v2.1-mc-chm13.log --reference CHM13 GRCh38 --mgSplit --lastTrain --snarlStats --filter 46 --giraffe --viz --haplo --vcf --vcfwave --chrom-vg --chrom-og --gfa clip full --gbz clip full filter --mgCores 8 --mapCores 4 --consCores 16 --indexCores 64 --maxMemory 1.3T --doubleMem true --batchSystem slurm --slurmTime 100:00:00 cactus-pangenome ./js-grch38 hprc-v2.1-mc-grch38.seqfile --outDir hprc-v2.1-mc-grch38 --outName hprc-v2.1-mc-grch38 --logFile hprc-v2.1-mc-grch38.log --reference GRCh38 CHM13 --mgSplit --lastTrain --snarlStats --filter 46 --giraffe --viz --haplo --vcf --vcfwave --chrom-vg --chrom-og --gfa clip full --gbz clip full filter --mgCores 8 --mapCores 4 --consCores 16 --indexCores 64 --maxMemory 1.2T --doubleMem true --batchSystem slurm --slurmTime 100:00:00 cactus-pangenome ./js-chm13-eval hprc-v2.1-mc-chm13-eval.seqfile --outDir hprc-v2.1-mc-chm13-eval --outName hprc-v2.1-mc-chm13-eval --logFile hprc-v2.1-mc-chm13-eval.log --reference CHM13 GRCh38 --mgSplit --lastTrain --snarlStats --filter 46 --giraffe --viz --haplo --vcf --vcfwave --chrom-vg --chrom-og --gfa clip full --gbz clip full filter --mgCores 8 --mapCores 4 --consCores 16 --indexCores 64 --maxMemory 1.2T --doubleMem true --batchSystem slurm --slurmTime 100:00:00 cactus-pangenome ./js-grch38-eval hprc-v2.1-mc-grch38-eval.seqfile --outDir hprc-v2.1-mc-grch38-eval --outName hprc-v2.1-mc-grch38-eval --logFile hprc-v2.1-mc-grch38-eval.log --reference GRCh38 CHM13 --mgSplit --lastTrain --snarlStats --filter 46 --giraffe --viz --haplo --vcf --vcfwave --chrom-vg --chrom-og --gfa clip full --gbz clip full filter --mgCores 8 --mapCores 4 --consCores 16 --indexCores 64 --maxMemory 1.2T --doubleMem true --batchSystem slurm --slurmTime 100:00:00 #### GRCh37 graph was created with v3.2.1 cactus-pangenome ./js-grch37 hprc-v2.1-mc-grch37.seqfile --outDir hprc-v2.1-mc-grch37 --outName hprc-v2.1-mc-grch37 --logFile hprc-v2.1-mc-grch37.log --reference GRCh37 GRCh38 CHM13 --mgSplit --lastTrain --snarlStats --filter 46 --giraffe --viz --haplo --vcf --vcfwave --chrom-vg --chrom-og --gfa clip full --gbz clip full filter --mgCores 8 --mapCores 4 --consCores 16 --indexCores 64 --maxMemory 1.2T --doubleMem true --batchSystem slurm --slurmTime 100:00:00 --vcfwaveCores 20