{ "cells": [ { "cell_type": "markdown", "id": "70e78126-74b8-451d-9651-6ca635484225", "metadata": {}, "source": [ "# Preprocessing for _MethylBERT_ fine-tuning training data\n", "\n", "_MethylBERT_ fine-tuning needs DNA methylation data from tumour (T) and normal (N) samples as training data. You can give a list of sample files with annotations in a tab-deliminated file. " ] }, { "cell_type": "code", "execution_count": 1, "id": "baa86abd-334e-4c46-a19b-32e2b79b6170", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "../test/data/T_sample.bam\tT\n", "../test/data/N_sample.bam\tN\n" ] } ], "source": [ "cat ../test/data/bam_list.txt" ] }, { "cell_type": "markdown", "id": "97d1fcb0-3641-48cb-8e6b-f55e68c46e12", "metadata": {}, "source": [ "\n", "As described in the [data preparation](https://github.com/hanyangii/methylbert/blob/main/tutorials/01_Data_Preparation.md) tutorial, DMRs and the reference genome should be prepared in the required format. \n", "\n", "_MethylBERT_ provides `finetune_data_generate` function to preprocess the given tumour and normal data." ] }, { "cell_type": "code", "execution_count": 2, "id": "def4854d-9be7-4ff5-b790-ed823ca2e384", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "DMRs sorted by areaStat\n", " chr start end length nCG meanMethy1 meanMethy2 \\\n", "1 chr10 134597480 134602875.0 5396 670 0.861029 0.140400 \n", "0 chr7 1268957 1277884.0 8928 753 0.793278 0.129747 \n", "2 chr4 1395812 1402597.0 6786 663 0.831162 0.185272 \n", "5 chr16 54962053 54967980.0 5928 546 0.783631 0.096095 \n", "9 chr18 76736906 76741580.0 4675 510 0.829475 0.104403 \n", "\n", " diff.Methy areaStat abs_areaStat abs_diff.Methy ctype dmr_id \n", "1 0.720629 6144.089331 6144.089331 0.720629 T 0 \n", "0 0.663531 5722.091790 5722.091790 0.663531 T 1 \n", "2 0.645891 4941.410089 4941.410089 0.645891 T 2 \n", "5 0.687536 4714.551799 4714.551799 0.687536 T 3 \n", "9 0.725072 4684.608381 4684.608381 0.725072 T 4 \n", "Number of DMRs to extract sequence reads: 20\n", "../test/data/T_sample.bam processing (T)...\n", "../test/data/N_sample.bam processing (N)...\n", "Fine-tuning data generated: name flag ref_name ref_pos \\\n", "0 SRR10165994.69237235_69237235_length=151 163 chr7 156797584 \n", "1 SRR10165464.24148712_24148712_length=151 99 chr10 131770809 \n", "2 SRR10165994.26664131_26664131_length=151 163 chr10 131766813 \n", "3 SRR10165464.61126854_61126854_length=150 147 chr10 131769430 \n", "4 SRR10165994.14375046_14375046_length=150 83 chr5 1884762 \n", "\n", " map_quality cigar next_ref_name next_ref_pos length \\\n", "0 24 151M = 156797848 344 \n", "1 42 151M = 131770846 188 \n", "2 42 149M = 131767027 365 \n", "3 24 151M = 131769291 -290 \n", "4 40 149M = 1884665 -246 \n", "\n", " seq ... PG XG \\\n", "0 GGGGAAGAAAAAAAACTAAATAATAATTTAACATACATACGTAAAC... ... MarkDuplicates GA \n", "1 GGTTTGTCGGGAAGGTTGTGAGTAGAGGCCAACGGAGGTCTCCCAG... ... MarkDuplicates CT \n", "2 GGGGGCCTCTAAAAACGCTCCAAATTCGTCTTACGCCACGAAATCA... ... MarkDuplicates GA \n", "3 GTTGGGTGGTAAGGTGGTTTAGGGTATAGTTAGGGGTTATGTAGAA... ... MarkDuplicates CT \n", "4 AATAATTATTTCTAAATTCTATATTAATTTCGCGACAAACCGCGTT... ... MarkDuplicates GA \n", "\n", " NM XM XR \\\n", "0 23 HHH.z..hhh.h..h...............h...h.....Z..h..... GA \n", "1 11 ..hxz.xZ.......xz.z...x.....HH..Z.....hH.HHX..... CT \n", "2 38 .Z.ZX.....x.h.h.Z......h...Z....h.Z....Zxhh...... GA \n", "3 31 .x....z..h....z..hhx....h....h......hh........... GA \n", "4 18 .......h.....x......x....h.....Z.Zx..xh..Z.Z..... CT \n", "\n", " dna_seq \\\n", "0 GGG GGC GCG CGA GAT ATG TGG GGG GGA GAG AGA GA... \n", "1 GGC GCC CCC CCG CGC GCC CCG CGG GGG GGA GAA AA... \n", "2 CGC GCG CGG GGC GCC CCT CTC TCT CTG TGA GAG AG... \n", "3 GCT CTG TGG GGG GGC GCG CGG GGC GCA CAA AAG AG... \n", "4 AAT ATA TAA AAT ATT TTG TGT GTT TTT TTC TCT CT... \n", "\n", " methyl_seq dmr_ctype dmr_label ctype \n", "0 2202222222222222222222222222222222222212222212... T 17 T \n", "1 2220221222222220202222222222222122222222222222... T 5 N \n", "2 2122222222222212222222222122222212222122222221... T 5 T \n", "3 2222202222222022222222222222222222222222222222... T 5 N \n", "4 2222222222222222222222222222212122222221212222... T 8 T \n", "\n", "[5 rows x 22 columns]\n", "Size - train 3051 seqs , valid 763 seqs \n" ] } ], "source": [ "from methylbert.data import finetune_data_generate as fdg\n", "\n", "f_bam_file_list = \"../test/data/bam_list.txt\"\n", "f_dmr = \"../test/data/dmrs.csv\"\n", "f_ref = \"../../../genome/hg19.fa\"\n", "out_dir = \"tmp/\"\n", "\n", "fdg.finetune_data_generate(\n", " sc_dataset = f_bam_file_list,\n", " f_dmr = f_dmr,\n", " f_ref = f_ref,\n", " output_dir=out_dir,\n", " split_ratio = 0.8, # Split ratio to make training and validation data\n", " n_mers=3, # 3-mer DNA sequences \n", " n_cores=20\n", ")" ] }, { "cell_type": "markdown", "id": "8338c2f6-3d11-48fd-813f-7bce45f2d3e1", "metadata": {}, "source": [ "After the preprocessing, you get three different files:\n", "1. dmrs.csv : Selected DMRs (when the number of DMRs is given) with `dmr_label` column\n", "2. train_seq.csv : Preprocessed training data\n", "3. test_seq.csv : Preprocessed evaluation data (20% of given data, due to the split_ratio=0.8)" ] }, { "cell_type": "code", "execution_count": 22, "id": "fae1e53b-a4bc-4d9c-a97d-73e98ef52885", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dmrs.csv test_seq.csv train_seq.csv\n" ] } ], "source": [ "ls tmp/" ] }, { "cell_type": "markdown", "id": "e4b4b84f-2d99-483c-b5e1-2642ddfc4356", "metadata": {}, "source": [ "Each preprocessed data is a tab-deliminated .csv file where each column contains the individual field of given BAM/SAM file. Additionally `dmr_ctype`, `dmr_label` and `ctype` are given:\n", "1. `dmr_ctype`: The specific cell type for each DMR\n", "2. `dmr_label`: DMR label. This is used for the read classifier fully-connected network in _MethylBERT_\n", "3. `ctype` : Cell-type of the read (indicated in the input file)" ] }, { "cell_type": "code", "execution_count": 11, "id": "c2d62b4b-c87a-4229-8f69-c6fd0725e9f3", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
nameflagref_nameref_posmap_qualitycigarnext_ref_namenext_ref_poslengthseq...PGXGNMXMXRdna_seqmethyl_seqdmr_ctypedmr_labelctype
0SRR10165464.6790597_6790597_length=15183chr217694354140151M=176943475-217AATTAACAATTTTCATCATAATCTACACATTATTAACATCAAACTT......MarkDuplicatesGA37h...hh........z.........x..........h.............CTGAT ATT TTG TGG GGC GCA CAA AAT ATT TTT TTT TT...2222222222220222222222222222222222222222222222...T12N
1SRR10165994.18752987_18752987_length=149163chr715748661640149M=157486650183AGGCACGCGACCACCCTAAACCTCGAACAAAACTAAAAAAACGCAA......MarkDuplicatesGA51..Z...Z.Zx.......xhh....Zx...xhh...hhhhh..Z..x...GACCG CGC GCA CAC ACG CGC GCG CGG GGC GCC CCA CA...1222121222222222222222122222222222222222122222...T11T
2SRR10165994.2935274_2935274_length=15083chr7127022242150M=1269981-391ACGAACATTAAAACGCACGGAACCGCCGCGACGCGGACTCGCTCTT......MarkDuplicatesGA27h.Z.h....hhh..Z...ZX.h..Z..Z.Zx.Z.ZX....Z........CTGCG CGA GAG AGC GCA CAT ATT TTG TGG GGG GGA GA...1222222222221222122222122121221212222212222222...T1T
3SRR10165464.56090327_56090327_length=151163chr217694951142149M=176949602242AGGATTTCTTACTACATAACCACAAAAATACATTAAACCCACACCT......MarkDuplicatesGA36h.Z.......h....z.hh..z.zx.hh.h....hhh...z.z......GAGCG CGC GCT CTT TTT TTC TCT CTT TTG TGC GCT CT...1222222222222022222020222222222222222202022222...T12N
4SRR10165464.47924911_47924911_length=150147chr7127248042151M=1272378-253AATTATTGGGAGTTTGATGTTGATAAGTAAAGTGTTGGAGTGTGGG......MarkDuplicatesCT31......z.....h...................z.xz......z......GAAAT ATT TTA TAT ATC TCG CGG GGG GGA GAG AGC GC...2222202222222222222222222222222022022222202220...T1N
\n", "

5 rows × 22 columns

\n", "
" ], "text/plain": [ " name flag ref_name ref_pos \\\n", "0 SRR10165464.6790597_6790597_length=151 83 chr2 176943541 \n", "1 SRR10165994.18752987_18752987_length=149 163 chr7 157486616 \n", "2 SRR10165994.2935274_2935274_length=150 83 chr7 1270222 \n", "3 SRR10165464.56090327_56090327_length=151 163 chr2 176949511 \n", "4 SRR10165464.47924911_47924911_length=150 147 chr7 1272480 \n", "\n", " map_quality cigar next_ref_name next_ref_pos length \\\n", "0 40 151M = 176943475 -217 \n", "1 40 149M = 157486650 183 \n", "2 42 150M = 1269981 -391 \n", "3 42 149M = 176949602 242 \n", "4 42 151M = 1272378 -253 \n", "\n", " seq ... PG XG \\\n", "0 AATTAACAATTTTCATCATAATCTACACATTATTAACATCAAACTT... ... MarkDuplicates GA \n", "1 AGGCACGCGACCACCCTAAACCTCGAACAAAACTAAAAAAACGCAA... ... MarkDuplicates GA \n", "2 ACGAACATTAAAACGCACGGAACCGCCGCGACGCGGACTCGCTCTT... ... MarkDuplicates GA \n", "3 AGGATTTCTTACTACATAACCACAAAAATACATTAAACCCACACCT... ... MarkDuplicates GA \n", "4 AATTATTGGGAGTTTGATGTTGATAAGTAAAGTGTTGGAGTGTGGG... ... MarkDuplicates CT \n", "\n", " NM XM XR \\\n", "0 37 h...hh........z.........x..........h............. CT \n", "1 51 ..Z...Z.Zx.......xhh....Zx...xhh...hhhhh..Z..x... GA \n", "2 27 h.Z.h....hhh..Z...ZX.h..Z..Z.Zx.Z.ZX....Z........ CT \n", "3 36 h.Z.......h....z.hh..z.zx.hh.h....hhh...z.z...... GA \n", "4 31 ......z.....h...................z.xz......z...... GA \n", "\n", " dna_seq \\\n", "0 GAT ATT TTG TGG GGC GCA CAA AAT ATT TTT TTT TT... \n", "1 CCG CGC GCA CAC ACG CGC GCG CGG GGC GCC CCA CA... \n", "2 GCG CGA GAG AGC GCA CAT ATT TTG TGG GGG GGA GA... \n", "3 GCG CGC GCT CTT TTT TTC TCT CTT TTG TGC GCT CT... \n", "4 AAT ATT TTA TAT ATC TCG CGG GGG GGA GAG AGC GC... \n", "\n", " methyl_seq dmr_ctype dmr_label ctype \n", "0 2222222222220222222222222222222222222222222222... T 12 N \n", "1 1222121222222222222222122222222222222222122222... T 11 T \n", "2 1222222222221222122222122121221212222212222222... T 1 T \n", "3 1222222222222022222020222222222222222202022222... T 12 N \n", "4 2222202222222222222222222222222022022222202220... T 1 N \n", "\n", "[5 rows x 22 columns]" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "pd.read_csv(\"tmp/test_seq.csv\", sep='\\t').head()" ] } ], "metadata": { "kernelspec": { "display_name": "dnabert", "language": "python", "name": "dnabert" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.13" } }, "nbformat": 4, "nbformat_minor": 5 }