基礎生物学研究所ゲノムインフォマティクス・トレーニングコース2011秋

NGSデータ解析基本ツール

!"#$"%&"'()*+,(-.//0122,(3456547

重信　秀治 / !89:7(!87;"<=&9

山口　勝司(>(?5$@9@87(A5%5;9B87

内山　郁夫(>(149=(CB87D5%5

1

!"##$%&'()*++

•  !"#$%シークエンスデータ可視化ツール%

•  &'()**+,$%マッピングデータ処理の必携ツール%

•  -./)**+,$%アノテーションデータ処理の便利ツール%

•  汎用タブ区切りテキストの処理法を身に付ける%

2

次世代ＤＮＡシーケンシングデータの可視化法お手軽ツールIntegrative Genomics Viewer(IGV)

2011.09.09

お手軽ツ ルIntegrative Genomics Viewer(IGV)

基礎生物学研究所・生物機能解析センター 山口勝司

NIBB CORE RESEARCH FACILITIES FUNCTIONAL GENOMICS FACILITYNIBB CORE RESEARCH FACILITIES FUNCTIONAL GENOMICS FACILITY

NIBB CORE RESEARCH FACILITIES FUNCTIONAL GENOMICS FACILITY

可視化ツールに求められるもの

膨大なデータを如何に直感的に理解できるようにするか

・遺伝子発現の数値情報・位置情報・SNPの位置情報・頻度情報・SNPの位置情報・頻度情報・様々なデータの精度情報

gene model / gene annotationと並べて比較複数のデータセットを並べて比較

色々なデータを比較・統合的に解釈できるようにしたい

ゲノムviewerに自分のデータを乗せ、統合的直感的に解釈できること統合的直感的に解釈できる と

3

可視化ツールの取捨選択基準を考える

1. お金を出す気があるか無料 / 有料 / 基本無料無料 / 有料 / 基本無料

2 誰が使うか2. 誰が使うか個人レベル/ コミュニティーレベル

3. 利用深度データを見るだけ/自分から色々工夫

4.利用しやすさ導入に必要なコンピュータスペック導入に必要なコンピュ タスペックマニュアルの分かりやすさ利用の簡便さ利用者が多いか/情報の多さ

お手軽ツールIntegrative Genomics Viewer(IGV)Integrative Genomics Viewer(IGV)・アカデミックウェアで無料コミュニティ での利用者が多いから 情報も多い・コミュニティーでの利用者が多いから、情報も多い・javaのプログラムなので、オールプラットフォーム対応・マニュアルは親切、サンプルデータのある・WEBサーバーではなく PCレベルでできる・WEBサ バ ではなく、PCレベルでできる

共同利用に供するには誰もが簡便に使える必要がある。

4

IGV(Integrative genome viewer)によるデータ可視化

次世代シーケンサーのデータを見るためには、BAMファイルをロードするだけ。

Produced by Broad Institutehttp://www.broadinstitute.org/igv/

5

6

簡易説明

7

ユーザーガイド

8

9

ゲノムViewerなので次世代ＤＮＡシーケンサーのデータに限定されない。マイクロアレイの結果や、ゲノムアノテーションの情報も随時表示できる。マイクロアレイの結果や、ゲノムアノテ ションの情報も随時表示できる。

対応するファイル形式に応じて、表示方法が決まる。

WEBサイト説明より

10

.gctファイル サンプル間・遺伝子間の発現プロファイル

WEBサイト説明より

Gene List ViewLoading/Defining Gene ListsLoading/Defining Gene ListsMy Lists

WEBサイト説明より

11

Nature Biotech. 29:24–26 (2011) Supplement figureからの抜粋

Nature Biotech. 29:24–26 (2011) Supplement figureからの抜粋

12

公開情報のviewerとして

Human body map project

13

さあ、実際使ってみましょうさあ、実際使 てみましょう

14

登録されていない生物種・配列でも 自分でimportすればOK登録されていない生物種 配列でも、自分でimportすればOK

スケール変更

今回のデータはChr2:1 2 000 000のみです今回のデータはChr2:1-2,000,000のみです

15

BAMファイルを読み込む→File->Load from file で用意されているファイルの中からLer.bamを読み込む次世代シーケンサーデータの神髄（リード配列の情報とゲノム上のマッピングに関する情報）

mutファイルを読み込む (File-> Load from fileからLer.mutをロード）

Chr1 711 711 Ler testChr1 892 892 Ler testChr1 956 956 Ler testChr1 956 956 Ler testChr1 10904 10904 Ler testChr1 32210 32210 Ler testChr1 37388 37388 Ler testChr1 37388 37388 Ler testChr1 49438 49438 Ler testChr1 71326 71326 Ler testChr1 71348 71348 Ler testChr1 71348 71348 Ler testChr1 88300 88300 Ler test

16

mut

bam

トラックの移動トラックの移動

← →による移動マウスのドラッグも移動

popupで情報を見ることが可能

マウスのドラッグも移動

gene modelのTrackを指定して、Ctr+F gene model単位で右に移動g 動Ctr+B gene model単位で左に移動

17

データを全部消し、別のデータセットへ

変異体 を確認 ま う変異体のSNPを確認してみましょう。

１．VCFファイルを読み込ませてみる。File-> Load from file からSummary.vcfをロード２ BEDフ イルを読み込ませてみる２．BEDファイルを読み込ませてみる。３．WIGファイルを読み込ませてみる。４．Chr2: 388,854に移動５ BAMファイルを複数並べてみる５．BAMファイルを複数並べてみる

VCF file (Variant call file)個々のサンプルのSNP位置と頻度、確からさが記載されている。（ここでは に対し ３株のデ タが含まれている）（ここではcontに対し、mutant３株のデータが含まれている）青色がheteroSNP、水色がhomoSNPを意味する。Samtools, Vcftoolsで作成可能。

SNPを確認してみましょう。

１．VCFファイルを読み込ませてみる。２．BEDファイルを読み込ませてみる。 File-> Load from file からtranspozon.bedをロード３．WIGファイルを読み込ませてみる。

移４．Chr1:28211145に移動４．BAMファイルを複数並べてみる

18

BEDファイルの描画

ここではトランスポゾン領域が分かるようにします。

SNPを確認してみましょう。

１ VCFファイルを読み込ませてみる１．VCFファイルを読み込ませてみる。２．BEDファイルを読み込ませてみる。３．WIGファイルを読み込ませてみる。 File-> Load from file からcg_ratio.wig.tdfをロード４ Chr2:28211145に移動４．Chr2:28211145に移動５．BAMファイルを複数並べてみる

マウス右クリックでtrack表示の設定変更が可能。Type of Graph > Line plotType of Graph -> Line plotWindowing functuib -> Set datqa range -> min 0 max 100Change track height

19

WIGファイル単純なベースごとの数値データを扱う時などに用いる。

定幅 塩基 デ タを表 さ まし う一定幅での塩基CG ratioのデータを表示させてみましょうテキスト形式の大きなファイルは描画に時間がかかる。バイナリー化すべき。

WIG → hdf(IGV lで可能 今回は処理済み 説明は省略(IGV toolで可能、今回は処理済み、説明は省略

File-> Run igvtoolsで起動可能、tileを選択）

variableStep chrom=Chr152 3653 3654 3455 3455 3456 3457 3258 3259 3260 3460 3461 3662 3663 3464 3265 32

SNPを確認してみましょう。

１．VCFファイルを読み込ませてみる。２．BEDファイルを読み込ませてみる。３ WIGファイルを読み込ませてみる３．WIGファイルを読み込ませてみる。４．Chr2:388854に移動５．BAMファイルを複数並べてみて、目的領域のリード状況をみてみましょう。

File-> Load from file からFile-> Load from file からcont.bammutant1.bammutant2.bammutant2.bammutant3.bamをロード

BAMファイルやVCFファイルの描画にはindexファイルも必要。igv toolsで作成可能、今回はすでに作成してあります。

20

リード配列が離れているものにも対応

Split Screen ViewSplit Screen View

メイトペアデータの場合のみ

Paired view

21

その他の便利機能

セッションの保存セッションの保存表示しているデータの読み込み状況を、それごと保存。セッションをロードすることで、意図した画面を表示できる。

データセットが揃っていること、フォルダー構造が同一である必要がある。

バッチ処理重要領域の画面スナップショットを自動で取ったりできる。

newload myfile.bamyfsnapshotDirectory mySnapshotDirectorygenome hg18goto chr1:65,289,335-65,309,335sort positioncollapsesnapshotgoto chr1:113,144,120-113,164,120sort basecollapsesnapshot

IGV紹介のまとめ

可視化ツールとして十分な機能を持つ

・無料

比較的簡単 お手軽・比較的簡単・お手軽

・次世代DNAシーケンサー解析の標準的ツールになりつつある

・自分で見るためにも良し、人に見せるためにも良し

・利用範囲は次世代DNAシーケンサーに限定しない利用範囲は次世代DNAシ ケンサ に限定しない広くゲノミクスの解析に有用

ウェブサイトを見ながら復習して頂けたら、もっと良く分かるはず

22

!"##$%&'()*++

samtools

!"##$%&'()*++

!"#$%%&' •  !"#$%!&'(&)*&$"+,-).&)/0#123$456.1/$,7$1$-&)&6,*$456.1/$456$7/56,)-$+16-&$)(*+&589&$7&'(&)*&$1+,-).&)/7:$$

•  !"#$;55+7$265<,9&$<16,5(7$(8+,8&7$456$.1),2(+18)-$1+,-).&)/7$,)$/=&$!"#$456.1/>$,)*+(9,)-$7568)->$.&6-,)->$,)9&?,)-$1)9$-&)&618)-$1+,-).&)/7$,)$1$2&6@257,85)$456.1/:$

References!Li et al. The Sequence Alignment/Map format and SAMtools. Bioinformatics (2009) vol. 25 (16) pp. 2078-9!http://samtools.sourceforge.net/!

23

!"##$%&'()*++!"#$%%&'(が活躍する場面 •  !"#0A"#はマッピングデータの標準フォーマット。$•  .122,)-結果がA"#でしか提供されない場合、内容をテキスト表示して確認。$

•  興味のある領域だけのアライメント情報を抽出。$•  ファイルサイズを節約する為に!"#をA"#に圧縮。$•  A"#をロードするプログラムのために756/$B$,)9&?,)-$%&?C$DEF3$

•  複数のマッピングデータをマージする。$•  .122,)-$結果の簡単な統計を表示$•  !GH$*1++の機能も優秀$

SAMtoolsは、NGSのマッピングデータを扱う様々な場面で活躍する必須ツール。!

!"##$%&'()*++

!"#$%%&' •  samtools$,7$1$*5..1)9@+,)&$/55+:$

Usage: samtools <command> [options]!

•  IGDJ$K1L$•  M6,N&)$,)$O:$O5.2,+&$1)9$,)7/1++$PL$make!•  command line!•  6&19$95*(.&)/7$*16&4(++L$

•  K&P7,/&$95*(.&)/7$•  =&+2$95*(.&)/$

•  QR"S#R$

24

!"##$%&'()*++

=N2C0071./55+7:75(6*&456-&:)&/0

!"##$%&'()*++

=N2C0071./55+7:75(6*&456-&:)&/071./55+7:7=/.+

25

!"##$%&'()*++

$ samtools !!Program: samtools (Tools for alignments in the SAM format)!Version: 0.1.18 (r982:295)!!Usage: samtools <command> [options]!!Command: view SAM<->BAM conversion! sort sort alignment file! mpileup multi-way pileup! depth compute the depth! faidx index/extract FASTA! tview text alignment viewer! index index alignment! idxstats BAM index stats (r595 or later)! fixmate fix mate information! flagstat simple stats! calmd recalculate MD/NM tags and '=' bases! merge merge sorted alignments! rmdup remove PCR duplicates! reheader replace BAM header! cat concatenate BAMs! targetcut cut fosmid regions (for fosmid pool only)! phase phase heterozygotes!

!"##$%&'()*++

)'*('+,-.%//012 $ samtools view!!Usage: samtools view [options] <in.bam>|<in.sam> [region1 [...]]!!Options: -b output BAM! -h print header for the SAM output! -H print header only (no alignments)! -S input is SAM! -u uncompressed BAM output (force -b)! -x output FLAG in HEX (samtools-C specific)! -X output FLAG in string (samtools-C specific)! -c print only the count of matching records! -t FILE list of reference names and lengths (force -S) [null]! -T FILE reference sequence file (force -S) [null]! -o FILE output file name [stdout]! -R FILE list of read groups to be outputted [null]! -f INT required flag, 0 for unset [0]! -F INT filtering flag, 0 for unset [0]! -q INT minimum mapping quality [0]! -l STR only output reads in library STR [null]! -r STR only output reads in read group STR [null]! -? longer help!

26

!"##$%&'()*++31'$0&&0$4%1 •  D)7/1++185)の方法は著者配布のドキュメントに従う$

$

# download samtools-0.1.18.tar.bz2!$ tar xjvf samtools-0.1.18.tar.bz2 # 解凍!$ cd samtools-0.1.18!$ less INSTALL # インストール方法をしらべる!…!Type `make' to compile samtools. !…!$ make!# pathの通ったディレクトリにsamtools（や付属ユーティリティ）をコピーする!

!"##$%&'()*++

H61*8*&$

$

練習用データセットをダウンロードしてください。$

$http://133.48.62.157/download/git2011a/samtools_practice.tgz!

$

$

$

27

!"##$%&'()*++5%16*7$(!"#($%(8"# $ samtools view -Sb ex1.sam -o ex1.bam!!

Try!!

-  Q1. less コマンドでex1.samの中身を確認しよう。!

-  Q2. less コマンドでRun1.bamの中身を確認できるだろうか？!

-  Q3. samtools view を使ってex1.samをbamに変換しよう。!

-  Q3. ls commandでファイルサイズがsam => bamで小さくなったことを確認

しよう。!

!"##$%&'()*++

view9(:4*;(8"#(<4&*

$ samtools view NA12878.chr16p.bam!!$ samtools view NA12878.chr16p.bam | less!

NA12878.chr16p.bam は、1000 Genomes ProjectのCEU (白人女性)のマッピングデータのうち、chromosome 16 : 48,000,000 – 50,000,000 を抽出したbam fileである。中身を確認してみよう。!

Try!!

-  Q1. samtools view を使ってNA12878.chr16p.bam　の内容をテキス

ト表示しよう。。!

28

!"##$%&'()*++view9(31'=*.$(>*02*7(=07$( •  T&19&6$216/$54$A"#$U+&$,)*+(9&7$(7&4(+$,)456.185)

$ samtools view –H NA12878.chr16p.bam!!@HD VN:1.0 GO:none SO:coordinate!@SQ SN:1 LN:249250621 AS:NCBI37 UR:file:/home/shige/hg19.fasta \ M5:1b22b98cdeb4a9304cb5d48026a85128!@SQ SN:2 LN:243199373 AS:NCBI37 UR:file:/home/shige/hg19.fasta \ M5:a0d9851da00400dec1098a9255ac712e!...!@RG ID:ERR001268 PL:ILLUMINA LB:NA12878.1 PI:200 DS:SRP000032 \ SM:NA12878 CN:MPIMG!@RG ID:ERR001269 PL:ILLUMINA LB:NA12878.1 PI:200 DS:SRP000032 \ SM:NA12878 CN:MPIMG!...!@PG ID:bwa VN:0.5.5!

-  Each header line begins with character ‘@’ followed by a two-letter record type code. !

-  Each line is TAB-delimited.!-  each data field follows a format ‘TAG:VALUE’ where TAG is a two-letter string!

!"##$%&'()*++

view9(5%16*7$(8"#($%(!"# $ samtools view -h NA12878.chr16p.bam > NA12878.chr16p.sam!

Try!!

-  Q1. 上のコマンドを使ってNA12878.chr16p.bamをsamフォーマットに変

換しよう。!

-  Q2. –h オプションの役割は？!

29

!"##$%&'()*++view9(:4*;(0('=*.4<4.(7*?4%1(

$ samtools view NA12878.chr16p.bam 16:49,000,000-49,000,100 !![bam_index_load] fail to load BAM index. # <== Error message![main_samview] random alignment retrieval only works for indexed BAM files.!!## build index!$ samtools index NA12878.chr16p.bam!# => NA12878.chr16p.bam.bai!!## try again!$samtools view NA12878.chr16p.bam 16:49,000,000-49,000,100!...!

16:49,000,000 -- 49,000,100 の部分だけ抽出せよ!

-  Q1: 何行出力されただろうか？!-  Q2: 別の領域を自分で設定して試してみよう。!-  Q3: 16:49,000,001 のみを出力しよう。(そのコマンドと結果の行数を報告せよ)!-  Q4: 16:49,900,000-50,000,100のみを含むbam fileを作製しよう。!

!"##$%&'()*++

merge9(/*7?*(8"#'(

## inspect Run1.bam & Run2.bam!# まず、Run1.bam とRun2.bamがいくつのリード数から成るか確認せよ。!!?!!## merge!$ samtools merge out.bam Run1.bam Run2.bam!!## check!#合計リード数がRun1+Run2の値になっているか？確認せよ。!

Run1.bam, Run2.bam をひとつのbam file にマージせよ。!

[注意]ヘッダに注意。今回はRun1, Run2のヘッダが同一なので問題ない。場合によっては、!samtools merge –h オプションや、samtools reheader サブコマンドを利用して適切なヘッダを書き込む必要がある。!

30

!"##$%&'()*++42@'$0$'9(/0==*2(7*02'数を集計する

$ samtools idxstats NA12878.chr16p.bam!...!15 102531392 0 0!16 90354753 2175422 78412!17  81195210 0 0!...!!

Retrieve and print stats in the index file. !!例えば、染色体ごとのヒットを集計するのに便利。!!

VW$;"A$9&+,.,/&9$/&?/$XC$6&4&6&)*&$7&'(&)*&$)1.&$YC$6&4&6&)*&$7&'(&)*&$+&)-/=$ZC$[$.122&9$6&197$\C$[$().122&9$6&197:$

!"##$%&'()*++

<&0?'$0$A(2*=$>

$ samtools flagstat NA12878.chr16p.bam!2253834 + 0 in total (QC-passed reads + QC-failed reads)!131828 + 0 duplicates!2175422 + 0 mapped (96.52%:nan%)!1907026 + 0 paired in sequencing!953675 + 0 read1!953351 + 0 read2!1589213 + 0 properly paired (83.33%:nan%)!1750199 + 0 with itself and mate mapped!78415 + 0 singletons (4.11%:nan%)!47076 + 0 with mate mapped to a different chr!27432 + 0 with mate mapped to a different chr (mapQ>=5)!

flagstat: Collect some statistics about alignment!!マッピングの簡単な統計をとるのに便利!

$ samtools depth NA12878.chr16p.bam |head!16 47999937 1!16 47999938 1!…!

depth: compute the depth!!1塩基毎にcoverage (depth)を表示!

31

!"##$%&'()*++tview9(:4'+0&4B*(0&4?1/*1$

## you need reference sequence!$ samtools tview NA12878.chr16p.bam human_chr16_partial.fasta!

# type “g” for go to the region of your interest (ex, 16:49000000)!# type “?” for display help!

IGVが使えない環境でも手軽にalignmentをvisualizeできる!

!"##$%&'()*++

view9(C4&$*741?(D02601.*2E

samtools view –f BAM_FILE!

-  Q1: sample = SRR010937のみ抽出せよAns : -r SRR010937 オプションで絞り込む!

-  Q2: PEで両方のタグが存在するもののみ抽出せよAns : フラグでフィルタリングする　-f 3 !

!

難しいので今回はやらない。!

]56$19<1)*&9$(7&6

32

!"##$%&'()*++F%7G(%1(0('$7*0/ !1./55+7$,7$9&7,-)&9$/5$K56^$5)$1$7/6&1.:$D/$6&-1697$1)$,)2(/$U+&$_@`$17$/=&$7/1)9169$,)2(/$%7/9,)3$1)9$1)$5(/2(/$U+&$_@`$17$/=&$7/1)9169$5(/2(/$%7/95(/3:$!&<&61+$*5..1)97$*1)$/=(7$P&$*5.P,)&9$K,/=$I),?$2,2&7:$!1./55+7$1+K1L7$5(/2(/$K16),)-$1)9$&6656$.&771-&7$/5$/=&$7/1)9169$&6656$5(/2(/$%7/9&663:$$

a$71./55+7$<,&K$@($1+):P1.$JCX>bbb>bbb@X>Xbb>bbb$c$71./55+7$2,+&(2$@$$$!

a71./55+7$<,&K$@=$,):P1.$c$-6&2$@<$defQECgCRQQbbbbXeWd$c$71./55+7$<,&K$@P!$@$W$5(/:P1.$$$$

]56$19<1)*&9$(7&6

!"##$%&'()*++

H*/%$*(0..*''(%6*7($>*(41$*71*$ !1./55+7$,7$1P+&$/5$52&)$1$A"#$%)5/$!"#3$U+&$5)$1$6&.5/&$];H$56$T;;H$7&6<&6$,4$/=&$A"#$U+&$)1.&$7/16/7$K,/=$_h2C00`$56$_=N2C00`:$!1./55+7$*=&*^7$/=&$*(66&)/$K56^,)-$9,6&*/56L$456$/=&$,)9&?$U+&$1)9$K,++$95K)+519$/=&$,)9&?$(25)$1P7&)*&:$!1./55+7$95&7$)5/$6&/6,&<&$/=&$&)86&$1+,-).&)/$U+&$()+&77$,/$,7$17^&9$/5$95$75:$

$ samtools view http://133.48.62.157/download/git2010/NA12878.chr16p.bam ! !

$ samtools view http://s3.amazonaws.com/1000genomes/Pilots_Bam/NA06984/NA06984.454.MOSAIK.SRP000033.2009_11.chr22_1_49691432.bam !

O+5(9$*5.2(8)-$54$GE!$,7$&17L$K,/=$71./55+7:$

]56$19<1)*&9$(7&6

33

!"##$%&'()*++まとめ •  !"#/55+7は、GE!のマッピングデータを扱う様々な場面で活躍する必須ツール。$

$

34

!"##$%&'()*++

!"#$%%&'

!"##$%&'()*++

!"#$%%&' •  !"#$%%&'()'(*('%+,*-.('/)$.(0%-($1.(2%34*-)'%56(3*5)4/&*7%5(*58(*55%$*7%5()5(!"#(*58(9::(0%-3*$;((

•  !"#$%%&'(*&'%('/44%-$'($1.(2%34*-)'%5(%0('.</.52.(*&)=53.5$'()5(!>?(0%-3*$($%(@%$1(!"#(*58(9::(0.*$/-.';(

References!Quinlan, A.R. & Hall, I.M. BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics 26, 841–842 (2010).!!http://code.google.com/p/bedtools!

35

!"##$%&'()*++

[15:28 19/2/2010 Bioinformatics-btq033.tex] Page: 841 841–842

BIOINFORMATICS APPLICATIONS NOTE Vol. 26 no. 6 2010, pages 841–842doi:10.1093/bioinformatics/btq033

Genome analysis Advance Access publication January 28, 2010

BEDTools: a flexible suite of utilities for comparing genomicfeaturesAaron R. Quinlan1,2,! and Ira M. Hall1,2,!1Department of Biochemistry and Molecular Genetics, University of Virginia School of Medicine and 2Center forPublic Health Genomics, University of Virginia, Charlottesville, VA 22908, USAAssociate Editor: Martin Bishop

ABSTRACTMotivation: Testing for correlations between different sets ofgenomic features is a fundamental task in genomics research.However, searching for overlaps between features with existing web-based methods is complicated by the massive datasets that areroutinely produced with current sequencing technologies. Fast andflexible tools are therefore required to ask complex questions of thesedata in an efficient manner.Results: This article introduces a new software suite for thecomparison, manipulation and annotation of genomic featuresin Browser Extensible Data (BED) and General Feature Format(GFF) format. BEDTools also supports the comparison of sequencealignments in BAM format to both BED and GFF features. The toolsare extremely efficient and allow the user to compare large datasets(e.g. next-generation sequencing data) with both public and customgenome annotation tracks. BEDTools can be combined with oneanother as well as with standard UNIX commands, thus facilitatingroutine genomics tasks as well as pipelines that can quickly answerintricate questions of large genomic datasets.Availability and implementation: BEDTools was written in C++.Source code and a comprehensive user manual are freely availableat http://code.google.com/p/bedtoolsContact: aaronquinlan@gmail.com; imh4y@virginia.eduSupplementary information: Supplementary data are available atBioinformatics online.

Received on November 24, 2009; revised on January 11, 2010;accepted on January 21, 2010

1 INTRODUCTIONDetermining whether distinct sets of genomic features (e.g. alignedsequence reads, gene annotations, ESTs, genetic polymorphisms,mobile elements, etc.) overlap or are associated with one another isa fundamental task in genomics research. Such comparisons serveto characterize experimental results, infer causality or coincidence(or lack thereof) and assess the biological impact of genomicdiscoveries. Genomic features are commonly represented by theBrowser Extensible Data (BED) or General Feature Format (GFF)formats and are typically compared using either the UCSC GenomeBrowser’s (Kent et al., 2002) ‘Table Browser’ or using the Galaxy(Giardine et al., 2005) interface. While these tools offer a convenientand reliable method for such analyses, they are not amenable tolarge and/or ad hoc datasets owing to the inherent need to interactwith a remote or local web site installation. Moreover, complicated

!To whom correspondence should be addressed.

analyses often require iterative testing and refinement. In this sense,faster and more flexible tools allow one to conduct a greater numberand more diverse set of experiments. This necessity is made moreacute by the data volume produced by current DNA sequencingtechnologies. In an effort to address these needs, we have developedBEDTools, a fast and flexible suite of utilities for common operationson genomic features.

2 FEATURES AND METHODS

2.1 Common scenariosGenomic analyses often seek to compare features that are discoveredin an experiment to known annotations for the same species. Whengenomic features from two distinct sets share at least one base pairin common, they are defined as ‘intersecting’ or ‘overlapping’. Forexample, a typical question might be ‘Which of my novel geneticvariants overlap with exons?’ One straightforward approach toidentify overlapping features is to iterate through each feature inset A and repeatedly ask if it overlaps with any of the features inset B. While effective, this approach is unreasonably slow whenscreening for overlaps between, for example, millions of DNAsequence alignments and the RepeatMasker (Smit et al., 1996–2004)track for the human genome. This inefficiency is compounded whenasking more complicated questions involving many disparate sets ofgenomic features. BEDTools was developed to efficiently addresssuch questions without requiring an installation of the UCSC orGalaxy browsers. The BEDTools suite is designed for use in a UNIXenvironment and works seamlessly with existing UNIX utilities (e.g.grep, awk, sort, etc.), thereby allowing complex experiments to beconducted with a single UNIX pipeline.

2.2 Language and algorithmic approachBEDTools incorporates the genome-binning algorithm used by theUCSC Genome Browser (Kent et al., 2002). This clever approachuses a hierarchical indexing scheme to assign genomic featuresto discrete ‘bins’ (e.g. 16 kb segments) along the length of achromosome. This expedites searches for overlapping features,since one must only compare features between two sets thatshare the same (or nearby) bins. As illustrated in SupplementaryFigure 1, calculating feature overlaps for large datasets (e.g.millions of sequence alignments) is substantially faster than usingthe tools available on the public Galaxy web site. The softwareis written in C++ and supports alignments in BAM format(Li et al., 2009) through use of the BAMTools libraries (Barnettet al., http://sourceforge.net/projects/bamtools/).

© The Author(s) 2010. Published by Oxford University Press.This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/2.5), which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

at Okazaki Ntl Res Inst on M

ay 17, 2011bioinform

atics.oxfordjournals.orgDownloaded from

(Quinlan et al 2010)!

1.3 Fundamental concepts regarding BEDTools usage.

1.3.1 What are genome features and how are they represented?Throughout this manual, we will discuss how to use BEDTools to manipulate, compare and ask questions of genome “features”. Genome features can be functional elements (e.g., genes), genetic polymorphisms (e.g. SNPs, INDELs, or structural variants), or other annotations that have been discovered or curated by genome sequencing groups or genome browser groups. In addition, genome features can be custom annotations that an individual lab or researcher defines (e.g., my novel gene or variant).

The basic characteristics of a genome feature are the chromosome or sca!old on which the feature “resides”, the base pair on which the feature starts (i.e. the “start”), the base pair on which feature ends (i.e. the “end”), the strand on which the feature exists (i.e. “+” or “-“), and the name of the feature if one is applicable.

The two most widely used formats for representing genome features are the BED (Browser Extensible Data) and GFF (General Feature Format) formats. BEDTools was originally written to work exclusively with genome features described using the BED format, but it has been recently extended to seamlessly work with BED, GFF and VCF files.

Existing annotations for the genomes of many species can be easily downloaded in BED and GFF format from the UCSC Genome Browser’s “Table Browser” (http://genome.ucsc.edu/cgi-bin/hgTables?command=start) or from the “Bulk Downloads” page (http://hgdownload.cse.ucsc.edu/downloads.html). In addition, the Ensemble Genome Browser contains annotations in GFF/GTF format for many species (http://www.ensembl.org/info/data/ftp/index.html)

Section 4 of this manual describes BED and GFF formats in detail and illustrates how to define your own annotations.

1.3.2 Overlapping / intersecting features. Two genome features (henceforth referred to as “features”) are said to overlap or intersect if they share at least one base in common. In the figure below, Feature A intersects/overlaps Feature B, but it does not intersect/overlap Feature C.

!

"#$%&'#!(!

"#$%&'#!)!

"#$%&'#!*!

9

(BEDtools Manual)!

!"##$%&'()*++

!"#$%%&'(が活躍する場面 •  興味のある遺伝子モデルとオーバーラップしているリードを抽出したりカウントする。(•  例）エクソン部分のみ抽出(

•  「新規遺伝子」探索のために、既存の遺伝子モデルとオーバーラップしないリードを抽出する。(

•  A1BCD'.<実験で得られたピークの&%2/'に再近傍の遺伝子を探す場合。(

•  オーバーラップする遺伝子モデルをマージする場合。(

•  フォーマット変換

BEDtoolsは、NGSデータと遺伝子アノテーションデータを扱う様々な場面で活躍する便利ツール。!

36

!"##$%&'()*++

1E4FGG'*3$%%&';'%/-2.0%-=.;5.$G

1E4FGG2%8.;=%%=&.;2%3G4G@.8$%%&'G

!"##$%&'()*++

!"#$%%&' •  bedtools )'(*(2%33*58D&)5.($%%&;(

Usage: <command> [options]!

•  HIBJ(,*K(•  L-)E.5()5(AMM;(A%34)&.(*58()5'$*&&(@K(make!•  command line!•  stream / pipe !•  -.*8(8%2/3.5$'(2*-.0/&&K(

•  ,.@')$.(8%2/3.5$'(•  1.&4(8%2/3.5$(•  N">#?"(

37

!"##$%&'()*++)*'$+&&

2. InstallationBEDTools is intended to run in a “command line” environment on UNIX, LINUX and Apple OS X operating systems. Installing BEDTools involves downloading the latest source code archive followed by compiling the source code into binaries on your local system. The following commands will install BEDTools in a local directory on a *NIX or OS X machine. Note that the “<version>“ refers to the latest posted version number on http://bedtools.googlecode.com/.

Note: The BEDTools “makefiles” use the GCC compiler. One should edit the Makefiles accordingly if one wants to use a di!erent compiler.

curl http://bedtools.googlecode.com/files/BEDTools.<version>.tar.gz > BEDTools.tar.gztar -zxvf BEDTools.tar.gz cd BEDTools-<version>make cleanmake allls bin

At this point, one should copy the binaries in BEDTools/bin/ to either usr/local/bin/ or some other repository for commonly used UNIX tools in your environment. You will typically require administrator (e.g. “root” or “sudo”) privileges to copy to usr/local/bin/. If in doubt, contact you system administrator for help.

3. “Quick start” guide

3.1 Install BEDToolscurl http://bedtools.googlecode.com/files/BEDTools.<version>.tar.gz > BEDTools.tar.gztar -zxvf BEDTools.tar.gz cd BEDToolsmake cleanmake allsudo cp bin/* /usr/local/bin/

3.2 Use BEDToolsBelow are examples of typical BEDTools usage. Additional usage examples are described in section 6 of this manual. Using the “-h” option with any BEDTools will report a list of all command line options.

A. Report the base-pair overlap between the features in two BED files.$ intersectBed -a reads.bed -b genes.bed

B. Report those entries in A that overlap NO entries in B. Like "grep -v"$ intersectBed -a reads.bed -b genes.bed –v

17

!"##$%&'()*++

C-*272.(

(

練習用データセットをダウンロードしてください。(

(http://133.48.62.157/download/git2011a/bedtools_practice.tgz!

(

(

(

38

!"##$%&'()*++

,-+.&+/&0($%%&'

1.2 Summary of available tools

BEDTools support a wide range of operations for interrogating and manipulating genomic features. The table below summarizes the tools available in the suite (tools that support BAM file are indicated).

Utility DescriptionintersectBed Returns overlapping features between two BED/GFF/VCF files.

Also supports BAM format as input and output.windowBed Returns overlapping features between two BED/GFF/VCF files within a “window”.

Also supports BAM format as input and output.closestBed Returns the closest feature to each entry in a BED/GFF/VCF file.coverageBed Summarizes the depth and breadth of coverage of features in one BED/GFF file (e.g.,

aligned reads) relative to another (e.g., user-defined windows).Also supports BAM format as input and output.

genomeCoverageBed Histogram or a “per base” report of genome coverage.Also supports BAM format as input and output.

pairToBed Returns overlaps between a BEDPE file and a regular BED/GFF/VCF file. Also supports BAM format as input and output.

pairToPair Returns overlaps between two BEDPE files.bamToBed Converts BAM alignments to BED and BEDPE formats.

Also supports BAM format as input and output.bedToBam Converts BED/GFF/VCF features (both blocked and unblocked) to BAM format.bedToIgv Creates a batch script to create IGV images at each interval defined in a BED/GFF/

VCF file.bed12ToBed6 Splits BED12 features into discrete BED6 features.subtractBed Removes the portion of an interval that is overlapped by another feature.mergeBed Merges overlapping features into a single feature.fastaFromBed Creates FASTA sequences from BED/GFF intervals.maskFastaFromBed Masks a FASTA file based upon BED/GFF coordinates.shu!eBed Permutes the locations of features within a genome.slopBed Adjusts features by a requested number of base pairs.sortBed Sorts BED/GFF files in useful ways.linksBed Creates an HTML links from a BED/GFF file.complementBed Returns intervals not spanned by features in a BED/GFF file.overlap Computes the amount of overlap (positive values) or distance (negative values) between

genome features and reports the result at the end of the same line.groupBy Summarizes a dataset column based upon common column groupings. Akin to the SQL

"group by" command.unionBedGraphs Combines multiple BedGraph files into a single file, allowing coverage/other

comparisons between them.annotateBed Annotates one BED/VCF/GFF file with overlaps from many others.

8

!"##$%&'()*++

•  コマンドの種類が多く、/48*$.も頻繁。(•  ＝＞コマンドの種類、使い方、オプションをすべてを覚える必要はない。(

•  何が出来るツールなのかを、おおまかに把握する。(•  マニュアルやヘルプを活用する。((

39

!"##$%&'()*++!"#(1%23+$

chr22 1000 5000 geneA 960 + !chr22 2000 6000 geneB 900 -!chrX 0 1000 geneC 80 -!

,-.+ ,-.) ,-./ ,-.0 ,-.1 ,-.2

$*@(OIPQ('4*2.R

3*58*$%-K %47%5*&

21-%3 '$*-$ .58 5*3. '2%-. '$-*58

!"##$%&'()*++

intersectBed4('5200*(%-02&+6(10+$720'

$ intersectBed -a bt1.bed -b bt2.bed !chr1 200 201 C/G!chr1 205 206 C/G!

(ex) あるソフトウェアで予測したSNP候補のリスト(bt1.bed)から、自分の興味のある遺伝子GeneAに含まれるもの(bt2.bedに記述)だけを抽出する。!![file: bt1.bed]!chr1 100 101 A/G!chr1 200 201 C/G!chr1 205 206 C/G!chrX 300 301 C/T!![file: bt2.bed]!chr1 150 251 GeneA 1 +!

5. The BEDTools suite

This section covers the functionality and default / optional usage for each of the available BEDTools. Example “figures” are provided in some cases in an e!ort to convey the purpose of the tool. The behavior of each available parameter is discussed for each tool in abstract terms. More concrete usage examples are provided in Section 6.

5.1 intersectBedBy far, the most common question asked of two sets of genomic features is whether or not any of the features in the two sets “overlap” with one another. This is known as feature intersection. intersectBed allows one to screen for overlaps between two sets of genomic features. Moreover, it allows one to have fine control as to how the intersections are reported. intersectBed works with both BED/GFF/VCF and BAM files as input.

5.1.1 Usage and option summary

Usage: $ intersectBed [OPTIONS] [-a <BED/GFF/VCF> || -abam <BAM>] -b <BED/GFF/VCF>

Option Description-a BED/GFF/VCF file A. Each feature in A is compared to B in search of overlaps. Use “stdin” if

passing A with a UNIX pipe.-b BED/GFF/VCF file B. Use “stdin” if passing B with a UNIX pipe.-abam BAM file A. Each BAM alignment in A is compared to B in search of overlaps. Use “stdin” if passing

A with a UNIX pipe: For example:samtools view –b <BAM> | intersectBed –abam stdin –b genes.bed

-ubam Write uncompressed BAM output. The default is write compressed BAM output.-bed When using BAM input (-abam), write output as BED. The default is to write output in BAM when

using -abam. For example:intersectBed –abam reads.bam –b genes.bed –bed

-wa Write the original entry in A for each overlap.-wb Write the original entry in B for each overlap. Useful for knowing what A overlaps. Restricted by -f

and -r.-wo Write the original A and B entries plus the number of base pairs of overlap between the two features.

Only A features with overlap are reported. Restricted by -f and -r.-wao Write the original A and B entries plus the number of base pairs of overlap between the two features.

However, A features w/o overlap are also reported with a NULL B feature and overlap = 0. Restricted by -f and -r.

-u Write original A entry once if any overlaps found in B. In other words, just report the fact at least one overlap was found in B. Restricted by -f and -r.

-c For each entry in A, report the number of hits in B while restricting to -f. Reports 0 for A entries that have no overlap with B. Restricted by -f and -r.

-v Only report those entries in A that have no overlap in B. Restricted by -f and -r.

25

0-%3(?*5/*&

40

!"##$%&'()*++intersectBed4('5200*(%-02&+6(10+$720'

$ intersectBed –abam NA12878.chr16p.bam -b ABCC11.exons.gtf > result.bam!$ samtools view result.bam!!# visualize on IGV [genome: Human (b37)]!

(ex) あるヒト個体ゲノムのilluminaシークエンスリード（NA12878.chr16p.bam）から、興味のある遺伝子のエクソン部分（ABCC11.exons.mod.gtf）のリードだけを抽出する。!![file: NA12878.chr16p.bam]!# mapping済みのbam file!![file: ABCC11.exons.gtf]!# exon annotation in GTF format!16 hg19_refGene exon 48200822 48201277 0.0 - . gene_id "NM_032583"; transcript_id "NM_032583"; !...!

!"##$%&'()*++

41

!"##$%&'()*++5&%'0'$!084(

(ex) あるソフトウェアで予測したSNP候補の(ex-bedtools-1.bed)それぞれについて、どの遺伝子の中にSNPが見つかったのかを調べる。遺伝子の中に見つからない場合は、最も近い遺伝子を挙げ、さらにどれくらいはなれているかも確認する。遺伝子モデルはex-bedtools-3.bedに記述されているとする

0-%3(?*5/*&

5.6 closestBedSimilar to intersectBed, closestBed searches for overlapping features in A and B. In the event that no feature in B overlaps the current feature in A, closestBed will report the closest (that is, least genomic distance from the start or end of A) feature in B. For example, one might want to find which is the closest gene to a significant GWAS polymorphism. Note that closestBed will report an overlapping feature as the closest---that is, it does not restrict to closest non-overlapping feature.

5.6.1 Usage and option summary

Usage: $ closestBed [OPTIONS] -a <BED/GFF/VCF> -b <BED/GFF/VCF>

Option Description-s Force strandedness. That is, find the closest feature in B overlaps A on the same strand.

By default, this is disabled.-d In addition to the closest feature in B, report its distance to A as an extra column. The reported

distance for overlapping features will be 0.

-t How ties for closest feature should be handled. This occurs when two features in B have exactly the same overlap with a feature in A. By default, all such features in B are reported.Here are the other choices controlling how ties are handled:all Report all ties (default). first Report the first tie that occurred in the B file. last Report the last tie that occurred in the B file.

5.6.2 Default behaviorclosestBed first searches for features in B that overlap a feature in A. If overlaps are found, the feature in B that overlaps the highest fraction of A is reported. If no overlaps are found, closestBed looks for the feature in B that is closest (that is, least genomic distance to the start or end of A) to A. For example, in the figure below, feature B1 would be reported as the closest feature to A1.

Chromosome ================================================================

BED File A =============

BED File B ======== ======

Result ======

50

[file: bt1.bed]!chr1 100 101 A/G!chr1 200 201 C/G!chr1 205 206 C/G!chrX 300 301 C/T!!!!!!

[file: bt3.bed]!chr1 50 80 GeneA!chr1 190 250 GeneB!chr1 280 350 GeneC!

$ $ closestBed -a bt1.bed –b bt3.bed -d !chr1 100 101 A/G chr1 50 80 GeneA 20!chr1 200 201 C/G chr1 190 250 GeneB 0!chr1 205 206 C/G chr1 190 250 GeneB 0!chrX 300 301 C/T . -1 -1 . -1!

!"##$%&'()*++

5&%'0'$!084(

$ closestBed –a bt4.bed -b hg19.exons.gtf -d!

(ex) あるソフトウェアでChIP-seqのピークを検出しテーブルに出力した (bt4.bed)。それぞれのピークについて、どの遺伝子の近傍にピークが見つかったのかを調べる。遺伝子アノテーションは、hg19.exons.gtf!

42

!"##$%&'()*++

タブ区切りテキスト処理法

!"##$%&'()*++

タブ区切りテキストに慣れよ •  ゲノムインフォマティクスの世界では、タブ区切りテキストが多用される。!

•  タブ区切りテキストは汎用性が高い。!

•  カスタムスクリプトで文字列処理することによって、様々なデータ処理が可能。!

•  "#!$%&'(などで閲覧可能。!

43

!"##$%&'()*++例）アノテーションファイル •  )*+,-.'/0(12-2%2-*3!は)4516!*'675'の全遺伝子モデルのアノテーションテーブルである。!

•  874.&'9!):;9<<)*=7>6(71=-&8'-4&8&-'=4<*7(='6?12)<)*+,<=121@18'<.'/0(12-2%2-*3!

•  .'/0(12!/7.5129!

):;9<<*'675'-4&8&-'=4<0AB<0AB/7.512-)25(C/7.512,!

•  "#!$%&'(で表示してみよう!

•  &75516=!(D6'の('88で中身を確認しよう!

•  "#!$%&'(で表示してみよう!

!"##$%&'()*++

タブ区切りテキスト処理法 •  EFG!8;.'1=!8)''2!87H>1.'!I'-*-!"#!$%&'(J!

•  FKGL!&75516=!(D6'!

•  8&.D;2!IM4@NO!?'.(O!?N2)76!'2&J!•  M!I821P8P&1(!161(N8D8J!!

44

!"##$%&'()*++スクリプト言語でより複雑な処理!例"

M4@N!自習テキスト!):;9<<*77-*(<#Q0>@

!"##$%&'()*++

まとめ •  以下のツールの概要を学んだ。!

•  可視化ツール：GER!

•  #A"277(8!

•  S$T277(8!

•  21@区切りテキストの重要性とその処理法の基礎を学んだ!

•  FKGLコマンドラインツールの作法に慣れた!

•  &75;D('!16=!D6821((!

•  )'(;O!51641(!を活用する!

•  これらは次々に登場する新しいツールを利用するための基礎力

45