jcmd #javacasual

jcmd 便利

KUBOTA Yuji @sugarlife <[email protected]>

NTT OSS Center

2016/Nov/07

Copyright©2016 NTT corporation

Who I amKUBOTA Yuji (@sugarlife)

* Java Technical Support Engineer at NTT

* OpenJDK author, Icedtea committer, Speaker


HeapStats/heapstats

https://twitter.com/sugarlife

https://twitter.com/JavaOneConf/status/778995251822284801

https://github.com/HeapStats/heapstats

•Introduction to Troubleshooting in JDK 9

• jcmd と jhsdb を使おう

•スライド：https://goo.gl/4jaujJ

JDK 9 からの診断ツール


https://goo.gl/4jaujJ

Java 8


http://docs.oracle.com/javase/8/docs/technotes/tools/unix/s11-troubleshooting_tools.html

•Introduction to Troubleshooting in JDK 9

•JDK 9 では jcmd / jhsdb を使おうという話

•スライド：https://goo.gl/4jaujJ

JavaOne 2016% jstack # JDK 8:Options:

-F to force a thread dump. Use when jstack <pid> does not respond (process is hung)-m to print both java and native frames (mixed mode)-l long listing. Prints additional information about locks-h or -help to print this help message

% ./jstack # JDK 9:Options:

-l long listing. Prints additional information about locks-h or -help to print this help message

# -F was removed by JDK-8155091: Remove SA related functions from tmtools

% ./jstack -mError: -m option usedCannot connect to core dump or remote debug server. Use jhsdb jstack instead% ./jhsdb jstack --mixedYou have to set --pid or --exe.

--locks to print java.util.concurrent locks--mixed to print both java and native frames (mixed mode)--exe executable image name--core path to coredump--pid pid of process to attach


https://goo.gl/4jaujJ

https://bugs.openjdk.java.net/browse/JDK-8155091

• jcmd process help•実行可能なコマンド一覧 (対象プロセス依存)

• jcmd process help command•コマンドのヘルプ(使い方、負荷、オプション等)

Help


% jcmd 999 help GC.run999:GC.runCall java.lang.System.gc().

Impact: Medium: Depends on Java heap size and content.

Syntax: GC.run

Command: Domain.suffix (Java 8)

DomainNum.

suffixAbstract

PerfCounter 1 各種 JVM 統計情報 (※)

ManagementAgent 3 JMX エージェント操作

Thread 1 スレッドダンプ

GC 7 GC 機構 (Heap / GC / Finalization)

VM 7 ランタイム操作、情報取得


※: 本来はコマンドとは別枠だが、見易さのため並べて記載している

Command: Domain.suffix (Java 9)

DomainNum.

suffixAbstract

PerfCounter 1 - 各種 JVM 統計情報 (※)

ManagementAgent 4 +1 JMX エージェント操作

Thread 1 - スレッドダンプ

GC 7 +2

-1

GC 機構 (Heap / GC / Finalization)

VM 15 +8 ランタイム操作、情報取得

Compiler 7 - コンパイラ操作、情報取得

JVMTI 2 - JVMTI エージェント操作

GC.rotate_log が VM ドメインへ移動


※: 本来はコマンドとは別枠だが、見易さのため並べて記載している

•JVM 統計情報

• jstat や jinfo はこれを基に加工して出力してる• jdk/src/jdk.jcmd/share/classes/sun/tools/jstat/resources/jstat_options

•無加工は辛いので加工スクリプトに渡して出力すると便利

PerfCounter


suffix Description

print JVM 統計情報を出力

•Management Agent 起動したり終わらせたり• JMX (Management Extensions)と JDP(Discovery Protocol)

• JMXリモートアクセスを忘れてても後から設定可能

ManagementAgent

suffix Description

status 状態表示

start リモートエージェントを有効化

start_local ローカルエージェントを有効化

stop リモートエージェントを無効化


JDK 9 から

$ jcmd 999 ManagementAgent.start jmxremote.port=777 jmxremote.ssl=true999:Commandexecuted successfully$ jcmd 999 ManagementAgent.statusAgent: enabled

Connection Type: remoteProtocol : rmiHost : fedora24.novalocalURL : service:jmx:rmi://XXX…XXXProperties :com.sun.management.jmxremote.authenticate = true [default]com.sun.management.jmxremote.ssl.need.client.auth = false [default]com.sun.management.jmxremote.ssl = true [default]:com.sun.management.jmxremote.port = 8070


[config.file | jmxremote.* | jdp.*]

•Thread

•mixed モードが欲しい場合は% jhsdb jstack --mixed --pid <pid>

Thread

suffix Description

print Thread dump


•GC機構 (Heap/GC/Finalization)

GC

suffix Description

class_stats* クラスメタデータ一覧 (-helpで各項目説明)

class_histogram クラスヒストグラム (jmap -histo:live)

heap_info ヒープ・メタスペース使用量

heap_dump ヒープダンプ (jmap -dump, -allで参照切れも)

finalizer_info Finalize 待ちキュー表示

run_finalization Finalization 実行 (System#runFinalization)

run GC 実行 (System#gc)Copyright©2016 NTT corporation

*:要 -XX:+UnlockDiagnosticVMOptions参考：hotspot/src/share/vm/runtime/globals.hpp

•ランタイム周り。情報出力系

VM

suffix Description

info Fatal Error Log (クラッシュ時等で出すログ)

uptime 起動時間

command_line VM 引数

system_properties システムプロパティ

version バージョン

flags 設定した(された)フラグ (-all で全フラグ)

dynlibs メモリマップ付き Dynamic library リストCopyright©2016 NTT corporation

https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/felog.html

•ランタイム周り。情報出力・動作変更系

VM

suffix Description

print_touched_methods* JVM 実行中に触ったメソッドを表示

classloader_stats メタデータ状況等の全クラスローダ情報

class_hierarchy 読込クラスをクラスローダ付き階層表示

native_memory** Native Memory Tracking (NMT)

stringtable StringTable (-verboseで全Stringダンプ)

symboltable SymbolTable (-verboseで全シンボル)

set_flag flgname [value] JVM フラグの設定 (一部のみ)Copyright©2016 NTT corporation

*:要 -XX:+UnlockDiagnosticVMOptions

**: 要 -XX:NativeMemoryTracking 設定

•Native メモリの状態を記録する1. Run Java with -XX:NativeMemoryTracking=summary

2. jcmd 999 VM.native_memory baseline

3. jcmd 999 VM.native_memory summary.diff

•詳細に見たい場合は上の summary を details に変える

•NativeMemoryTracking was available since Java 8• https://docs.oracle.com/javase/8/docs/technotes/

guides/troubleshoot/tooldescr007.html

Native Memory Tracking


https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

$ jcmd 999 VM.native_memory baseline999:Baseline succeeded# --- Run method which wants to check the memory consumption ---$ jcmd 999 VM.native_memory summary.diff999:Native Memory Tracking:Total: reserved=5664359KB -36096KB, committed=371555KB -36096KB- Java Heap (reserved=4108288KB, committed=258048KB)

(mmap: reserved=4108288KB, committed=258048KB)- Class (reserved=1056953KB +15KB, committed=5433KB +15KB)

(classes #767 +3)(malloc=185KB +3KB #741 -4)(mmap: reserved=1056768KB, committed=5248KB)

- Thread (reserved=36128KB -112KB, committed=36128KB -112KB)(thread #36 +1)(stack: reserved=35980KB +325KB, committed=35980KB+325KB)(malloc=107KB +2KB #197 +4)(arena=41KB -130KB #70 +2)

:

Native Memory Tracking

±n shows diff from baseline


•ランタイム周り。ログ。

• -XX:+PrintGCDetails -Xloggc:gc.log.`date +%Y-

%m-%d_%H-%M-%S` -XX:+PrintGCTimeStamps

• -Xlog:gc*=debug:gc.log.%t:time,level

• -Xlog:[<selection>]:[<output>]:[<decorators>]

VM

suffix Description

log JVM Unified Logging (JEP158) 制御


http://openjdk.java.net/jeps/158

• jcmd 999 VM.log [options]• -Xlog:[<selection>]:[<output>]:[<decorators>]

GC ログを設定するケース


Options Description

output output (出力先)

output_options output (その他の設定)

what selection (ロギング対象とそのレベル)

decorators decorators (出力内容調整)

disable Clears all configurations and turns off all loggings

list Lists current log configurations

rotate Rotate all logs (instead of GC.log_rotate in Java 8)

•Confirms current loggings



$ jcmd 999 VM.log list999:Available log levels: off, trace, debug, info, warning, errorAvailable log decorators: time (t), uptime (u), timemillis (tm), …Available log tags: add, age, alloc, arguments, annotation, barrier, …:Described tag combinations:logging: Logging for the log framework itself

Log output configuration:#0: stdout all=warning uptime,level,tags#1: stderr all=off uptime,level,tags

default configurations

•Turns on a new logging: GC.log



$ jcmd 999 VM.log output="file=gc.log" output_options="filecount=5,filesize=10m"what="gc=debug" decorators="time,level"

999:Command executed successfully$ jcmd 999 VM.log list999:: Log output configuration:#0: stdout all=warning uptime,level,tags#1: stderr all=off uptime,level,tags#2: gc.log gc=debug time,level added a new GC logging

gc=log_level

name or path

options (log

rotation)

•Turns off all existing loggings

• If you want to turn off #2 only• jcmd 999 VM.log output="#2" what="all=off"



$ jcmd 999 VM.log disable999:Command executed successfully$ jcmd 999 VM.log list999::

Log output configuration:#0: stdout all=off uptime,level,tags#1: stderr all=off uptime,level,tags

set all=off at stdout / stderr, and removed

the other configuration(s) such as #2

output nothing

• jcmd 999 VM.log rotate

•Rotates logs satisfying the following1. Configured ‘output’ as file

2. Configured ‘filecount’ (by ‘output_option’)

手動ログローテート

$ jcmd 999 VM.log list:Log output configuration:#0: stdout all=warning uptime,level,tags#1: stderr all=off uptime,level,tags#2: gc.log gc=debug filecount=5,filesize=20M level,tags

#2 will be rotated,

not #0 and #1

Obsoleted GC.rotate_log


•JIT Compiler• JEP 165: Compiler Control (JDK 9 Feature)

Compiler


suffix Description

directive_add Adds directives of JIT by JSON format same as -XX:CompilerDirectivesFile

directive_remove Removes a latest directive

directive_clear Clears all directives

directive_print Prints current directives

codecache Prints summary of code cache

codelist Lists all code cache

queue Lists methods in waiting queue for JIT

http://openjdk.java.net/jeps/165

% cat disable_c2_at_string-hashCode.json[

{ "match":[ "*String.hashCode" ],"c2":{ "Exclude":true }

}] % jcmd 999 Compiler.directives_add disable_c2_at_string-hashCode.json999:1 compiler directives added% jcmd 999 Compiler.directives_print999:

Directive:matching: *String.hashCodec1 directives:inline: -Enable:false Exclude:false BreakAtExecute:false Log:false PrintAssembly:false …

c2 directives:inline: -Enable:true Exclude:true BreakAtExecute:false Log:false PrintAssembly:false …

Switch false to true


• jcmd 999 Compiler.codelist•Prints JIT compiled methods (in code cache)

How JIT Compiler works?

% jcmd 999 Compiler.codelist999::18 3 java.lang.String.hashCode()I [0x00007ff301594090, 0x00007ff301594280 - 0x00007ff3015946e8]:

ID

Compile level of -XX:+TieredCompilation

0: Interpreter

1-3: C1

1: C1 without profiling (full optimization)

2: C1 with basic profiling (invocation and back-edge)

3: C1 with full profiling

4: C2

Method name nmethod start address code start address

code end address


• jcmd 999 Compiler.codelist•Prints JIT compiled methods (in code cache)


% jcmd 999 Compiler.codelist999::18 3 java.lang.String.hashCode()I [0x00007ff301594090, 0x00007ff301594280 - 0x00007ff3015946e8]:

ID

Compile level of -XX:+TieredCompilation

0: Interpreter

1-3: C1

1: C1 without profiling (full optimization)

2: C1 with basic profiling (invocation and back-edge)

3: C1 with full profiling

4: C2

Method name nmethod start address code start address

code end address


http://1.bp.blogspot.com/-5Gb6-8H0hcw/VkxNZ7DLMGI/AAAAAAAA0vc/FeeCA2fbvbI/s800/computer_mojibake.png


•JVM JIT compilation overview•http://www.slideshare.net/ZeroTurnaround/vladimir-ivanovjvmjitcompilationoverview-24613146

• Intrinsic Methods in HotSpot VM•http://www.slideshare.net/RednaxelaFX/green-teajug-hotspotintrinsics02232013


http://www.slideshare.net/ZeroTurnaround/vladimir-ivanovjvmjitcompilationoverview-24613146

http://www.slideshare.net/RednaxelaFX/green-teajug-hotspotintrinsics02232013

http://1.bp.blogspot.com/-5Gb6-8H0hcw/VkxNZ7DLMGI/AAAAAAAA0vc/FeeCA2fbvbI/s800/computer_mojibake.png

•JVMTI Agent 制御

JVMTI

suffix Description

agent_load library_path [option] Attach JVMTI agent of library path

data_dump Fire DataDumpRequest JVMTI

event


% jcmd 999 JVMTI.agent_load /path/to/libheapstats.so% jcmd 999 JVMTI.agent_load ¥

/path/to/libheapstats.so=heapstats.conf


•DiagnosticCommandMBean (jconsole / jvisualvm)

リモートプロセスに jcmd

same as jcmd

commands

click then run jcmd

VM.system_properties

result

HeapStatsLightweight JVMTI agent



•Gathers runtime information continuously, then draws up time-series graphs

•Lightweight agent to run on the production systems

•New solution to troubleshooting for OpenJDK users.

HeapStats


http://www.irasutoya.com/2015/12/blog-post_168.html


http://3.bp.blogspot.com/-qzx9BMg9gxQ/UxbLz4cvIzI/AAAAAAAAeE4/QPfCtKS3UuI/s800/happy_woman6.png




Overhead of HeapStats

467.59 461.43

0

50

100

150

200

250

300

350

400

450

500

Without HeapStats With HeapStats

(ops/m)

1.32 %

Overhead Rate

SPECjvm2008 Composite Result


$ jcmd 999 JVMTI.agent_load libheapstats.so999:Commandexecuted successfully

or

java.lang.OutOfMemoryError: Java heap space

$ java –jar heapstats-analyzer.jar

$ rpm -ivh heapstats-2.0.1-0.*.rpm$ java -agentpath:heapstats … Your_Application

then

How to use HeapStats




biegebyte[]


byte[]

Most doubtful class which has many references to byte[]


Monitor wait

RunSleep

Legend


•https://github.com/HeapStats/heapstats

•http://icedtea.classpath.org/wiki/HeapStats

Your contributions are welcome!



http://icedtea.classpath.org/wiki/HeapStats

おしまい@sugarlife

[email protected]


jcmd #javacasual

Technology