jcmd #javacasual

jcmd 便利

KUBOTA Yuji @sugarlife <kubota.yuji@gmail.com>

NTT OSS Center

2016/Nov/07

Who I amKUBOTA Yuji (@sugarlife)

* Java Technical Support Engineer at NTT

* OpenJDK author, Icedtea committer, Speaker

HeapStats/heapstats

•Introduction to Troubleshooting in JDK 9

• jcmd と jhsdb を使おう

•スライド：https://goo.gl/4jaujJ

JDK 9 からの診断ツール

Java 8

|-- Alignment.java:

jdk9/jdk/src/jdk.attach/linux/classes/sun/tools/attach/VirtualMachineImpl.javajdk9/hotspot/src/share/vm/services/diagnosticCommand.cpp

Dynamic Attach (on Linux)

Unix domain socket

1. Send SIGQUIT

2. Start

AttachListener thread

4. Processing

3. Send request

5. Receive response

Serviceability

Tool(jcmdを含む)

Target

process

•Introduction to Troubleshooting in JDK 9

•JDK 9 では jcmd / jhsdb を使おうという話

•スライド：https://goo.gl/4jaujJ

JavaOne 2016% jstack # JDK 8:Options:

-F to force a thread dump. Use when jstack <pid> does not respond (process is hung)-m to print both java and native frames (mixed mode)-l long listing. Prints additional information about locks-h or -help to print this help message

% ./jstack # JDK 9:Options:

-l long listing. Prints additional information about locks-h or -help to print this help message

# -F was removed by JDK-8155091: Remove SA related functions from tmtools

% ./jstack -mError: -m option usedCannot connect to core dump or remote debug server. Use jhsdb jstack instead% ./jhsdb jstack --mixedYou have to set --pid or --exe.

--locks to print java.util.concurrent locks--mixed to print both java and native frames (mixed mode)--exe executable image name--core path to coredump--pid pid of process to attach

•CLI の Local JVM process 診断ツール

• jcmd [-l | -h | process command]• -h : help, -l | no argument: jps

•process: <pid | main-class>• pid に 0 を指定すると全部• main-class は一致したプロセス全部

• command: <command[ options] | -f file>• command は Domain.suffix 形式

• jcmd process help•実行可能なコマンド一覧 (対象プロセス依存)

• jcmd process help command•コマンドのヘルプ(使い方、負荷、オプション等)

% jcmd 999 help GC.run999:GC.runCall java.lang.System.gc().

Impact: Medium: Depends on Java heap size and content.

Syntax: GC.run

Command: Domain.suffix (Java 8)

DomainNum.

suffixAbstract

PerfCounter 1 各種 JVM 統計情報 (※)

ManagementAgent 3 JMX エージェント操作

Thread 1 スレッドダンプ

GC 7 GC 機構 (Heap / GC / Finalization)

VM 7 ランタイム操作、情報取得

※: 本来はコマンドとは別枠だが、見易さのため並べて記載している

Command: Domain.suffix (Java 9)

DomainNum.

suffixAbstract

PerfCounter 1 - 各種 JVM 統計情報 (※)

ManagementAgent 4 +1 JMX エージェント操作

Thread 1 - スレッドダンプ

GC 7 +2

GC 機構 (Heap / GC / Finalization)

VM 15 +8 ランタイム操作、情報取得

Compiler 7 - コンパイラ操作、情報取得

JVMTI 2 - JVMTI エージェント操作

GC.rotate_log が VM ドメインへ移動

※: 本来はコマンドとは別枠だが、見易さのため並べて記載している

•JVM 統計情報

• jstat や jinfo はこれを基に加工して出力してる• jdk/src/jdk.jcmd/share/classes/sun/tools/jstat/resources/jstat_options

•無加工は辛いので加工スクリプトに渡して出力すると便利

PerfCounter

suffix Description

print JVM 統計情報を出力

•Management Agent 起動したり終わらせたり• JMX (Management Extensions)と JDP(Discovery Protocol)

• JMXリモートアクセスを忘れてても後から設定可能

ManagementAgent

suffix Description

status 状態表示

start リモートエージェントを有効化

start_local ローカルエージェントを有効化

stop リモートエージェントを無効化

JDK 9 から

$ jcmd 999 ManagementAgent.start jmxremote.port=777 jmxremote.ssl=true999:Commandexecuted successfully$ jcmd 999 ManagementAgent.statusAgent: enabled

Connection Type: remoteProtocol : rmiHost : fedora24.novalocalURL : service:jmx:rmi://XXX…XXXProperties :com.sun.management.jmxremote.authenticate = true [default]com.sun.management.jmxremote.ssl.need.client.auth = false [default]com.sun.management.jmxremote.ssl = true [default]:com.sun.management.jmxremote.port = 8070

[config.file | jmxremote.* | jdp.*]

•Thread

•mixed モードが欲しい場合は% jhsdb jstack --mixed --pid <pid>

Thread

suffix Description

print Thread dump

•GC機構 (Heap/GC/Finalization)

suffix Description

class_stats* クラスメタデータ一覧 (-helpで各項目説明)

class_histogram クラスヒストグラム (jmap -histo:live)

heap_info ヒープ・メタスペース使用量

heap_dump ヒープダンプ (jmap -dump, -allで参照切れも)

finalizer_info Finalize 待ちキュー表示

run_finalization Finalization 実行 (System#runFinalization)

*:要 -XX:+UnlockDiagnosticVMOptions参考：hotspot/src/share/vm/runtime/globals.hpp

•ランタイム周り。情報出力系

suffix Description

info Fatal Error Log (クラッシュ時等で出すログ)

uptime 起動時間

command_line VM 引数

system_properties システムプロパティ

version バージョン

flags 設定した(された)フラグ (-all で全フラグ)

•ランタイム周り。情報出力・動作変更系

suffix Description

print_touched_methods* JVM 実行中に触ったメソッドを表示

classloader_stats メタデータ状況等の全クラスローダ情報

class_hierarchy 読込クラスをクラスローダ付き階層表示

native_memory** Native Memory Tracking (NMT)

stringtable StringTable (-verboseで全Stringダンプ)

symboltable SymbolTable (-verboseで全シンボル)

*:要 -XX:+UnlockDiagnosticVMOptions

**: 要 -XX:NativeMemoryTracking 設定

•Native メモリの状態を記録する1. Run Java with -XX:NativeMemoryTracking=summary

2. jcmd 999 VM.native_memory baseline

3. jcmd 999 VM.native_memory summary.diff

•詳細に見たい場合は上の summary を details に変える

•NativeMemoryTracking was available since Java 8• https://docs.oracle.com/javase/8/docs/technotes/

guides/troubleshoot/tooldescr007.html

Native Memory Tracking

$ jcmd 999 VM.native_memory baseline999:Baseline succeeded# --- Run method which wants to check the memory consumption ---$ jcmd 999 VM.native_memory summary.diff999:Native Memory Tracking:Total: reserved=5664359KB -36096KB, committed=371555KB -36096KB- Java Heap (reserved=4108288KB, committed=258048KB)

(mmap: reserved=4108288KB, committed=258048KB)- Class (reserved=1056953KB +15KB, committed=5433KB +15KB)

(classes #767 +3)(malloc=185KB +3KB #741 -4)(mmap: reserved=1056768KB, committed=5248KB)

- Thread (reserved=36128KB -112KB, committed=36128KB -112KB)(thread #36 +1)(stack: reserved=35980KB +325KB, committed=35980KB+325KB)(malloc=107KB +2KB #197 +4)(arena=41KB -130KB #70 +2)

Native Memory Tracking

±n shows diff from baseline

•ランタイム周り。ログ。

• -XX:+PrintGCDetails -Xloggc:gc.log.`date +%Y-

%m-%d_%H-%M-%S` -XX:+PrintGCTimeStamps

• -Xlog:gc*=debug:gc.log.%t:time,level

• -Xlog:[<selection>]:[<output>]:[<decorators>]

suffix Description

log JVM Unified Logging (JEP158) 制御

• jcmd 999 VM.log [options]• -Xlog:[<selection>]:[<output>]:[<decorators>]

GC ログを設定するケース

Options Description

output output (出力先)

output_options output (その他の設定)

what selection (ロギング対象とそのレベル)

decorators decorators (出力内容調整)

disable Clears all configurations and turns off all loggings

list Lists current log configurations

rotate Rotate all logs (instead of GC.log_rotate in Java 8)

•Confirms current loggings

$ jcmd 999 VM.log list999:Available log levels: off, trace, debug, info, warning, errorAvailable log decorators: time (t), uptime (u), timemillis (tm), …Available log tags: add, age, alloc, arguments, annotation, barrier, …:Described tag combinations:logging: Logging for the log framework itself

Log output configuration:#0: stdout all=warning uptime,level,tags#1: stderr all=off uptime,level,tags

default configurations

•Turns on a new logging: GC.log

$ jcmd 999 VM.log output="file=gc.log" output_options="filecount=5,filesize=10m"what="gc=debug" decorators="time,level"

999:Command executed successfully$ jcmd 999 VM.log list999:: Log output configuration:#0: stdout all=warning uptime,level,tags#1: stderr all=off uptime,level,tags#2: gc.log gc=debug time,level added a new GC logging

gc=log_level

name or path

options (log

rotation)

•Turns off all existing loggings

• If you want to turn off #2 only• jcmd 999 VM.log output="#2" what="all=off"

$ jcmd 999 VM.log disable999:Command executed successfully$ jcmd 999 VM.log list999::

Log output configuration:#0: stdout all=off uptime,level,tags#1: stderr all=off uptime,level,tags

set all=off at stdout / stderr, and removed

the other configuration(s) such as #2

output nothing

• jcmd 999 VM.log rotate

•Rotates logs satisfying the following1. Configured ‘output’ as file

2. Configured ‘filecount’ (by ‘output_option’)

手動ログローテート

$ jcmd 999 VM.log list:Log output configuration:#0: stdout all=warning uptime,level,tags#1: stderr all=off uptime,level,tags#2: gc.log gc=debug filecount=5,filesize=20M level,tags

#2 will be rotated,

not #0 and #1

Obsoleted GC.rotate_log

•JIT Compiler• JEP 165: Compiler Control (JDK 9 Feature)

Compiler

suffix Description

directive_add Adds directives of JIT by JSON format same as -XX:CompilerDirectivesFile

directive_remove Removes a latest directive

directive_clear Clears all directives

directive_print Prints current directives

codecache Prints summary of code cache

codelist Lists all code cache

queue Lists methods in waiting queue for JIT

% cat disable_c2_at_string-hashCode.json[

{ "match":[ "*String.hashCode" ],"c2":{ "Exclude":true }

}] % jcmd 999 Compiler.directives_add disable_c2_at_string-hashCode.json999:1 compiler directives added% jcmd 999 Compiler.directives_print999:

Directive:matching: *String.hashCodec1 directives:inline: -Enable:false Exclude:false BreakAtExecute:false Log:false PrintAssembly:false …

c2 directives:inline: -Enable:true Exclude:true BreakAtExecute:false Log:false PrintAssembly:false …

Switch false to true

• jcmd 999 Compiler.codelist•Prints JIT compiled methods (in code cache)

How JIT Compiler works?

% jcmd 999 Compiler.codelist999::18 3 java.lang.String.hashCode()I [0x00007ff301594090, 0x00007ff301594280 - 0x00007ff3015946e8]:

Compile level of -XX:+TieredCompilation

0: Interpreter

1-3: C1

1: C1 without profiling (full optimization)

2: C1 with basic profiling (invocation and back-edge)

3: C1 with full profiling

Method name nmethod start address code start address

code end address

• jcmd 999 Compiler.codelist•Prints JIT compiled methods (in code cache)

% jcmd 999 Compiler.codelist999::18 3 java.lang.String.hashCode()I [0x00007ff301594090, 0x00007ff301594280 - 0x00007ff3015946e8]:

Compile level of -XX:+TieredCompilation

0: Interpreter

1-3: C1

1: C1 without profiling (full optimization)

2: C1 with basic profiling (invocation and back-edge)

3: C1 with full profiling

Method name nmethod start address code start address

code end address

•JVM JIT compilation overview•http://www.slideshare.net/ZeroTurnaround/vladimir-ivanovjvmjitcompilationoverview-24613146

• Intrinsic Methods in HotSpot VM•http://www.slideshare.net/RednaxelaFX/green-teajug-hotspotintrinsics02232013

•JVMTI Agent 制御

suffix Description

agent_load library_path [option] Attach JVMTI agent of library path

data_dump Fire DataDumpRequest JVMTI

% jcmd 999 JVMTI.agent_load /path/to/libheapstats.so% jcmd 999 JVMTI.agent_load ¥

/path/to/libheapstats.so=heapstats.conf

•DiagnosticCommandMBean (jconsole / jvisualvm)

リモートプロセスに jcmd

same as jcmd

commands

click then run jcmd

VM.system_properties

result

HeapStatsLightweight JVMTI agent

•Gathers runtime information continuously, then draws up time-series graphs

•Lightweight agent to run on the production systems

•New solution to troubleshooting for OpenJDK users.

HeapStats

Overhead of HeapStats

467.59 461.43

Without HeapStats With HeapStats

(ops/m)

1.32 %

Overhead Rate

SPECjvm2008 Composite Result

$ jcmd 999 JVMTI.agent_load libheapstats.so999:Commandexecuted successfully

java.lang.OutOfMemoryError: Java heap space

$ java –jar heapstats-analyzer.jar

$ rpm -ivh heapstats-2.0.1-0.*.rpm$ java -agentpath:heapstats … Your_Application

How to use HeapStats

biegebyte[]

byte[]

Most doubtful class which has many references to byte[]

Monitor wait

RunSleep

Legend

•https://github.com/HeapStats/heapstats

•http://icedtea.classpath.org/wiki/HeapStats

Your contributions are welcome!

おしまい@sugarlife

kubota.yuji@gmail.com

jcmd #javacasual

Technology

#sxsw 2014 recap mslgroup pbjs mike standish

teoria del color (nx power lite)

kurs i sosiale medier for offentlig sektor

content analysis: methods and mentoring

german aerospace day 2013

uma introdução ao open geospatial foundation

nouveau catalogue général de lika electronic en langue...

grass gis e sextante

30@30: newsletter gennaio

et bibliotek i skyene (bibliotek 2.0, sosiale medier)

volare information kit italian

linq inside

cat smag e

the dream wall project

sosiale medier og personvern

priv 2.0: rapport om sosiale medier og personvern

談 openstreetmap @ hp42

lika products catalogue for wind generator industry -...

an overview of open source & foss4g

30@30: newsletter aprile