スローダウン、ハングを一発解決...

Yusuke Yamamoto @yusuke #wlstudy

スローダウン、ハングを一発解決

スレッドダンプはトラブルシューティングの味方山本裕介 @yusuke


スローダウン、ハング

#wlstudyYusuke Yamamoto @yusuke

スローダウン• アプリケーションは動いている• レスポンスがなかなか帰ってこない


ハング• アプリケーションが固まる• 応答がない• 503が帰ってくる


スローダウン、ハングの原因• マシンスペック不足• キャパシティプランニングに間違い‣ キャパシティプランニングをしていない‣ 台数不足

• 割り当てCPU/メモリ不足

• 仮想化時のリソース配分


スローダウン、ハングの原因• チューニング不足• JVMオプション

• カーネルオプション• アプリケーションサーバ設定


スローダウン、ハングの原因• アプリケーション• 非効率なロジック• リソースの奪い合い• I/O待ち


切り分け• GCログの取得

• GC時の挙動を確認

• スレッドダンプの取得、解析


スレッドダンプとは• JVMのスレッド状態のスナップショット

• わかること• ある時点で• それぞれのスレッドが• 何をしているか


スレッドダンプでわからないこと

• クラッシュ / ストールの原因

• プロセスが落ちたらスレッドダンプは取れない


スレッドダンプの利点• 追加ツール不要• JProfiler

• JProbe

• YourKit Java Profiler

• 取得時に負荷がかからない• 運用環境でも手軽に使える• オフラインで解析できる


スレッドダンプの見た目


2013-01-23 16:32:13Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode):

"Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"[STANDBY] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011c40fc88> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2f568> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd1bf000 nid=0x9103 in Object.wait() [0x0000000144278000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2e4a0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba80b000 nid=0x9003 in Object.wait() [0x0000000144175000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2d1b0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba479800 nid=0x8f03 in Object.wait() [0x0000000144072000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab27da0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959f000 nid=0x8e03 in Object.wait() [0x0000000143f6f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab26c90> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a4000 nid=0x8d03 in Object.wait() [0x0000000143e6c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29978> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b800 nid=0x8c03 in Object.wait() [0x0000000143d69000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29a28> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b000 nid=0x8b03 in Object.wait() [0x0000000143c66000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab25c70> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd54800 nid=0x8a03 in Object.wait() [0x00000001435d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab28da8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f800 nid=0x8903 in Object.wait() [0x00000001434d0000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2a9a0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f000 nid=0x8803 in Object.wait() [0x0000000143b63000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab229f0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb4e000 nid=0x8703 in Object.wait() [0x0000000143a60000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab23a10> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc91800 nid=0x8603 in Object.wait() [0x000000014395d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab24a30> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb015800 nid=0x8503 in Object.wait() [0x000000014385a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1f7b0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba709800 nid=0x8403 in Object.wait() [0x0000000143757000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab207d0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb54800 nid=0x8303 in Object.wait() [0x000000013f1cd000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab21860> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc945800 nid=0x8203 in Object.wait() [0x000000014322a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1d8c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959d800 nid=0x8103 in Object.wait() [0x0000000143127000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab12600> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa7000 nid=0x8003 in Object.wait() [0x0000000143024000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab13620> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb832800 nid=0x7f03 in Object.wait() [0x0000000142f21000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab164e0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5d800 nid=0x7e03 in Object.wait() [0x0000000142e1e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)

at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)
















"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Thread.run(Thread.java:722)

"DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Thread.run(Thread.java:722)

"DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Thread.run(Thread.java:722)

"DynamicListenThread[Default[4]]" daemon prio=5 tid=0x00007f8dbc8c7800 nid=0x6903 runnable [0x0000000142244000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Thread.run(Thread.java:722)

"DynamicListenThread[Default[8]]" daemon prio=5 tid=0x00007f8dba13f800 nid=0x6803 runnable [0x0000000142141000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Thread.run(Thread.java:722)

"DynamicListenThread[Default[2]]" daemon prio=5 tid=0x00007f8dbbd06000 nid=0x6703 runnable [0x000000014203e000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Thread.run(Thread.java:722)

"weblogic.GCMonitor" daemon prio=5 tid=0x00007f8dbe0c0800 nid=0x4507 in Object.wait() [0x0000000140eed000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x000000011a6d83f0> (a java.lang.ref.ReferenceQueue$Lock) at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88) at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)

"Thread-11" prio=5 tid=0x00007f8dbc884000 nid=0x4007 waiting on condition [0x0000000140dea000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000011a50cfa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676) at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751) at java.lang.Thread.run(Thread.java:722)

"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011a4b00e8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbad9f800 nid=0x4207 in Object.wait() [0x00000001404e7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011a4b0d30> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"Thread-10" prio=5 tid=0x00007f8dbad89000 nid=0x6603 waiting on condition [0x00000001402db000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000011a208ea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676) at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751) at java.lang.Thread.run(Thread.java:722)

"DoSManager" daemon prio=5 tid=0x00007f8dbd84c800 nid=0x6503 waiting on condition [0x00000001400ce000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.octetstring.vde.DoSManager.run(DoSManager.java:433)

"VDE Transaction Processor Thread" daemon prio=5 tid=0x00007f8dbc82b000 nid=0x6303 in Object.wait() [0x000000013ffcb000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367) - locked <0x0000000119edae88> (a com.octetstring.vde.backend.standard.TransactionProcessor) at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8dbac3a000 nid=0x6103 runnable [0x000000013fdc5000] java.lang.Thread.State: RUNNABLE at weblogic.socket.PosixSocketMuxer.poll(Native Method) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102) - locked <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 waiting for monitor entry [0x000000013fcc2000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8dbaa1b800 nid=0x5f03 waiting for monitor entry [0x000000013fbbf000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba4cc800 nid=0x5e03 in Object.wait() [0x000000013fabc000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119fa2d20> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"weblogic.timers.TimerThread" daemon prio=5 tid=0x00007f8db9738000 nid=0x5d03 in Object.wait() [0x000000013f9b9000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262) - locked <0x0000000119e7f158> (a weblogic.timers.internal.TimerThread)

"weblogic.time.TimeEventGenerator" daemon prio=5 tid=0x00007f8db9724000 nid=0x5c03 in Object.wait() [0x000000013f8b6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286) - locked <0x0000000119dfc2e0> (a weblogic.time.common.internal.TimeTable) at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117) at java.lang.Thread.run(Thread.java:722)

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb232000 nid=0x5b03 in Object.wait() [0x000000013f7b3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119fa37e0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

- locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)





































at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)









































"[ACTIVE] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd1bf000 nid=0x9103 in Object.wait() [0x0000000144278000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2e4a0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba80b000 nid=0x9003 in Object.wait() [0x0000000144175000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2d1b0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba479800 nid=0x8f03 in Object.wait() [0x0000000144072000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab27da0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959f000 nid=0x8e03 in Object.wait() [0x0000000143f6f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab26c90> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a4000 nid=0x8d03 in Object.wait() [0x0000000143e6c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29978> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b800 nid=0x8c03 in Object.wait() [0x0000000143d69000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29a28> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b000 nid=0x8b03 in Object.wait() [0x0000000143c66000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab25c70> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd54800 nid=0x8a03 in Object.wait() [0x00000001435d3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab28da8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f800 nid=0x8903 in Object.wait() [0x00000001434d0000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2a9a0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f000 nid=0x8803 in Object.wait() [0x0000000143b63000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab229f0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb4e000 nid=0x8703 in Object.wait() [0x0000000143a60000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab23a10> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc91800 nid=0x8603 in Object.wait() [0x000000014395d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab24a30> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb015800 nid=0x8503 in Object.wait() [0x000000014385a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1f7b0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba709800 nid=0x8403 in Object.wait() [0x0000000143757000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab207d0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb54800 nid=0x8303 in Object.wait() [0x000000013f1cd000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab21860> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc945800 nid=0x8203 in Object.wait() [0x000000014322a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1d8c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959d800 nid=0x8103 in Object.wait() [0x0000000143127000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab12600> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa7000 nid=0x8003 in Object.wait() [0x0000000143024000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab13620> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb832800 nid=0x7f03 in Object.wait() [0x0000000142f21000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab164e0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5d800 nid=0x7e03 in Object.wait() [0x0000000142e1e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)
















"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)

at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)





































- locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)





































at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)









































スレッドダンプ取得方法• jps / jstack

• kill -3　$PID (Unix / Linux)

• ctrl-Break (Windows)

• VisualVM


スレッドダンプ取得方法• jps

JVMプロセスID,mainクラス名一覧


スレッドダンプ取得方法• jps

“Server”がWebLogicのプロセスmainクラスはweblogic.Serverなので

JVMプロセスID,mainクラス名一覧


スレッドダンプ取得方法• jstack [pid]


スレッドダンプ取得のコツ• 時間を空けて複数回取得• スレッドの動きを見るため• 取得した複数のスレッドダンプを比較


スレッドダンプ取得方法• jstack [pid] >> dump.txt

同じファイルに追記するのが現実的


WebLogic独自のスレッドダンプ取得方法

• WLST

connect('weblogic','weblogic' ,'t3://localhost:7001')cd ('Servers')cd ('AdminServer')threadDump()


WebLogic独自のスレッドダンプ取得方法

• 管理コンソール


スレッドダンプ解析は・・・• 結構難しい• トラブルが起きる前に練習を


スレッドヘッダとスタックトレース

• スタックトレースの見た目

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)



スレッドヘッダ• スレッドの先頭行

スレッドヘッダ


スレッド名• スレッドに付けられた区別用の名前• weblogic.socket.Muxer[n] : ソケットマクサー

• weblogic.kernel.Default[n] : 実行スレッド

‣ 通常weblogic.kernel.Default[n]を解析"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

スレッド名


スレッドの状態• スレッドがしていることの概要• runnable

• waiting for monitor entry

• in Object.wait()


スレッドの状態


スレッドの状態• runnable

• Javaコードを実行中

• ネイティブコードを実行中• 基本的に正常• データース待ちも runnable

"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490)


スレッドの状態• 待ち状態• in Object.wait()

• 他のスレッドから叩かれるのを待っている



スレッドの状態• 待ち状態• waiting for monitor entry

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 waiting for monitor entry [0x000000013fcc2000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 runnable [0x000000013fcc2000] java.lang.Thread.State: RUNNABLE at weblogic.socket.PosixSocketMuxer.poll(Native Method) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102) - locked <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)


スレッドダンプ解析の肝• スレッドの状態を解析• waiting for monitor entryのスレッドが多い場合は注目

• ロックしているスレッドで握っているリソースは？

• 外部リソース• 大きなファイルの読み込み


スレッドダンプ解析の肝• waiting for monitor entryがほとんどない

• runnableなスレッドは何をしている？

• 何度とっても同じ、似たスタックトレースのスレッドは注意


スレッドダンプ解析は面倒• ツールで解析• 侍

• Java Thread Dump Analyser

http://mchr3k.github.com/javathreaddumpanalyser/

http://samuraism.jp/samurai/ja/index.html






侍

http://samuraism.jp/samurai/




侍の起動• Java Web Startで起動

クリック


侍の起動• アプリケーションをダウンロード(Mac)

http://samuraism.jp/samurai/ja/samurai.zip

ダブルクリック




スレッドダンプを読み込ませる

ファイルをドラッグ&ドロップするだけ


スレッドダンプをビジュアルに表現



列: スレッドダンプ１回分



行: 同じスレッドの状態遷移


スレッドの状態で色分け表示

色: スレッドの状態


１つのスレッドを時系列で確認


状態に変化のないスレッド検出


• テーブルビューの場合状態に変化のないスレッド検出


• 時系列ビューの場合状態に変化のないスレッド検出


スレッドダンプパターン


アイドル状態• ほとんどのスレッドが灰色


がんばっている状態• ほとんどのスレッドが緑色


詰まっている状態• ほとんどのスレッドが赤、一部オレンジ


デッドロック


WebLogicのデッドロック自動検出

• BEA-000394


WebLogicのデッドロック自動検出

スローダウン、ハングを一発解決...

Technology