How to match the JVM thread id with Linux Process ID

THREAD: A process is divided into multiple  lite weight process. Each lite weight process is called  thread.

Thread dump provide a snap shot of current active threads. It provide stack trace of all java threads in jvm.

When do we take thread dumps:

There could be scenarios like

1.server hang

2.crash

3. Performance delays.

4. In case of stuck threads.

Different ways to take thread dumps:

Method 1 : unix command

 kill -3 pid

Method 2:  java command

cd /home/oracle/Middleware/Oracle_Home/user_projects/domains/pavan_domain/bin

[oracle@OEL6 bin]$ . ./setDomainEnv.sh

[oracle@OEL6 bin]$/home/oracle/jdk1.7.0_21/bin/java weblogic.Admin -url t3://localhost:9001 -username weblogic -password weblogic123 THREAD_DUMP

Method 3: Console 

Click on MS1 -> Monitoring -> Performance -> Dump Thread Stacks

Method 4: wlst

[oracle@OEL6 bin]$ . ./setDomainEnv.sh

[oracle@OEL6 pavan_domain]$ java weblogic.WLST

wls:/offline> connect('weblogic','weblogic','t3://localhost:9001')

wls:/pavan_domain/serverConfig> cd('Servers')

wls:/pavan_domain/serverConfig/Servers> ls()

dr--   AdminServer

dr--   ms1

dr--   ms2

wls:/pavan_domain/serverConfig/Servers> cd('ms1')

wls:/pavan_domain/serverConfig/Servers/ms1> threadDump()

How to match JVM Thread ID with Linux Process id

[oracle@OEL6 pavan_domain]$ ps -ef | grep -i Dweblogic.Name=ms1

oracle    8296  8244  0 02:20 pts/0    00:00:45 /home/oracle/jdk1.7.0_21/bin/java -server -Xms256m -Xmx256m -XX:MaxPermSize=256m -Dweblogic.Name=ms1 -Djava.security.policy=/home/oracle/Middleware/Oracle_Home/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/home/oracle/Middleware/Oracle_Home/wlserver/server/lib/cacerts -Xverify:none -Djava.endorsed.dirs=/home/oracle/jdk1.7.0_21/jre/lib/endorsed:/home/oracle/Middleware/Oracle_Home/wlserver/../oracle_common/modules/endorsed -da -Dwls.home=/home/oracle/Middleware/Oracle_Home/wlserver/server -Dweblogic.home=/home/oracle/Middleware/Oracle_Home/wlserver/server -Dweblogic.management.server=http://localhost:9001 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/home/oracle/Middleware/Oracle_Home/user_projects/domains/pavan_domain/gc.log weblogic.Server

[oracle@OEL6 pavan_domain]$ top -b -H -p 8296

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 8298 oracle    20   0 1284m 224m 1112 S  2.0 22.7   0:03.61 java

 8313 oracle    20   0 1284m 224m 1112 S  2.0 22.7   0:00.88 java

 8296 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8297 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:04.40 java

 8299 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.01 java

 8300 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.03 java

 8301 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8302 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:03.01 java

 8303 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:03.25 java

 8304 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8305 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:03.93 java

 8306 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8309 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.66 java

 8310 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:10.81 java

 8311 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8312 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:02.77 java

 8314 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:08.24 java

 8315 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.12 java

 8316 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.32 java

 8343 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8345 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.02 java

 8349 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:01.90 java

 8381 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.94 java

 8383 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.04 java

 8386 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8387 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.04 java

 8388 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8389 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.05 java

 8622 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

 8623 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java

Stuck Thread from thread dump:

"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fe980d03000 nid=0x20c4 in Object.wait() [0x00007fe98d86a000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x00000000e71abac8> (a weblogic.work.ExecuteThread)

        at java.lang.Object.wait(Object.java:503)

        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:238)

        - locked <0x00000000e71abac8> (a weblogic.work.ExecuteThread)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:264)

convert nid 0x20c4 to decimal number 8388 and compare against top -b -H -p 8296 out put

 8388 oracle    20   0 1284m 224m 1112 S  0.0 22.7   0:00.00 java