This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Thread and Memory Dumps¶

Creating a Thread Dump¶

find pod

$ oc get pods -l run=nice --show-all=false
NAME           READY     STATUS    RESTARTS   AGE
nice-3-2nl1q   2/2       Running   0          49m

create thread dump

Nice >= 3.0:

$ oc exec -c nice nice-3-2nl1q -- stack-trace
Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):

"pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000]
  java.lang.Thread.State: TIMED_WAITING (parking)

Nice < 3.0:

$ oc exec -c nice nice-3-2nl1q -- jstack 1
Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):

"pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000]
java.lang.Thread.State: TIMED_WAITING (parking)
…

Creating a Memory Dump Manually¶

find pod

$ oc get pods -l run=nice --show-all=false
NAME           READY     STATUS    RESTARTS   AGE
nice-3-2nl1q   2/2       Running   0          49m

create dump

Nice >= 3.0:

$ oc exec -c nice nice-3-2nl1q -- memory-dump --stdout > dump.hprof

Nice < 3.0

$ oc exec -c nice nice-3-2nl1q -- jmap -dump:format=b,file=/app/var/heap_dumps/forced_dump.hprof 1
Dumping heap to /app/var/heap_dumps/forced_dump.hprof ...
Heap dump file created

copy dump

Nice < 3.0:

$ oc cp -c nice nice-3-2nl1q:/app/var/heap_dumps/forced_dump.hprof forced_dump.hprof
$ ls -lh forced_dump.hprof
-rw-r--r-- 1 user user 2.3G Mar  6 13:55 forced_dump.hprof

Creating Memory Dump on OOM¶

Note

Memory dumps are always created when an OutOfMemoryError is thrown and this is logged:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /app/var/heap_dumps/memory-dump-hlediiivhc.hprof ...

Skip ahead to copy the dump and copy the dump from the location printed in the logs.

enable automatic memory dumps

oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE=true

NICE2_MEMORY_DUMP_ON_TERMINATE

Create a memory dump on every pod termination. This does not only include pod termination as result of a pod restart triggered by the livenessProbe as result of an OOM-condition. Consequently, some dumps may not be related to an OOM-condition at all.

NICE2_DUMP_ON_OOM

Tell the Java JVM to create a memory dump when an OutOfMemoryError is thrown. When memory is raising slowly, the application is often terminated due to a failing livenessProbe before such an exception is thrown.

This in enabled by default.

Warning

This will restart Nice automatically!

This isn’t very reliable. It’s very possible that the application becomes unresponsive and is, thus, terminated before a memory dump is created.

wait for OOM crash

find a running pod

$ oc get pods -l run=nice --show-all=false
NAME           READY     STATUS    RESTARTS   AGE
nice-3-2nl1q   2/2       Running   0          49m

find the dump you want

$ oc exec -c nice nice-3-2nl1q -- ls -lh /app/var/heap_dumps/
-rw-r--r-- 1 peter peter 1.2G Mar  6 13:57 memory-dump-asfkosfnkt.hprof

copy the dump

$ oc cp -c nice nice-3-2nl1q:/app/var/heap_dumps/memory-dump-asfkosfnkt.hprof dump.hprof
$ ls -lh dump.hprof
-rw-r--r-- 1 user user 1.2G Mar  6 14:00 dump.hprof

disable dumps on OOM
oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE-
Warning

This will restart Nice automatically!

Hint

Dumps are stored in /app/var/heap_dumps/ within the container, an emptyDir by default. The directory is bound to exactly one pod and will survive a restart. It will be deleted when the pod seizes to exist.

Warning

The following will restart Nice automatically!

Alternatively, a persistent volume can be added which is shared among all pods and will survive pod termination:

oc set volume dc/nice -c nice --add --name=heap-dumps --claim-name=oom --claim-size=15G --mount-path=/app/var/heap_dumps --overwrite

Remove volume again after use and switch back to emptyDir:

oc set volume dc/nice -c nice --add --name=heap-dumps --type emptyDir --mount-path /app/var/heap_dumps --overwrite