This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.
Thread and Memory Dumps¶
Creating a Thread Dump¶
find pod
$ oc get pods -l run=nice --show-all=false NAME READY STATUS RESTARTS AGE nice-3-2nl1q 2/2 Running 0 49m
create thread dump
Nice >= 3.0:
$ oc exec -c nice nice-3-2nl1q -- stack-trace Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode): "pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000] java.lang.Thread.State: TIMED_WAITING (parking)
Nice < 3.0:
$ oc exec -c nice nice-3-2nl1q -- jstack 1 Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode): "pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000] java.lang.Thread.State: TIMED_WAITING (parking) …
Creating a Memory Dump Manually¶
find pod
$ oc get pods -l run=nice --show-all=false NAME READY STATUS RESTARTS AGE nice-3-2nl1q 2/2 Running 0 49m
create dump
Nice >= 3.0:
$ oc exec -c nice nice-3-2nl1q -- memory-dump --stdout > dump.hprof
Nice < 3.0
$ oc exec -c nice nice-3-2nl1q -- jmap -dump:format=b,file=/app/var/heap_dumps/forced_dump.hprof 1 Dumping heap to /app/var/heap_dumps/forced_dump.hprof ... Heap dump file created
copy dump
Nice < 3.0:
$ oc cp -c nice nice-3-2nl1q:/app/var/heap_dumps/forced_dump.hprof forced_dump.hprof $ ls -lh forced_dump.hprof -rw-r--r-- 1 user user 2.3G Mar 6 13:55 forced_dump.hprof
Creating Memory Dump on OOM¶
Note
Memory dumps are always created when an OutOfMemoryError is thrown and this is logged:
java.lang.OutOfMemoryError: Java heap space Dumping heap to /app/var/heap_dumps/memory-dump-hlediiivhc.hprof ...
Skip ahead to copy the dump and copy the dump from the location printed in the logs.
enable automatic memory dumps
oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE=true
NICE2_MEMORY_DUMP_ON_TERMINATE
Create a memory dump on every pod termination. This does not only include pod termination as result of a pod restart triggered by the livenessProbe as result of an OOM-condition. Consequently, some dumps may not be related to an OOM-condition at all.
NICE2_DUMP_ON_OOM
Tell the Java JVM to create a memory dump when an OutOfMemoryError is thrown. When memory is raising slowly, the application is often terminated due to a failing livenessProbe before such an exception is thrown.
This in enabled by default.
Warning
This will restart Nice automatically!
This isn’t very reliable. It’s very possible that the application becomes unresponsive and is, thus, terminated before a memory dump is created.
wait for OOM crash
find a running pod
$ oc get pods -l run=nice --show-all=false NAME READY STATUS RESTARTS AGE nice-3-2nl1q 2/2 Running 0 49m
find the dump you want
$ oc exec -c nice nice-3-2nl1q -- ls -lh /app/var/heap_dumps/ -rw-r--r-- 1 peter peter 1.2G Mar 6 13:57 memory-dump-asfkosfnkt.hprof
copy the dump
$ oc cp -c nice nice-3-2nl1q:/app/var/heap_dumps/memory-dump-asfkosfnkt.hprof dump.hprof $ ls -lh dump.hprof -rw-r--r-- 1 user user 1.2G Mar 6 14:00 dump.hprof
disable dumps on OOM
oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE-
Warning
This will restart Nice automatically!
Hint
Dumps are stored in /app/var/heap_dumps/
within the container, an
emptyDir by default. The directory is bound to exactly one pod
and will survive a restart. It will be deleted when the pod seizes to exist.
Warning
The following will restart Nice automatically!
Alternatively, a persistent volume can be added which is shared among all pods and will survive pod termination:
oc set volume dc/nice -c nice --add --name=heap-dumps --claim-name=oom --claim-size=15G --mount-path=/app/var/heap_dumps --overwrite
Remove volume again after use and switch back to emptyDir:
oc set volume dc/nice -c nice --add --name=heap-dumps --type emptyDir --mount-path /app/var/heap_dumps --overwrite