This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Thread and Memory Dumps

Creating a Thread Dump

  1. find pod

    $ oc get pods -l run=nice --show-all=false
    NAME           READY     STATUS    RESTARTS   AGE
    nice-3-2nl1q   2/2       Running   0          49m
  2. create thread dump

    Nice >= 3.0:

    $ oc exec -c nice nice-3-2nl1q -- stack-trace
    Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):
    
    "pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000]
      java.lang.Thread.State: TIMED_WAITING (parking)

    Nice < 3.0:

    $ oc exec -c nice nice-3-2nl1q -- jstack 1
    Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):
    
    "pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    …

Creating a Memory Dump Manually

  1. find pod

    $ oc get pods -l run=nice --show-all=false
    NAME           READY     STATUS    RESTARTS   AGE
    nice-3-2nl1q   2/2       Running   0          49m
  2. create dump

    Nice >= 3.0:

    $ oc exec -c nice nice-3-2nl1q -- memory-dump --stdout > dump.hprof

    Nice < 3.0

    $ oc exec -c nice nice-3-2nl1q -- jmap -dump:format=b,file=/app/var/heap_dumps/forced_dump.hprof 1
    Dumping heap to /app/var/heap_dumps/forced_dump.hprof ...
    Heap dump file created
  3. copy dump

    Nice < 3.0:

    $ oc cp -c nice nice-3-2nl1q:/app/var/heap_dumps/forced_dump.hprof forced_dump.hprof
    $ ls -lh forced_dump.hprof
    -rw-r--r-- 1 user user 2.3G Mar  6 13:55 forced_dump.hprof

Creating Memory Dump on OOM

Note

Memory dumps are always created when an OutOfMemoryError is thrown and this is logged:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /app/var/heap_dumps/memory-dump-hlediiivhc.hprof ...

Skip ahead to copy the dump and copy the dump from the location printed in the logs.

  1. enable automatic memory dumps

    oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE=true
    

    NICE2_MEMORY_DUMP_ON_TERMINATE

    Create a memory dump on every pod termination. This does not only include pod termination as result of a pod restart triggered by the livenessProbe as result of an OOM-condition. Consequently, some dumps may not be related to an OOM-condition at all.

    NICE2_DUMP_ON_OOM

    Tell the Java JVM to create a memory dump when an OutOfMemoryError is thrown. When memory is raising slowly, the application is often terminated due to a failing livenessProbe before such an exception is thrown.

    This in enabled by default.

    Warning

    This will restart Nice automatically!

    This isn’t very reliable. It’s very possible that the application becomes unresponsive and is, thus, terminated before a memory dump is created.

  2. wait for OOM crash

  3. find a running pod

    $ oc get pods -l run=nice --show-all=false
    NAME           READY     STATUS    RESTARTS   AGE
    nice-3-2nl1q   2/2       Running   0          49m
  4. find the dump you want

    $ oc exec -c nice nice-3-2nl1q -- ls -lh /app/var/heap_dumps/
    -rw-r--r-- 1 peter peter 1.2G Mar  6 13:57 memory-dump-asfkosfnkt.hprof
  5. copy the dump

    $ oc cp -c nice nice-3-2nl1q:/app/var/heap_dumps/memory-dump-asfkosfnkt.hprof dump.hprof
    $ ls -lh dump.hprof
    -rw-r--r-- 1 user user 1.2G Mar  6 14:00 dump.hprof
  6. disable dumps on OOM

    oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE-
    

    Warning

    This will restart Nice automatically!

Hint

Dumps are stored in /app/var/heap_dumps/ within the container, an emptyDir by default. The directory is bound to exactly one pod and will survive a restart. It will be deleted when the pod seizes to exist.

Warning

The following will restart Nice automatically!

Alternatively, a persistent volume can be added which is shared among all pods and will survive pod termination:

oc set volume dc/nice -c nice --add --name=heap-dumps --claim-name=oom --claim-size=15G --mount-path=/app/var/heap_dumps --overwrite

Remove volume again after use and switch back to emptyDir:

oc set volume dc/nice -c nice --add --name=heap-dumps --type emptyDir --mount-path /app/var/heap_dumps --overwrite