This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Thread and Memory Dumps

Creating a Thread Dump

  1. find pod

    $ oc get pods -l run=nice --show-all=false
    NAME           READY     STATUS    RESTARTS   AGE
    nice-3-2nl1q   2/2       Running   0          49m
  2. create thread dump

    $ oc exec -c nice nice-3-2nl1q -- stack-trace
    Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):
    
    "pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000]
      java.lang.Thread.State: TIMED_WAITING (parking)

Creating a Memory Dump Manually

  1. find pod

    $ oc get pods -l run=nice --show-all=false
    NAME           READY     STATUS    RESTARTS   AGE
    nice-3-2nl1q   2/2       Running   0          49m
  2. create dump

    $ oc exec -c nice nice-3-2nl1q -- memory-dump
    1:
    Dumping heap to /app/var/heap_dumps/manual-dump-2023-08-30T09:18:33.hprof ...
    Heap dump file created [664273346 bytes in 2.475 secs]
    Uploading '/app/var/heap_dumps/manual-dump-2023-08-30T09:18:33.hprof.zst' to 's3://memory-dumps/nice-demo/manual-dump-2023-08-30T09:18:33.hprof.zst'.
  3. copy dump to local machine

    s3cmd get s3://memory-dumps/nice-demo/manual-dump-2023-08-30T09:18:33.hprof.zst

    Tip

    s3cmd has to be set up in accordance with S3 Access via s3cmd (Optional).

  4. Decompress:

    zstd --rm -d manual-dump-2023-08-30T09:18:33.hprof.zst

Creating Memory Dump on OOM

Note

Memory dumps are always created when an OutOfMemoryError is thrown (see NICE2_DUMP_ON_OOM) and this is logged:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /app/var/heap_dumps/memory-dump-hlediiivhc.hprof ...

An easy way to see if any such dumps where created is to list all dumps.

  1. enable automatic memory dumps

    oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE=true
    

    Warning

    This will restart Nice automatically!

    NICE2_MEMORY_DUMP_ON_TERMINATE

    Create a memory dump on every pod termination. This does not only include pod termination as result of a pod restart triggered by the livenessProbe as result of an OOM-condition. Consequently, some dumps may not be related to an OOM-condition at all.

    NICE2_DUMP_ON_OOM

    This is enabled by default.

    Tell the Java JVM to create a memory dump when an OutOfMemoryError is thrown. When memory is raising slowly, the application is often terminated due to a failing livenessProbe before such an exception is thrown.

  2. wait for OOM crash

  3. list dumps

    All dumps:

    $ s3cmd ls -r s3://memory-dumps
    2023-08-30 07:18    127709619  s3://memory-dumps/nice-demo/manual-dump-2023-08-30T09:18:33.hprof.zst
    2023-08-30 07:57    101418514  s3://memory-dumps/nice-test301/automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst

    Or just for a specific installation:

    $ s3cmd ls s3://memory-dumps/nice-test301
    2023-08-30 07:57    101418514  s3://memory-dumps/nice-test301/automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst

    Tip

    s3cmd has to be set up in accordance with S3 Access via s3cmd (Optional).

  4. copy dump to local machine:

    $ s3cmd get s3://memory-dumps/nice-test301/automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst
  5. Decompress:

    zstd --rm -d automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst
  6. disable automatic dumps

    $ oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE-
    

    Warning

    This will restart Nice automatically!