This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Thread and Memory Dumps

Creating a Thread Dump

1. find pod

   $ oc get pods -l run=nice --show-all=false
   NAME           READY     STATUS    RESTARTS   AGE
   nice-3-2nl1q   2/2       Running   0          49m

2. create thread dump

   $ oc exec -c nice nice-3-2nl1q -- stack-trace
   Full thread dump OpenJDK 64-Bit Server VM (25.151-b12 mixed mode):
   
   "pool-7-thread-1182" #4356 prio=5 os_prio=0 tid=0x00007f9b2c040000 nid=0x12ec waiting on condition [0x00007f9a765fa000]
     java.lang.Thread.State: TIMED_WAITING (parking)

Creating a Memory Dump Manually

1. find pod

   $ oc get pods -l run=nice --show-all=false
   NAME           READY     STATUS    RESTARTS   AGE
   nice-3-2nl1q   2/2       Running   0          49m

2. create dump

   $ oc exec -c nice nice-3-2nl1q -- memory-dump
   1:
   Dumping heap to /app/var/heap_dumps/manual-dump-2023-08-30T09:18:33.hprof ...
   Heap dump file created [664273346 bytes in 2.475 secs]
   Uploading '/app/var/heap_dumps/manual-dump-2023-08-30T09:18:33.hprof.zst' to 's3://memory-dumps/nice-demo/manual-dump-2023-08-30T09:18:33.hprof.zst'.

3. copy dump to local machine

   s3cmd get s3://memory-dumps/nice-demo/manual-dump-2023-08-30T09:18:33.hprof.zst

   Tip

   s3cmd has to be set up in accordance with S3 Access via s3cmd (Optional).

4. Decompress:

   zstd --rm -d manual-dump-2023-08-30T09:18:33.hprof.zst

Creating Memory Dump on OOM

Note

Memory dumps are always created when an OutOfMemoryError is thrown (see NICE2_DUMP_ON_OOM) and this is logged:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /app/var/heap_dumps/memory-dump-hlediiivhc.hprof ...

An easy way to see if any such dumps where created is to list all dumps.

1. enable automatic memory dumps

   oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE=true
   

   Warning

   This will restart Nice automatically!

   NICE2_MEMORY_DUMP_ON_TERMINATE	Create a memory dump on every pod termination. This does not only include pod termination as result of a pod restart triggered by the livenessProbe as result of an OOM-condition. Consequently, some dumps may not be related to an OOM-condition at all.
   NICE2_DUMP_ON_OOM	This is enabled by default. Tell the Java JVM to create a memory dump when an OutOfMemoryError is thrown. When memory is raising slowly, the application is often terminated due to a failing livenessProbe before such an exception is thrown.

2. wait for OOM crash

3. list dumps

   All dumps:

   $ s3cmd ls -r s3://memory-dumps
   2023-08-30 07:18    127709619  s3://memory-dumps/nice-demo/manual-dump-2023-08-30T09:18:33.hprof.zst
   2023-08-30 07:57    101418514  s3://memory-dumps/nice-test301/automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst

   Or just for a specific installation:

   $ s3cmd ls s3://memory-dumps/nice-test301
   2023-08-30 07:57    101418514  s3://memory-dumps/nice-test301/automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst

   Tip

   s3cmd has to be set up in accordance with S3 Access via s3cmd (Optional).

4. copy dump to local machine:

   $ s3cmd get s3://memory-dumps/nice-test301/automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst

5. Decompress:

   zstd --rm -d automatic-dump-2023-08-30T09-56-41-nice-558-qrw8c.hprof.zst

6. disable automatic dumps

   $ oc set env dc/nice NICE2_MEMORY_DUMP_ON_TERMINATE-
   

   Warning

   This will restart Nice automatically!