In a previous article, I mentioned how to profile a native GraalVM image with perf. If you are not familiar with the GraalVM tool and the limitations it brings, I suggest you reread my article, or at least the beginning of it.

As seen in my previous article, a native image will contain a minimalist JVM, called SubstrateVM, which does not support JVM-TI, and therefore does not allow the use of Java debuggers. To debug a native image, you therefore need a system debugger (OS-level), here I suggest using GDB – The GNU Project Debugger which works on most Linux distributions as well as MacOS.

First, we will create a Hello World Quarkus project with the following Maven command:

mvn io.quarkus:quarkus-maven-plugin:1.13.4.Final:create \
    -DprojectGroupId=fr.loicmathieu.quarkus.debug \
    -DprojectArtifactId=helloworld-debug \
    -DclassName="fr.loicmathieu.quarkus.debug.GreetingResource" \
    -Dpath="/hello"

Next, we’ll modify the GreetingResource class to take as a parameter the name of the person to say Hello to:

@Path("/hello")
public class GreetingResource {

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String hello(@QueryParam("name") String name) {
        return "Hello " + (name == null ? "World" : name);
    }
}

You can then launch the application via mvn quarkus:dev, and test it via curl localhost:8080/hello/myself.

We will now package the application into a native image, including the debug symbols so that our debugger can find them. To do this we need to use the following command: mvn package -Pnative -Dquarkus.native.debug.enabled=true -DskipTests.

• -Pnative: tells Maven to use the native profile which tells the Maven Quarkus plugin to generate a native image of your application. The Quarkus plugin will take care of everything for you (including generating the very long native-image command line).
• quarkus.native.debug.enabled=true: configuration option that enables debugging features for native applications. It allows to include debug symbols in the native image. For more information about debug symbols with the native image tool: DebugInfo
• -DskipTests: we don’t run the tests because as we modified the resource code but not the tests, they won’t work

If all goes well, after a minute or so, you will have your native image in the target directory. But before launching it in debug mode, we need to copy the sources of our libraries so that GDB can access them (the sources of your code are already available for GDB).
For this we can use the following Maven command: mvn dependency:sources.

If all went well, you will have in the target directory the following directories and files:

• helloworld-debug-1.0.0-SNAPSHOT-runner: the native image of your application
• helloworld-debug-1.0.0-SNAPSHOT-runner.debug: the debug symbols
• sources: the source directory of the application and its third-party libraries

On Linux or MacOS, if you don’t have the helloworld-debug-1.0.0-SNAPSHOT-runner.debug file, it is because you are missing the binutils package. The debug symbols will then be integrated into the native image.

On classic Linux distributions, you can easily install GDB from the repos of your distribution. On Ubuntu, it’s as simple as sudo apt-get install gdb.

To run your application in debug mode, go to the target directory and then run gdb helloworld-debug-1.0.0-SNAPSHOT-runner. GDB will automatically detect your application’s symbols and sources, as we are using the default files and directories.

At this point, your application is not yet launched. Before launching it, we will put a breakpoint in the beginning of our resource’s method, so we can debug it via break GreetingResource.java:1. This will put a breakpoint on the first line of instruction in your class.

Then we can launch the application via run&, the & allows you to go back to the GDB shell (hit enter if you don’t see the GDB shell prompt).

At this point, the application is launched, we can use it. You can call your endpoint via your browser, or a curl in an external shell. GDB even allows you to call a shell command directly from its own shell via shell curl localhost:8080/hello?name=me &.

You should normally have a message in the GDB shell that the breakpoint has been reached:

Thread 31 "ecutor-thread-2" hit Breakpoint 1, fr.loicmathieu.quarkus.debug.GreetingResource.hello(java.lang.String)(void) () at fr/loicmathieu/quarkus/debug/GreetingResource.java:15
15          return "Hello " + (name == null ? "World" : name);

Now we will run some commands to debug our application.

To see the value of a variable we will use print.

(gdb) print name
$6 = '\000' 

Unfortunately, strings are seen as arrays of chars, and GDB displays the first element. This is not very useful here.

To see what the application’s threads are doing, type info threads. You will then get an output like the following.

  Id   Target Id                                            Frame 
  1    Thread 0x7ffff7d6f2c0 (LWP 154724) "helloworld-debu" futex_wait_cancelable (private=, expected=0, futex_word=0x2223dfc) at ../sysdeps/nptl/futex-internal.h:183
  2    Thread 0x7ffff6bff640 (LWP 154729) "gnal Dispatcher" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x221c060 )
    at ../sysdeps/nptl/futex-internal.h:320
  4    Thread 0x7ffff55ff640 (LWP 154731) "ecutor-thread-1" futex_abstimed_wait_cancelable (private=, abstime=0x7ffff55fec78, clockid=-178263168, expected=0, futex_word=0x7fffe8000dac)
    at ../sysdeps/nptl/futex-internal.h:320
  5    Thread 0x7ffff4bff640 (LWP 154732) "-thread-checker" futex_abstimed_wait_cancelable (private=, abstime=0x7ffff4bfebe8, clockid=-188749072, expected=0, futex_word=0x7fffec000da8)
    at ../sysdeps/nptl/futex-internal.h:320
  6    Thread 0x7fffe7fff640 (LWP 154733) "ntloop-thread-0" 0x00007ffff7e8956e in epoll_wait (epfd=9, events=0x2227f90, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  7    Thread 0x7fffe75ff640 (LWP 154734) "ntloop-thread-1" 0x00007ffff7e8956e in epoll_wait (epfd=12, events=0x222afa0, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  8    Thread 0x7fffe6dfe640 (LWP 154735) "ntloop-thread-2" 0x00007ffff7e8956e in epoll_wait (epfd=15, events=0x222dfb0, maxevents=1024, timeout=1800000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  9    Thread 0x7fffe61ff640 (LWP 154736) "ntloop-thread-3" 0x00007ffff7e8956e in epoll_wait (epfd=18, events=0x2230fc0, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
  10   Thread 0x7fffe57ff640 (LWP 154737) "ntloop-thread-4" 0x00007ffff7e8956e in epoll_wait (epfd=21, events=0x2233fd0, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
[...]

The backtrace command allows you to retrieve the breakpoint call stack:

#0  fr.loicmathieu.quarkus.debug.GreetingResource.hello(java.lang.String)(void) () at fr/loicmathieu/quarkus/debug/GreetingResource.java:15
#1  0x00000000004e1300 in com.oracle.svm.reflect.GreetingResource_hello_ff2c07ee54b83f7b7e9993013704c94177591f6c_119.invoke(java.lang.Object, java.lang.Object[])(void) ()
    at com/oracle/svm/core/graal/snippets/TypeSnippets.java:75
#2  0x0000000000f04dd7 in org.jboss.resteasy.core.MethodInjectorImpl.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object, java.lang.Object[])(void) ()
    at java/lang/reflect/Method.java:566
#3  0x0000000000f048fa in org.jboss.resteasy.core.MethodInjectorImpl.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object)(void) ()
    at org/jboss/resteasy/core/MethodInjectorImpl.java:130
#4  0x0000000000f11cdd in org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object)(void) ()
    at org/jboss/resteasy/core/ResourceMethodInvoker.java:660
#5  0x0000000000f129e3 in org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object)(void) ()
    at org/jboss/resteasy/core/ResourceMethodInvoker.java:524
#6  0x0000000000f103b8 in org.jboss.resteasy.core.ResourceMethodInvoker..Lambda.1337/1751592192.get(void) ()
#7  0x0000000000f2c557 in org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(void) ()
    at org/jboss/resteasy/core/interception/jaxrs/PreMatchContainerRequestContext.java:364
#8  0x0000000000f126d0 in org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object)(void) ()
    at org/jboss/resteasy/core/ResourceMethodInvoker.java:476
#9  0x0000000000f123a9 in org.jboss.resteasy.core.ResourceMethodInvoker.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object)(void) ()
    at org/jboss/resteasy/core/ResourceMethodInvoker.java:433
#10 0x0000000000f121a5 in org.jboss.resteasy.core.ResourceMethodInvoker.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse)(void) ()
    at org/jboss/resteasy/core/ResourceMethodInvoker.java:402
#11 0x0000000000f2381c in org.jboss.resteasy.core.SynchronousDispatcher.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, org.jboss.resteasy.spi.ResourceInvoker)(void) ()
    at org/jboss/resteasy/core/ResourceMethodInvoker.java:69
#12 0x0000000000f23d90 in org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse)(void) ()
    at org/jboss/resteasy/core/SynchronousDispatcher.java:261
[...]

You can then use the step command to go to the next statement, and continue& to go to the next breakpoint.

When your debugging session is over, you can exit GDB with quit.

If take tall the commands we just runs, it looks like this:

(gdb) break GreetingResource.java:1
Breakpoint 1 at 0x5158d0: file fr/loicmathieu/quarkus/debug/GreetingResource.java, line 15.
(gdb) run&
Starting program: /data/dev/helloworld-debug/target/helloworld-debug-1.0.0-SNAPSHOT-runner 
(gdb) [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff6bff640 (LWP 154729)]
[New Thread 0x7ffff61ff640 (LWP 154730)]
[Thread 0x7ffff61ff640 (LWP 154730) exited]
[...]
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2021-05-27 12:45:05,992 INFO  [io.quarkus] (main) helloworld-debug 1.0.0-SNAPSHOT native (powered by Quarkus 1.13.4.Final) started in 0.064s. Listening on: http://0.0.0.0:8080
2021-05-27 12:45:05,999 INFO  [io.quarkus] (main) Profile prod activated. 
2021-05-27 12:45:05,999 INFO  [io.quarkus] (main) Installed features: [cdi, resteasy]
(gdb) shell curl localhost:8080/hello?name=me &
(gdb) 
Thread 31 "ecutor-thread-2" hit Breakpoint 1, fr.loicmathieu.quarkus.debug.GreetingResource.hello(java.lang.String)(void) () at fr/loicmathieu/quarkus/debug/GreetingResource.java:15
15          return "Hello " + (name == null ? "World" : name);
(gdb) print name
$6 = '\000' 
(gdb) info threads
  Id   Target Id                                            Frame 
  1    Thread 0x7ffff7d6f2c0 (LWP 154724) "helloworld-debu" futex_wait_cancelable (private=, expected=0, futex_word=0x2223dfc) at ../sysdeps/nptl/futex-internal.h:183
  2    Thread 0x7ffff6bff640 (LWP 154729) "gnal Dispatcher" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x221c060 )
    at ../sysdeps/nptl/futex-internal.h:320
  4    Thread 0x7ffff55ff640 (LWP 154731) "ecutor-thread-1" futex_abstimed_wait_cancelable (private=, abstime=0x7ffff55fec78, clockid=-178263168, expected=0, futex_word=0x7fffe8000dac)
    at ../sysdeps/nptl/futex-internal.h:320
[...]
(gdb) backtrace
#0  fr.loicmathieu.quarkus.debug.GreetingResource.hello(java.lang.String)(void) () at fr/loicmathieu/quarkus/debug/GreetingResource.java:15
#1  0x00000000004e1300 in com.oracle.svm.reflect.GreetingResource_hello_ff2c07ee54b83f7b7e9993013704c94177591f6c_119.invoke(java.lang.Object, java.lang.Object[])(void) ()
    at com/oracle/svm/core/graal/snippets/TypeSnippets.java:75
#2  0x0000000000f04dd7 in org.jboss.resteasy.core.MethodInjectorImpl.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object, java.lang.Object[])(void) ()
    at java/lang/reflect/Method.java:566
#3  0x0000000000f048fa in org.jboss.resteasy.core.MethodInjectorImpl.invoke(org.jboss.resteasy.spi.HttpRequest, org.jboss.resteasy.spi.HttpResponse, java.lang.Object)(void) ()
    at org/jboss/resteasy/core/MethodInjectorImpl.java:130
[...]
(gdb) step
[Thread 0x7ffff55ff640 (LWP 154731) exited]
[New Thread 0x7ffff61ff640 (LWP 154825)]
[Switching to Thread 0x7ffff61ff640 (LWP 154825)]

Thread 32 "ecutor-thread-3" hit Breakpoint 1, fr.loicmathieu.quarkus.debug.GreetingResource.hello(java.lang.String)(void) () at fr/loicmathieu/quarkus/debug/GreetingResource.java:15
15          return "Hello " + (name == null ? "World" : name);
(gdb) step
Hello mejava.lang.String.valueOf(java.lang.Object)(void) () at java/lang/String.java:2951
2951            return (obj == null) ? "null" : obj.toString();
(gdb) continue&
Continuing.
(gdb) Hello me
(gdb) quit

To conclude, even if debugging from the command line is not necessarily very practical, GDB allows us to have a view on the internal execution of our application when it is packaged as a native image relatively easily. It is an interesting tool to have in your toolbox.