Introduction to Quickperf
QuickPerf is a test library for Java to quickly evaluate and improve the performance of your application.
The great interest of Quickperf lies in the fact that this is done via unit tests, which allows, after detecting and correcting a performance problem, to have non-regression tests so that it never comes back! You can then continuously evaluate the performance of your application via your continuous integration environment!
Quickperf can be used to simply measure performance characteristics, or to assert on it via the definition of expectation.
Quickperf requires Java 7 minimum and works with JUnit 4 or 5 and TestNG. The examples in this article are done in Java 11 via JUnit 5.
To use Quickperf you need to add the corresponding library to your test framework, here is an example with Maven for JUnit 5.
<dependencyManagement>
<dependencies>
<dependency>
<groupid>org.junit</groupid>
<artifactid>junit-bom</artifactid>
<version>5.7.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupid>org.quickperf</groupid>
<artifactid>quick-perf-bom</artifactid>
<version>1.0.1</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupid>org.junit.jupiter</groupid>
<artifactid>junit-jupiter-engine</artifactid>
<scope>test</scope>
</dependency>
<dependency>
<groupid>org.junit.platform</groupid>
<artifactid>junit-platform-launcher</artifactid>
<scope>test</scope>
</dependency>
</dependencies>
First example
Before going into detail about the features of Quickperf, a small example :
@QuickPerfTest
public class JvmAnnotationsJunit5Test {
@MeasureHeapAllocation
@Test
public void test_method_measuring_heap_allocation() {
// java.util.ArrayList: 24 bytes + Object[]: 16 + 100 x 4 = 416 => 440 bytes
ArrayList data = new ArrayList<>(100);
}
@ExpectNoHeapAllocation
@Test
public void should_not_allocate() {
}
}
@QuickPerfTest: is used to initialize the Quickperf JUnit5 extension (optional)@MeasureHeapAllocation: will measure the memory allocations made by the test@ExpectNoHeapAllocation: will create an expectation so that the test should not perform memory allocation. If this is false, the test will fail (assertion failure)
Running these tests will give you the following result:
[INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.quickperf.jvm.JvmAnnotationsJunit5Test [QUICK PERF] Measured heap allocation (test method thread): 440 bytes [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.203 s - in org.quickperf.jvm.JvmAnnotationsJunit5Test [INFO] [INFO] Results: [INFO] [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------
Build Success! Great!
By the way, you could notice Quickperf’s measurement of the memory allocation of the test test_method_measuring_heap_allocation:
[QUICK PERF] Measured heap allocation (test method thread): 440 bytes
Let’s now modify the code to add an expectation for the test test_method_measuring_heap_allocation so that it must allocate less than 400 bytes of memory.
@MeasureHeapAllocation
@ExpectMaxHeapAllocation(value = 400, unit = AllocationUnit.BYTE)
@Test
public void test_method_measuring_heap_allocation() {
// java.util.ArrayList: 24 bytes + Object[]: 16 + 100 x 4 = 416 => 440 bytes
ArrayList data = new ArrayList<>(100);
}
Running these modified tests will give you :
[INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.quickperf.jvm.JvmAnnotationsJunit5Test [QUICK PERF] Measured heap allocation (test method thread): 440 bytes [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.258 s <<< FAILURE! - in org.quickperf.jvm.JvmAnnotationsJunit5Test [ERROR] test_method_measuring_heap_allocation Time elapsed: 1.106 s <<< FAILURE! java.lang.AssertionError: a performance property is not respected [PERF] Expected heap allocation (test method thread) to be less than 400 bytes but is 440 bytes. [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] JvmAnnotationsJunit5Test.test_method_measuring_heap_allocation a performance property is not respected [PERF] Expected heap allocation (test method thread) to be less than 400 bytes but is 440 bytes. [INFO] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------
Build Failure 🙁
But with a human readable explanation of the cause.
[PERF] Expected heap allocation (test method thread) to be less than 400 bytes but is 440 bytes.
One of the specificities of Quickperf is to give you as much information as possible about the reason of the failure of the test, through sentences understandable by a human being.
Core annotations
Core annotations are available within Quickperf’s main library, they offer the following features:
@MeasureExecutionTime: measures the execution time of the test method.@ExpectMaxExecutionTime: create an expectation that the test should not run for more than a certain amount of time.
There are also a number of annotations to configure Quickperf, more information here.
JVM annotations
Most JVM annotations are included in the main Quickperf library except the annotations related to JFR – JDK Flight Recorder which require the addition of the library org.quickperf:quick-perf-jfr-annotations.
Warning: the use of JVM annotation causes a JVM to be started for each test method.
JVM annotations provide the following features :
- Configuration of the JVM that will run the test via
@HeapSize(to configure Xmx and Xms),@Xms,@Xmx,@UseGC,@EnableGcLoggingand@JvmOptions. - Measurement of memory allocations via
@MeasureHeapAllocation,@ExpectMaxHeapAllocationand@ExpectNoHeapAllocation. Uses the library ByteWatcher. - Measurement of the resident set size of your JVM process via
@MeasureRSSand@ExpectMaxRSS. - Profiling your application via JFR.
Small zoom on the profiling of your application via JFR. The annotation @ProfileJvm will launch a JFR record for the execution of your method that you can then open with JMC – JDK Mission Control. This annotation will also display some global JFR metrics in the console.
------------------------------------------------------------------------------ ALLOCATION (estimations) | GARBAGE COLLECTION | THROWABLE Total : 20,1 MiB | Total pause : 21,434 ms | Exception: 0 Inside TLAB : 16,2 MiB | Longest GC pause: 21,434 ms | Error : 0 Outside TLAB: 3,95 MiB | Young: 1 | Throwable: 0 Allocation rate: 255 MiB/s | Old : 0 | ------------------------------------------------------------------------------ COMPILATION | CODE CACHE Number : 0 | The number of full code cache events: 0 Longest: 0 | ------------------------------------------------------------------------------
It is possible to define expectations for this profile via the annotation @ExpectNoJvmIssue which will be based on the JMC rules to fail the test.
Rule: Primitive To Object Conversion Severity: INFO Score: 73 Message: 78 % of the total allocation (15,1 MiB) is caused by conversion from primitive types to object types. The most common object type that primitives are converted into is 'java.lang.Integer', which causes 15,1 MiB to be allocated. The most common call site is 'org.quickperf.jvm.IntegerAccumulator.accumulateInteger(int):26'. [...]
More information on JVM annotations here
SQL annotations
To use SQL annotations, you need to add the library org.quickperf:quick-perf-sql-annotations (unless you use Spring which has dedicated support). Depending on your application, you then need to configure its DataSource to add a proxy that will allow Quickperf to trace SQL queries. More information about this in the next section.
Quickperf has a lot of annotations to check the execution of SQL queries, and to detect the most common problems: duplicate queries, n+1 select, batch, …
Here are some of the most used annotations:
@ExpectJdbcQueryExecutionand@ExpectMaxQueryExecutionTime: expectation as for the execution time of a SQL query, the first one expect the exact time, the second one will fail if the measured time is greater (most SQL annotations come in pairs according to this same scheme).@ExpectSelectedColumnetand@ExpectMaxSelectedColumn: expectation as to the number of selected columns@ExpectSelectand@ExpectMaxSelect: expectation as to the number of selects generated/li>@ExpectJdbcBatching: expectation as to the usage of JDBC batching
Small zoom on a particular annotation : @DisableSameSelectTypesWithDifferentParamValues, this one allows to detect problems with n+1 select with JPA / Hibernate..
This well known problem happens when two entities have a parent-child relationship, and loading the parent entity involves a select query to load it, plus n select to load its children. It is a well-known anti-pattern that is usually solved by performing a JPA JOIN FETCH.
Quickperf can detect this with the annotation @DisableSameSelectTypesWithDifferentParamValues which will cause your test to fail if several identical selects are executed with different parameters. And if it detects the presence of Hibernate, Quickperf will even give you advice on how to solve the problem!
Let’s imagine the following Player and Team entities:
@Entity
public class Player implements Serializable {
@Id
@GeneratedValue(strategy = GenerationType.SEQUENCE)
private Long id;
private String firstName;
private String lastName;
@ManyToOne(targetEntity = Team.class)
@JoinColumn(name = "team_id")
private Team team;
// getter and setters ...
}
@Entity
public class Team implements Serializable {
@Id
@GeneratedValue(strategy = GenerationType.SEQUENCE)
private Long id;
private String name;
// getter and setters ...
}
And the following test:
@DisableSameSelectTypesWithDifferentParamValues
@Test
public void should_find_all_players() {
TypedQuery fromPlayer = entityManager.createQuery("FROM Player", Player.class);
List players = fromPlayer.getResultList();
assertThat(players).hasSize(2);
}
Loading the class Player will result in n+1 SQL queries which will be detected by Quickperf which will then give you the following error message suggesting a problem with n+1 select and possible solutions :
ERROR] Failures:
[ERROR] HibernateJUnit5Test.should_find_all_players a performance property is not respected
[PERF] Same SELECT types with different parameter values
💣 You may have even more select statements with production data.
Be careful with the cost of JDBC server roundtrips: https://blog.jooq.org/2017/12/18/the-cost-of-jdbc-server-roundtrips/
💡 Perhaps you are facing an N+1 select issue
* With Hibernate, you may fix it by using JOIN FETCH
or LEFT JOIN FETCH
or FetchType.LAZY
or ...
Some examples: https://stackoverflow.com/questions/32453989/what-is-the-solution-for-the-n1-issue-in-jpa-and-hibernate";
https://stackoverflow.com/questions/52850442/how-to-get-rid-of-n1-with-jpa-criteria-api-in-hibernate/52945771?stw=2#52945771
https://thoughts-on-java.org/jpa-21-entity-graph-part-1-named-entity/
[...]
Frameworks integration
As mentioned in the previous section, Quickperf requires a proxy from your DataSource to be able to detect SQL problems. Depending on the frameworks you use, you will need to set up this proxy.
This consists of a few lines of code that will define a wrapper on top of your DataSource:
public DataSource wrapDatasource(DataSource dataSource){
ProxyDataSource proxyDataSource = QuickPerfSqlDataSourceBuilder.aDataSourceBuilder()
.buildProxy(dataSource);
return proxyDataSource;
}
The repository quickperf-examples contains many examples of implementation of this proxy for the most common frameworks (Spring, Hibernate, Micronaut, …).
If you use Spring, the configuration is simplified, you don’t need to add the library org.quickperf:quick-perf-sql-annotations, instead you have to add a dedicated library and everything is done automatically. For example, for Spring Boot 2, you have to add the starter quick-perf-springboot2-sql-starter.
<dependency>
<groupid>org.quickperf</groupid>
<artifactid>quick-perf-springboot2-sql-starter</artifactid>
<scope>test</scope>
</dependency>
More information about SQL annotation support with Spring here.
Quickperf will not only detect performance problems, it will also give advice on how to solve them by listing the most common causes. It will detect the frameworks used by your application (Hibernate, Spring Boot, …) and customize the advice given accordingly.
According to the detected frameworks, Quickperf will also suggest solutions to the problems of using Quickperf itself.
For example, in case of n+1 problem with an application using Spring Data JPA, Quickperf will give you a help on how to fix it specifically with Spring Data JPA (the tips for Hibernate have been removed from the example to improve its readability).
[PERF] You may think that <1> select statement was sent to the database
But there are in fact <3>...
💣 You may have even more select statements with production data.
Be careful with the cost of JDBC roundtrips: https://blog.jooq.org/2017/12/18/the-cost-of-jdbc-server-roundtrips/
💡 Perhaps you are facing an N+1 select issue
* With Hibernate [...]
* With Spring Data JPA, you may fix it by adding @EntityGraph(attributePaths = { "..." })
on repository method: https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.entity-graph
Conclusion
Quickperf is a tool that allows you to add continuous performance non-regression thanks to its integration via test frameworks (JUnit, TestNG).
It allows you to detect the most common performance problems in Java (memory allocation, querying, …), to profile your application via a test, and more. Once these problems have been detected and corrected thanks to the advice it provides, your test will become the guarantee of non-regression in the performance of your application.
Quickperf was successfully used to measure Maven’s memory consumption improvements between versions 3.6.1 and 3.6.2. As a result, a bench has been created to continuously monitor Maven’s memory allocations. This is a good example of how Quickperf was used on a complex project and how much it saved.