Multithreaded performance testing checklist
In the past months we did some performance testing at work, making quite some mistakes on the way, and probably still doing some. In between, I’ve read "Java Concurrency in practice", by Brian Goetz, and learned once again some more goodies. As such, I’ve written this quick checklist, in order to forget less next time I’ll try to do some performance testing.
As usual, let me know of any mistake, lack or questions.
before starting the test
- check your issue still with the latest JVM to make sure the trouble still exists.
- use the latest JVM: java.util.concurrent had some great improvements in Java 6. Furthermore the various JVM options are more complete (includes display of Lock and its sub classes locking state).
while coding the test
- mind locking in utilities libraries. For example UUID.randomUUID() uses SecureRandom, which is thread safe… Prefer less secure random value generators which don’t imply locking, like System.nanoTime().
- measure time before and after all threads/tests, dividing it as needed, instead of many small measurements (less impacts on the tested code).
- do unpredictable stuff: the JVM is very good at optimizing out code whose result can be guessed, like by pre computing it once and for all. For example, compare some hashCode() with System.nanoTime(), printing out something in (highly unlikely) case of equality.
- do production representative work: the JVM always tries to optimize the code. It can take shortcuts which became invalids with further class loading. Restricting yourself to few classes increase the risk of optimization which won’t happen in production.
test launch set up
Be as close from production as possible
- mind the -server option, which has important effects.
- define your memory using -Xms and -Xmx (see java -X for more information).
setup runtime information display
add -verbose:gc to see garbage collection.
By default, 2 types of garbage collection exist:
minor ones (iterative), which garbages out the low hanging fruits. Quite frequent, these ones don’t affect performance too much. A typical minor collection is displayed this way in the console:
[GC 325407K->83000K(776768K), 0.2300771 secs]
It shows the decrease in memory consumption and the time it took to get rid of this memory.
"stop the world" garbage collection. Nothing else happens during these majors garbage collections, hence the stopping world name. As such, it’s a really performance hit: plan your test to run long enough to streamline its effects.
A stop the world collection is shown this way in the console:
[Full GC 267628K->83769K(776768K), 1.8479984 secs]
more options exist to monitor garbage collection, for example -XX:+PrintGCDetails. See Java HotSpot VM Options
and Tuning Garbage Collection with the 5.0 Java[tm] Virtual Machine.
- minor ones (iterative), which garbages out the low hanging fruits. Quite frequent, these ones don’t affect performance too much. A typical minor collection is displayed this way in the console:
add -XX:-PrintCompilation to see the JVM compiling bytecode to assembler code where it sees it fits.
- have accordingly a "warm up" period when running your test, to put this compilation out of the tested code.
- add -verbose:gc to see garbage collection.
while running the test
look at what’s going on in the JVM
- use jps to locate the test process.
use jstack to see the current stack.
- triggered frequently it gives a very nice overview of what the program is doing most of the time.
- displays information about lock held, waiting and deadlocking threads. More about it there Detecting Java Thread Deadlocks with ‘jstack’.
monitor your system
- what about the network traffic. Is the application network bound?
- what about CPU usage: a properly multi threaded software should make good use of them all.
- what about RAM, before starting and while running. Swapping shouldn’t be used normally.
- Be patient: the test should run some minutes to be conclusive…
all this time: test your intuitions/assumptions/known facts
- Surprises are lurking all over the place.
- Facts of yesterday are today’s illusions!
That’s all for now folks!