In the past months we did some performance testing at work, making quite some mistakes on the way, and probably still doing some. In between, I’ve read "Java Concurrency in practice", by Brian Goetz, and learned once again some more goodies. As such, I’ve written this quick checklist, in order to forget less next time I’ll try to do some performance testing.
As usual, let me know of any mistake, lack or questions.
before starting the test
- check your issue still with the latest JVM to make sure the trouble still exists.
- use the latest JVM: java.util.concurrent had some great improvements in Java 6. Furthermore the various JVM options are more complete (includes display of Lock and its sub classes locking state).
while coding the test
- mind locking in utilities libraries. For example UUID.randomUUID() uses SecureRandom, which is thread safe… Prefer less secure random value generators which don’t imply locking, like System.nanoTime().
- measure time before and after all threads/tests, dividing it as needed, instead of many small measurements (less impacts on the tested code).
- do unpredictable stuff: the JVM is very good at optimizing out code whose result can be guessed, like by pre computing it once and for all. For example, compare some hashCode() with System.nanoTime(), printing out something in (highly unlikely) case of equality.
- do production representative work: the JVM always tries to optimize the code. It can take shortcuts which became invalids with further class loading. Restricting yourself to few classes increase the risk of optimization which won’t happen in production.
test launch set up
Be as close from production as possible
- mind the -server option, which has important effects.
- define your memory using -Xms and -Xmx (see java -X for more information).
setup runtime information display
add -verbose:gc to see garbage collection.
By default, 2 types of garbage collection exist:
minor ones (iterative), which garbages out the low hanging fruits. Quite frequent, these ones don’t affect performance too much. A typical minor collection is displayed this way in the console:
[GC 325407K->83000K(776768K), 0.2300771 secs]
It shows the decrease in memory consumption and the time it took to get rid of this memory.
"stop the world" garbage collection. Nothing else happens during these majors garbage collections, hence the stopping world name. As such, it’s a really performance hit: plan your test to run long enough to streamline its effects.
A stop the world collection is shown this way in the console:
[Full GC 267628K->83769K(776768K), 1.8479984 secs]
more options exist to monitor garbage collection, for example -XX:+PrintGCDetails. See Java HotSpot VM Options
and Tuning Garbage Collection with the 5.0 Java[tm] Virtual Machine.
- minor ones (iterative), which garbages out the low hanging fruits. Quite frequent, these ones don’t affect performance too much. A typical minor collection is displayed this way in the console:
add -XX:-PrintCompilation to see the JVM compiling bytecode to assembler code where it sees it fits.
- have accordingly a "warm up" period when running your test, to put this compilation out of the tested code.
- add -verbose:gc to see garbage collection.
while running the test
look at what’s going on in the JVM
- use jps to locate the test process.
use jstack to see the current stack.
- triggered frequently it gives a very nice overview of what the program is doing most of the time.
- displays information about lock held, waiting and deadlocking threads. More about it there Detecting Java Thread Deadlocks with ‘jstack’.
monitor your system
- what about the network traffic. Is the application network bound?
- what about CPU usage: a properly multi threaded software should make good use of them all.
- what about RAM, before starting and while running. Swapping shouldn’t be used normally.
- Be patient: the test should run some minutes to be conclusive…
all this time: test your intuitions/assumptions/known facts
- Surprises are lurking all over the place.
- Facts of yesterday are today’s illusions!
That’s all for now folks!
Well, I just wanted to speak about an issue for which I didn’t find any proper answer yet.
The matter is gui testing. Currently, at work, we do proper unit test for "framework level code". By "framework code", I mean code being in some utilities or services projects which are put there to be reused. These tests are usually fairly easy to do, since the context is well known. Furthermore, as once as we change this code, these tests prove their value. As changes are made quite often, be it only when adding new stuff and thus refactoring, we feel like these tests are good bang for bucks.
For example, recently, we saw a bug in some on "framework code". We wrote a failing test for it (well, I’m not 100% sure of that, but we should have had anyway ;)), fixed it and, on the go, spotted some possible improvements. We did them as well, then ran again all tests and we were pretty confident about it.
But, then, what about "project specific code" ?
Hum, what’s in it anyway ? Some "framework level code", for sure, which is already tested, and then some JPA, wicket and business related code. Testing the JPA code would basically mean testing the JPA layer, which is pointless. What about the GUI then ? Well, wicket isn’t exactly the easiest framework for it… The way its generates the ids make it hard(er) for most of the html test tools, like for example selenium. And even if, what to test actually ? Should we do some "clicks through" tests for all use cases ? All possible navigation paths ? And what about the data needed by these tests ? Using WicketTester, then we would be at pain with jquery heavy pages (well, I guess most of the selenium like frameworks are at pain in this case…). Overall, the efforts to put there are likely to be important.
So, what about the gain ? The odds that we’ll touch again these classes are quite low, way lower than framework level code. Basically, we don’t reuse these stuff. And if some functionalities/pages are fine enough, then we may even not touch them again before very long.
In the end, we have some layers which are, at the same time, harder to test and with less "return on investment". It looks like not testing these "project specific/gui layers" is a good idea…
That’s where my issue with ui test lies… This ui layer is a pain to test, and would be tested mainly for just a development, without much reuse. But, it’s an error prone layer. And it’s the most visible one… So, if, like me, you’re kind of fond of tests and bullet proof code, then you might well share my dilemma…
What to do then ? Well, for the time, I’ve no proper answer. I guess I should try to spend more time testing my ui before giving it to the outer world, even it feels awkward to do it manually. For sure, exploratory tests might help, but again they’re manual and in no way comprehensive… No clear solution here for me, unfortunately…
Any hint deeply welcome ! In fact, I guess that’s the (hidden) aim of this post : maybe someone’ll come with some magic bullet(s). After all, Christmas is coming soon, so it’s dreaming time anyway
The presentation mainly consisted of 2 parts : one about tests and agility, the others about Smartesting tool for… testing. Surprising isn’t
Anyway, the first part provided good insight on testing, with some surprise in it : they spoke of "Exploratory test". What’s that ? In a few words (the wikipedia tells way more), it’s just the idea that the developers should have time to test the application without strict guideline, relying instead on their knowledge and gust feeling.
Per se, the idea isn’t new, even if I never encountered a name for it up to now. Furthermore, we speaking more of it with Frédéric, it appeared that it could be done in different ways, like : let each developer do exploratory tests on other dev. code or plan some time for the whole team to do such tests before the sprint is over, in order to bullet proof it.
For sure, exploratory testing has to be part of a bigger testing plan. Furthermore, we tend all to do such kind of tests. Yet, to put a name on it and then plan for it is quite different, and may let enough time to actually do it, which sounds nice. For sure, there are drawbacks as well, notably about inefficiency if 2 teams members do the same test every time. As usual, communication could help here
Regarding Smartesting product, named Test Designer, it’s a Eclipse RCP application, using the work flow tools available there. Its main goal is to be able to describe some application work flow. In fact, it starts with some business requirements list (ordered by priority), then links them to part of some UML models describing the application’s work flow. Then, some guys have to make it happen, meaning that they have to write the tests needed by these UML models. This automation process, as they name it, is not part of the product.
I won’t say it’s revolutionary in itself, but to be able to use the same tools for both requirements gathering and functional testing sounds sensible.
They claim as well that it helps not having to redo all tests every time something change. Indeed, these functional tests are written per page (or I guess per page transition). As such, changing some part of the work flow involves only the related tests. Once again, nothing amazingly new, but this idea of slicing functional tests as well, to help maintain them, makes sense again.
In the end, as usual, one still has to provide the actual tests and then to maintain them. As such, it appears that this tool is used mainly by "big players". These big players include some in India… yes, they sell their app there, without developing it there… fun isn’t it ?
In the end, this talk provided me with elements on my quest to find some way to do proper GUI testing, so I enjoyed this session as well
2 points stroke me when reading this "Pragmatic Unit Testing" book :
- in order to easily test some class, they suggest to open the required methods by making them protected. I had first to check that protected included package readability (it does) but still I think it’s going a bit too far : if the method was supposed to be private, package private is the nearest visibility which would fit. I would only make it protected if I intent to override it. Do I miss something ?
- the last paragraph is called "Testing invalid parameters", and speaks of where to test the validity of the parameters. Their answer is "we find the easiest rule to adopt is the ‘keep the barbarians out at the gate" approach’, meaning internal components can trust the data they get. At work, we do it differently : every non private method has to check for the validity of its input (for which we have a dedicated assertion lib.). Would you suggest otherwise ?
During my holidays, I notably read Pragmatic Unit Testing In Java with JUnit.
Basically, this book might be nice for beginners and new comers to unit test. If you’ve never done any and are fearful of starting on your own, then consider it. Otherwise, it doesn’t bring much. It’s an easy book to read but the noise/info ratio is too much in favour of noise : I sometime had the feeling the authors were trying to fill the space.
In fact, I do like the Summary Card offered, it sums up nicely what is to be known, and what I could say is of interest in the book.
Furthermore, the book shows its age : late 2003 is like prehistory ;). So the code samples are a bit outdated and, worst, maven wasn’t popular at the time : lost of lines are lost explaining out to setup the tests’ folders, which is pointless with maven.
In case of questions/comments, do not hesitate.