[Yanel-dev] [Fwd: Jackrabbit 3: Benchmark suite]

[Michael Wechner]

maybe something to learn from.

-------- Original Message --------
Subject: 	Jackrabbit 3: Benchmark suite
Date: 	Fri, 5 Mar 2010 14:32:09 +0100
From: 	Jukka Zitting <jukka.zitting at gmail.com>
Reply-To: 	dev at jackrabbit.apache.org
To: 	Jackrabbit Developers <dev at jackrabbit.apache.org>



Hi,

Many of the jr3 design threads call for performance or scalability
improvements in specific areas, or may have implications that harm
performance in others. We should be able to measure such changes much
better than with the ad-hoc tests we've been doing until now. To
achieve this, I'd like to see a JCR benchmark suite that's designed
like the backwards compatiblity test suite we have in the sandbox.

The test suite would consist of a base component where we define all
the performance tests we want to run. Each test should produce a
single number that indicates the amount of time, memory or other
resources consumed by the test. The test suite should take care of
eliminating the effect of things like repository or cache startup
costs and should run each test a few times to be able to calculate the
mean and standard deviation of the test results. This base component
should only depend on JCR 1.0.

In addition to the base component we'd have test components for all
Jackrabbit (and perhaps other JCR implementations) versions and
configurations we want to include in the test. All these components
should be included in a single Maven build so that we can we can
easily run all the tests on the same computer under similar conditions
and get comparable results. It's OK if the full test suite takes hours
or even days to complete, as we'd only need to run it for major new
releases and whenever major performance-affecting changes are made.

To start with, the initial tests I had in mind are:

* Session login/logout (both time and memory impact)
* The above plus reading a few nodes and properties (to simulate a
simple webapp use case)
* Tree traversal
* Writing N^K nodes (i.e. K layers of N nodes each; (N,K) in { (1e1,
6), (1e3, 2), (1e6, 1) })
* Reading an nt:file with N bytes (N in { 1e2, 1e4, 1e6 })
* Writing an nt:file with N bytes (N in { 1e2, 1e4, 1e6 })

We should probably extend this to cover at least search, versioning
and access control (and perhaps observation).

BR,

Jukka Zitting