Overview
We have introduced two new workloads, tradebeans and tradesoap, each based on the Apache DayTrader J2EE workload. Our code is currently based on the Geronimo 767347 svn revision. DayTrader was originally an IBM workload (Trade), and was subsequently donated to Apache and made open source.
We are very interested in community feedback on these workloads. Our hope is to get significant community feedback prior to our intended release of the next version of the suite in October 2009. If you have feedback, please send it to the mailing list or email Steve Blackburn directly.
Client, Server and Database
A real-world app-server will typically comprise clients, servers and a database, often each running on distinct hardware. Configuring a real app server is a complex process with many degrees of freedom. In the interests of coherent analysis of JVM performance, DaCapo's workloads remove those degrees of freedom, trading realism for methodological and analyitical clarity. Thus the benchmarks are executed within the Geronimo application server and utilize the Derby in-memory database that comes with Geronimo, both client and server run within a single JVM, and use the loopback device to communicate between client and server.
Beans and Soap
We include the DayTrader workload in two flavors; tradebeans and tradesoap. The beans and soap configurations correspond to different modes of client/server interaction. Tradebeans performs the entire workload and each of its operations directly on the server. On the other hand, tradesoap runs the workload on the client, communicating with the server at a fine grain, sending each request via the soap protocol. Both use exactly the same workload engine and the only difference in the code executed is that the trade API is implemented with direct calls in one case, and with indirect calls through soap in the other. The overhead of sending (relatively) fine grained operations to the server means that tradesoap has much lower throughput than tradebeans. Tradebeans' default configuration performs exactly the same workload as tradesoap's large configuration, and similarly tradebean's small configuration uses exactly the same workload as tradesoap's default configuration. Tradesoap is more representative of real workloads. We provide tradebeans to researchers as an interesting counterpoint.
The Workload
The workloads used by both tradebeans and tradesoap are scripted, meaning that exactly the same set of operations will be performed each time the workloads are executed (although the ordering and interleaving of operations will vary from run to run according to the way threads are scheduled). By contrast, the pre-existing DayTrader workloads were randomly generated on the fly (according to a prescribed probabilistic distribution of operation types). The workloads comprise a database of stocks, a database of users and a pool of user "sessions", where each session comprises a login, a string of operations, and a logout. The databases are rebuilt upon each iteration of each workload, so each iteration starts in the same state. The workloads are currently configured to concurrently execute 2P client threads, where P is the number of available processors. This is something we want to tune, and are interested in feedback on how we should configure threads for these workloads. Threads cooperatively work their way through the set of user sessions, each thread working through a single user session at a time.
The workloads were constructed using a perl script which can be found in the daytrader directory of the dacapo distribution (see below for directions on accessing it via svn). Researchers are free to generate new workloads using this script, however, a performance run of the benchmark will not validate unless the correct pre-computed workload is used. We are interested in user feedback on the workload construction. If you have improvements by all means pass them on.
We constructed a realistic set of stocks, based on real stock information (names, quotes, volumes) at the time (previously DayTrader had simply used random integers to reflect these names and values).
We also constructed a realistic set of randomized users, using real (randomized) names, addresses, etc. Each user has a randomly determined portfolio, although we took care not avoid a simple uniform distribution of stocks, instead skewing stocks on the basis of volume and/or market capitalization. We applied a similar approach when creating the workloads, biasing operations toward popular stocks. We believe that this will give more realistic levels of contention than simple uniform distributions. Each workload size requires a different number of users. We over-provision the number of users by 50% relative to the number of unique user-sessions. This means that the user database is (slightly) sparsely utilized. Realistically, this over-provisioning might be higher, however, initializing the user database is relatively expensive and we do not wish for it to dominate the benchmark execution. The current configurations are thus:
| tradebeans | | tradesoap | |
| config | users | sessions | users | sessions |
| small | 48 | 32 | 48 | 32 |
| default | 384 | 256 | 48 | 32 |
| large | 1536 | 1024 | 384 | 256 |
Finally, we constructed scripted workloads comprising user sessions. We have designed the workloads such that the smaller workloads are strict subsets of the larger workloads. The smaller workloads involve fewer users and by drawing those from the start of the user database, we are able to initialize the user database with a smaller user set for smaller workloads. This is significant because populating the user database is relatively expensive. Each user session comprises a mix of DayTrader operations. Some of these operations (such as "quote") are simple while others (such as "portfolio") are compound. For example, the portfolio operation first requests the users' holdings and then performs a quote operation on each holding (so a single portfolio operation may involve a number of quote operations). The workload generation script tries to achieve a random mix of operations according to target proportions. The table below shows each of the operations, the target mix and the actual mix for each of the three workload sizes. Note that "register" includes an implicit login (so logout = login + register).
| operation | target | 32 | 256 | 1024 |
|---|
| home | 20% | 20% | 18.8% | 18.8% |
| portfolio | 9% | 6.3% | 8.3% | 8.1% |
| quote | 45% | 44.6% | 41.6% | 41.5% |
| buy | 9% | 8.8% | 8.2% | 8.7% |
| sell | 9% | 7.2% | 8.0% | 7.9% |
| update | 1.5% | 1.2% | 1.5% | 1.4% |
| register | 0.5% | 0.6% | 0.6% | 0.5% |
| login | 6.0% | 5.0% | 6.2% | 6.2% |
| logout | 6.5% | 5.6% | 6.8% | 6.7% |
Typical output:
$ java -jar dacapo-9.12-bach.jar tradebeans
Using scaled threading model. 2 processors detected, 2 threads used to drive the workload, in a possible range of [1,512]
Booting Geronimo Kernel (in Java 1.6.0_16)...
Geronimo startup complete
Successfully created tables
===== DaCapo 9.12 tradebeans starting =====
Resetting database and populating with 1098 stocks...
Populating database with 768 users...
Finished repopulating database
Running 512 trade sessions directly on server
Completed 512 trade sessions comprising 8862 trader actions
Home .................... 1653 (18.7%)
Portfolio ............... 738 ( 8.3%)
Quote ................... 3758 (42.4%)
Buy ..................... 726 ( 8.2%)
Sell .................... 761 ( 8.6%)
Update .................. 126 ( 1.4%)
Register ................ 38 ( 0.4%)
Login ................... 512 ( 5.8%)
Logout .................. 550 ( 6.2%)
===== DaCapo 9.12 tradebeans PASSED in 11478 msec =====
Shutting down Geronimo...
$
Getting The Source
The latest version of the DaCapo benchmarks is always available via mercurial (hg):
hg clone http://dacapobench.hg.sourceforge.net:8000/hgroot/dacapobench/dacapobench
The benchmarks themselves reside within the benchmarks directory. The code is organized with daytrader as a library used by both tradebeans and tradesoap, and is built within the libs/daytrader directory. The easiest way to do this is to invoke ant from within the benchmarks directory with either tradebeans or tradesoap targets. Important: you need maven (mvn) in your path before you can build either of the trade workloads within ant, since the daytrader build includes nested calls to mvn which fail unless mvn is in the path. We include a recent maven release in the tools directory.
For convenience, and to encourage people to explore the source, we provide the sources pre-patched here. You can also see the patch against r767347 here.