I am co-organizing the Data Challenge Track at ICPE-2022 with Cor-Paul Bezemer and Weiyi Shang. The data challenge dataset is a snapshot of MongoDBs performance testing results and analysis (see the CFP for more detail). Recently, a group of researchers reached out to me with some questions for more detail about the data. I wanted to share the answers to those questions more widely, in case they are helpful to others working on the data challenge.
- MongoDB configuration
- We test representative versions of the major configurations (standalone, replica set, sharded cluster). We also have some smaller versions of those (1-node replica set, and shard-lite). These have essentially the same configuration, but fewer nodes in the system.
- We have some configurations with key features turned on and off (e.g., flow control, different security features).
- We have some configurations specifically designed for testing initial sync. They add a node to the cluster during the test.
- We also have tests that show up in "performance" project, as opposed to "sys-perf". These run on a single dedicated host, so are either standalone or a single node replica set.
- Hardware configuration
- Sys-perf runs predominantly on AWS c3.8xlarge instances with hyperthreading disabled, and processes pinned to a single socket.
- We have another configuration using different instance types, and a matching MongoDB configuration. Those were tuned to be closer to our Atlas configuration, and are labelled "M60-like".
- The "performance" project uses dedicated hardware.
A2: 28 in sys-perf, 4 in performance on our main branch. That may increase slightly when reviewing stable release branches (e.g., sys-perf-5.0).
A3: As discussed in https://dl.acm.org/doi/10.1145/3427921.3450234 we have added latency measures (median, 90th percentile, 95th percentile) client side measurements. We also collect some system metrics across the runs (cpu usage, IO usage, ...). We haven't talked much about those, but they are hopefully fairly self explanatory. If you find something completely cryptic, reach out and I'll see if I can explain it. Not all of the metrics are always interesting (e.g., Worker's Max is a client side metric that I think should be constant across each timeseries).
A4: Everything was deployed and run using DSI as described in https://dl.acm.org/doi/abs/10.1145/3395032.3395323 The configurations do change over time (slowly and infrequently), but they are version controlled so we can isolate exactly when they change. There are change points in the system associated with configuration changes and with workload changes.
A5: The largest set are c3.8xlarge AWS instances. Those were picked a while ago as they were the lowest noise option. The c3.8xlarge is the largest of the c3 family. It's not bare-metal, but it appears to be unshared and provides access to low level configuration.
Q6: What kind of workload has been applied (read/write ratio, complexity of queries, intensity, data set size, runtime) ?
- YCSB (load, 100% read, 50%read 50% update, ...) and a YCSB configuration with a much larger data set (YCSB_60GB)
- Linkbench (we are running from a private fork, but it's pretty close)
- Genny workloads (this is our internal load generator. We run from this public repo).
A7: No. We run all the above workloads directly, not by traces.