Repeatable Performance Tests: EBS Instances are the Stable Option

 Note: This was originally published on the MongoDB Engineering Blog on April 30, 2019 here by Henrik Ingo and myself. Please read it there assuming the link works. I have copied it here to ensure the content does not disappear. The links in the article are the original links.

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. At the beginning of the project, it was unclear whether our goal of running repeatable performance tests in a public cloud was achievable. Instead of debating the issue based on assumptions and beliefs, we decided to measure noise itself and see if we could make configuration changes to minimize it.

After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. However, we found that the results of repeated tests had a high variance. Given our test data and our knowledge of the production system, we had observed that many of the noisiest tests did the most IO (being either sensitive to IO latency, or bandwidth). After performing the first baseline tests, we therefore decided to focus on IO performance through testing both AWS instance types and IO configuration on those instances.

Investigate IO

As we are explicitly focusing on IO in this step, we added an IO specific test (fio) to our system. This allows us to isolate the impact of IO noise to our existing benchmarks. The IO specific tests focus on:

• Operation latency

• Streaming bandwidth

• Random IOPs (IO per second)

We look first at the IO specific results, and then our general MongoDB benchmarks. In the below graph, we are graphing the "noise" metric as a percentage computed from (max-min)/median and lower is better. c3.8xlarge with ephemeral storage is our baseline configuration which we were using in our production environment.

i2.8xlarge shows best results with low noise on throughput and latency

The IO tests show some very interesting results.

1. The c3.8xlarge with EBS PIOPS shows less noise than the c3.8xlarge with its ephemeral disks. This was quite unexpected. In fact the c3.8xlarge with ephemeral storage (our existing configuration) is just about the worst choice.

2. The i2.8xlarge looks best all around with low noise on throughput and latency.

3. The c4.8xlarge shows higher latency noise than the c3.8xlarge. We would have expected any difference to favor the c4.8xlarge instances, as they are EBS optimized.

After these promising results, we examined the results of our MongoDB benchmarks next. At the time that we did this work, MongoDB had two storage engines (wiredTiger and MMAPv1), with MMAPv1 being the default, but now deprecated, option. There were differences in the results between the two storage engines, but they shared a common trend.

c3.8xlarge with PIOPS performs best with all results below 10% noise for the wiredTiger storage engine

c3.8xlarge with PIOPS performs best with most results below 10% noise for the mmap storage engine

There were no configurations that were best across the board. That said, there was a configuration with below 10% noise for all but 1 test: c3.8xlarge with EBS PIOPS. Interestingly, while i2 was the best for our focused IO tests, it was not for our actual tests.

Valuable lessons learned:

• As far as repeatable results are concerned, the "local" SSDs we had been using performed worse compared to any other alternative we could have possibly chosen!

• Contrary to popular belief, when using Provisioned IOPS with EBS, the performance is both good in absolute terms, and very very stable! This is true for our IO tests and our general tests. The latency of disk requests does have more variability than the SSD alternatives, but the IOPS performance was super stable. For most of our tests, the latter is the important characteristic.

• The i2 instance family has a much higher performance SSD, and in fio tests showed almost zero variability. It also happens to be a very expensive instance type. However, while this instance type was indeed a great choice in theory, it turns out that our MongoDB test results were quite noisy. Upon further investigation, we learned that the noisy results were due to unstable performance of MongoDB itself. As i2.8xlarge has more RAM than c3.8xlarge, MongoDB on i2.8xlarge is able to hold much more dirty data in RAM. Flushing that much dirty data to disk was causing issues.

Switching from ephemeral to EBS disks in production

Based on the above results, we changed our production configuration to run on EBS disks instead of ephemeral SSD. (We were already running on c3.8xlarge instance types, which turned out to have the lowest noise in the above comparison, so decided to keep using those.)

Performance becomes more stable when using EBS

After running with the changes for a couple of weeks, you could clearly see how the day-to-day variation of test results decreased dramatically. This instantly made the entire System Performance project more useful to the development team and MongoDB as a whole.

Conclusion

Focusing on IO performance proved useful. As it turns out using Ephemeral (SSD) disks was just about the worst choice for our performance test. Instead, using Provisioned IOPS showed the most stable rate results. While i2 instances were the best in our non-MongoDB benchmark tests, they proved less than ideal in practice. This highlights quite clearly that you need to measure your actual system and assume nothing to get the best results.

This is the second of three bigger experiments we performed in our quest to reduce variability in performance tests on EC2 instances. You can read more about the top level setup and results as well as how we found out that EC2 instances are neither good nor bad and that CPU options are best disabled.

Reading to Answer Questions

Putting down my book, I stood up from the bench and walked over to my 10-year-old son by the water. His fishing line was tangled. Again. This was the third time he had needed me to fix something. He hadn't caught a single fish and was getting very upset. He loved the idea of fishing and had visions of catching impressive fish like those caught in the videos he watched. Sadly, he was not good at fishing and neither was I.

I recently wrote about how to read academic papers in volume. When I researched how to read academic papers, I also learned how to read more effectively in general. Those skills have helped me better achieve my goals, such as helping my son catch fish.

This post covers how I use books to answer my bigger questions, and is largely based on ideas from the book How to Read a Book by Mortimer Adler.

My son holding a small fish he caught.

---

Learning Through Books

To help my son catch fish, I had to learn both facts and skills. I needed to learn about a subject (fishing) and a skill (how to catch fish). It was one of my larger questions, requiring time and effort. I’ve investigated other larger questions focused on work problems such as working with a challenging colleague, personal goals such as remembering things better, and personal challenges such as supporting my family with a medical issue. The best way to learn facts and skills quickly is to learn from those who have already learned the topic. Books are a treasure trove of other people’s learnings.

I follow three main steps on a learning project:

1. Find many potential resources

2. Filter those results to a manageable number of the best resouces

3. Extract what I need from those resources. 

In my life I am constantly asking questions and searching for answers. Only a few of those questions merit the time and effort required for this process. For those that do, I adjust my effort to my question and stop when I have what I need.

Find Many Potential Resources

I start with a wide search using the internet and my local library. I also ask friends, colleagues, and social media for suggestions. When I do this right, I turn up a lot of resources. For each resource, I’ll quickly check what I can about it, such as summaries and reviews, to see whether I think this book might be on topic. Key word searches often turn up things that are clearly unrelated to my question. For example, Incredible--and True!--Fishing Stories may or may not be entertaining, but it’s not going to help my son catch fish. I can immediately reject that book.

 
My search extends beyond books to include articles, blogs, podcasts, and videos. Books tend to be higher quality and I have a personal bias to the written word, but I want the best available resources regardless of its medium. 

Then I do a second search based on the results from the first search. Did I find other search terms? What other resources are related to these? Do any of these books reference other resources?
Filter For The Best Resources

Now I have a large list of books and other resources. I request as many of the books as I can from my local library. I download or bookmark the online resources. Then I scan all of these together quickly so that I can think about the ideas from multiple books at the same time. Speed is essential for that mixing of ideas. For each book I want to know:

 

1. What is this book about?
2. How does it address my question? What techniques does it propose? 
3. What words does it use for my question? What do they mean? 
4. Do I believe the answers this book proposes? Alternatively, is my bullshit detector going off? 

I quickly inspect each book, scanning the table of contents, reading the publisher's blurb, skimming the end of the last chapter, and checking for summaries at the end of key chapters. I take notes on anything that addresses my prompts, anything that catches my attention, and any new questions that arise as I read. By the time I’m done, the books are usually covered in sticky notes.

I look for common words and ideas, and I usually see patterns across the books. Sometimes several of the books have the same "revolutionary" or "game-changing" ideas. This used to annoy me, but I’ve since learned this is a great result. This common “revolutionary” idea is likely what I'm looking for. Other times I learn from overlapping words and ideas in the books, even if the books don't agree on everything. When they disagree I can see which ideas stand up to criticism and which fall over.

At this point, I have two follow-up questions:

1. Does one book capture a common idea better than the others? 
2. Are there interesting differences between the books? 

If one book does capture the key idea better than the others, I focus on that one. When there are interesting differences, I learn more from reading the books together than by reading them separately, just as I learned more from scanning them together.

Extract Answers 

I now have at least one book to read deeply. First, I revisit my existing notes to answer questions: What do I expect to get out of each book? What is common between them? What words do they use for important items? Where do they differ?

 
From these answers I form focused questions for each book: Book A says X, what does Book B say about that? Book C and D seem to differ on point C. Is one of them right? Is there something more complicated going on? Book E claims point D. What’s the evidence for it?

Next, I read each book, trying to answer my questions, still taking copious notes. I listen for the conversations among the books. The differences among books may be true disagreements requiring real thought and evaluation on my part. Or they may merely be different framings of the same idea. The common themes become clearer, and the differences enlighten me. In both cases I end up with a deeper understanding of the underlying issue.

 
When I’ve finished reading everything, I review and organize my notes. I often write a summary for myself. I hopefully have answers to my questions. If I don’t, I restart the process by looking for more sources focused on whatever is missing. If I’m really excited about the result, I may write a blog post or share my learnings with others.

Catching Fish

That day that I took my son fishing, I asked him two questions when we were back home: Would you like my help figuring out how to catch fish? Yes, he did. I told him if he wanted my help, we had to use the goal of "catch fish — any kind of fish." We couldn't only aim for the big fish he'd seen in videos. Would that still be worthwhile? He thought briefly: Yes, it would be.

 
I then got every book about fishing I could from my local library. I scanned each of them and noticed trends. It was clear that we should start by targetting small fish such as sunfish, with small hooks and bobbers, and we should fish in smaller bodies of water. I bought a couple of the more promising books, as well as the appropriate fishing gear, including the small hooks and bobbers. We then went to a local pond and he caught fish! That summer he caught a lot of fish. It was wonderful for him, and wonderful for me to help my son succeed at something that he loved so much.

 
I hope this post helps you achieve similar success with your large questions. I would love to hear about it when it does.

Reading Papers in Volume

I sat at my desk with a pile of academic papers next to me waiting to be read. I worked at a research lab and reading papers was part of my job. My colleagues amazed me by reading scores of academic papers, calling up the right paper at the right time. I couldn’t compete. I couldn’t even keep up. I had received my Ph.D. a few years prior and felt I should have been able to. I wondered if I just didn’t belong at a research lab.

 
Then I wondered if I was missing something. I searched the internet and quickly found I was. I found Steve McConnel’s How to Read a Technical Article and Greg Phillips How to read technical papers—in quantity! Steve McConnel’s article was itself based upon (and acknowledged) How to Read a Book, the classic by Mortimer Adler. I learned that efficient reading is a learnable skill, just like any other. Here is what I wish I had been taught when I started grad school.

First, a caveat. If you are in school, you need to learn to fully read individual technical papers, in detail and considering every aspect, before you can read them in volume.

Image by Kindel Media

---

I am not an academic. While I have a Ph.D. and used to work in a research lab, I am not paid to do research or to publish papers. Yet I still find it valuable to read papers such as conference proceedings and journals, as well as blog posts and white papers. Reading helps me to stay up to date in my field and to do my job better. Adding other people’s ideas to my experiences helps me to come up with better solutions to my technical problems.

I am paid to solve problems, not to read. Therefore, I must make the most of my reading time. I cannot read everything. Instead, I read papers through the lens of my problems.

 
Selecting Papers to Read

I read papers in three different situations, all in service of doing my job better:

1. I’m trying to solve a specific technical problem.
2. I’m keeping up on a general professional interest.
3. A colleague sends me something interesting.

When I try to solve a technical problem, it often involves a lot of very focused reading. I cast a wide net for papers, using keyword searches in Google Scholar, tracing references from other papers, and asking colleagues in person and beyond. I generate a large pile of papers that need to be thinned into a manageable collection.

For example, I collected and summarized papers on flaky tests for my colleagues. We run a very large number of correctness tests to ensure our software works properly and without bugs. Flaky tests—tests that sproadically fail—are a distraction. They slow development and they increase the risk of bugs slipping into our code base. Based on my reading, we reduced the pain of flaky tests.

Keeping up on general professional interests is a more continual background activity. I am an expert on performance testing of software. I follow academic literature on the subject, keep an eye out for relevant industrial papers, and look out for other resources, all so that I can remain an expert in the field. Important conferences are a rich source of academic papers in one place.

Finally, I enjoy new and interesting ideas. Colleagues forward things to me since they know I appreciate when they do so. These papers may be performance related, performance adjacent, or touch on other interests of mine such as peer groups, technical education, and DEI, among others.

I suspect that I could fill all my time reading the papers I accumulate and still not read all of them. I must be efficient if I want to get the most out of them.

How I Read Papers

When I have a set of papers to read I follow a specific process. I take the first paper and set a timer for 5 minutes. Within my five-minute limit, I proceed in the following order: I read the abstract, the conclusion, the introduction, and scan the figures in the paper. Doing so gives me the best opportunity to get a sense for each paper. I actively look for and record the questions that come to my mind as I scan the papers: questions to dig deeper on, questions about the parts that confuse me. “How do they do X?” “How do they prove Y?” “Could we use Z?”

At the end of the 5 minutes I review my notes. If I have no questions, I am done with this paper. If I have questions, I put the paper aside for a deeper read specifically to answer those questions. Then I pick up the next paper to scan. 

With this process I can scan 12 papers in an hour. That might not sound like much, but without my process in place, I could easily spend an hour or more fully understanding one paper.

A Deeper Read

After triaging the collection of papers, I have a smaller collection of papers, each with a list of questions to answer. I’ll pick up a promising paper. I review my notes and questions for it, then read the paper solely to answer those questions. The questions provide a focus, allowing me to read faster. If the introduction is not relevant to my question, I skip it. If I only have a question about the implementation, I go straight to the implementation section.

This all felt unnatural when I started doing it. I was not reading the paper as the author intended. That is okay. I (and you) do not owe it to the author to read their paper, nor to read it as it was “meant to be read.” Good readers and writers know this. We owe it to ourselves to get the most out of the papers we read. We only owe the authors our gratitude and an acknowledgement when we use their ideas. Sometimes I send a thank-you note to acknowledge ideas I find particularly helpful.

Remembering Papers

My wonder at my colleagues’ reading ability was not just because they read so much, but also because they remembered details and made use of those details. I must take notes if I want to do that.

I use two tools to take, organize, and remember my notes from technical papers: Zotero and Anki.

I use Zotero to organize all my academic papers. New papers go directly into a triage folder in Zotero. Zotero supports highlighting PDFs and extracting highlighted text. It also automatically collects the bibliographical data for each paper.

In Zotero, I use folders to organize papers by subject. When I want to find details on a given topic, I look through the appropriate folder and review my notes on the papers. Sometimes I rescan papers. Each part of this process is so much faster and more efficient than when I photocopied papers and stored them in a filing cabinet in graduate school.

While I can find things quickly in Zotero, simply using the tool does not put the information at the tip of my tongue. For ideas and details I want to talk about or otherwise have available in my mind, I create digital flashcards in Anki. See The Nerdiest Thing I Do for more on my Anki usage.

Reading to Learn to Read

I am able to read much more efficiently since that day sitting at my desk overwhelmed by papers. There are still people who can read and retain much more than I can, but my reading is no longer a weakness. It’s a strength.

Writing (And More) That Was Useful to Me in 2023

Sitting at my computer recently, I looked back across my notes for 2023. I was following my annual review process, reviewing the goals I had set for myself last January and the notes accumulated since then. I had a good 2023, accomplishing many of the things I had set out for myself. Those accomplishments spanned changes to my job (new role), my writing, and how I take and organize my notes. 

Several resources helped me achieve those goals in 2023. These are the books, posts, videos, and tools that helped me change my world for the better. Maybe some of them will be useful to you.

The Staff Engineer's Path by Tanya Reilly was one of the books particularly helpful to me in 2023. 

The Nerdiest Thing I Do

I stepped up to the end of a growing line of people at work. We were having an end-of-summer event and the line was for ice cream! To make the most of my wait, I introduced myself to the person in front of me. To my horror, he knew me and we had talked before. With prompting, I vaguely remembered our conversation. I wanted to crawl into a small cave and hide.

 
I have always been impressed with people with great memories: The people who remember others’ names, along with supporting details, after meeting them once. Those people are so dang thoughtful. To my continual disappointment, I cannot do that. I just can’t. Since I cannot remember those things naturally, I try to achieve the same effect through brute force, using flash cards.

Screen shot of my Anki decks before studying one morning. 

Should I Go to Grad School and Get a Ph.D.?

One of my favorite things to do is to formally and publicly address newly minted Ph.D.s with their title. “Congratulations Dr. Smith!” I’ve done it in person, on LinkedIn, and Twitter. Getting a Ph.D. is hard. It requires time, luck, and personal sacrifice. It makes me happy to acknowledge that person and their accomplishment.

I get to work with many extraordinary college students through my company's internship program. When I talk to them, some are deciding between going to graduate school or getting a job after graduation, but aren’t sure what to do. This post is my advice to those students.

Graduation day many years ago, holding my daughter after receiving my Ph.D., with my mother to the left. 

Hiking My Way To My Job

I got my current job because I hike. Rather, I wouldn’t have gotten my current job if I didn’t regularly hike.

 
Many people think that their success is solely due to their hard work. They planned, put in the effort, and achieved what they set out to do. They deserve credit for what they have done. However, I also think they benefited from a large helping of luck. More importantly, I think we do everyone a disservice when we claim complete control over and credit for our own success.

Me on a recent hiking group outing with the Hudson River behind me.

---

A Story of Success

The short version: I was a smart kid. I did well in school, and got into a good college and a better grad program. I earned a Ph.D., got a great job after graduating, and then an even better job several years later. Along the way, I published papers and patents and grew my network. I worked hard, charted my path, and succeeded. This is the story that teenage David would tell, of completing my destiny to have a great career in computing.

That story is short because it leaves out a lot of details. Let’s start with the easy omissions: I was born to two loving parents who raised me in a well-to-do neighborhood with good schools. I may have been an awkward child, but all the adults (and many of the children) expected and encouraged me to succeed. Unfortunately, many people do not grow up with that support. I was very lucky.

A Story of Chance

The long version includes a number of lucky breaks. I ended up at graduate school at the University of Illinois at Urbana-Champaign (UIUC) in part because my friend and classmate went there a year earlier. Seven years later I was hunting for a job as I finished my doctorate. The company and research lab I ended up with had a strong pipeline from UIUC. If I had gone to another school, I would have been less likely to end up where I did. Further, I was hired right before a hiring freeze. My hiring manager sped up my paperwork to make sure I got in. So, if I had been on the market a few months later, I would not have ended up with the same job. I loved working in that lab and got to know many amazing people, including IEEE and ACM Fellows, National Academy Members, and a Turing award winner.

Soon after joining the company, I joined a weekly hiking group at work. Many fellow hikers became friends. I know them solely because they also like hiking and worked at the same company.

A Second Chance

Moving forward several years, I had built a nice role for myself, in which I got to work on interesting projects and was well rewarded. However, my company was not growing and had gone through periodic layoffs. I didn’t worry about my job, but I did worry about how much I could accomplish in that environment. Then, across a year, several unrelated but important things happened.

Several members of my team had a side project that suddenly became successful. They and their project were moved to a new division. I was excited for my colleagues, but it also meant I needed to find a new team. Around the same time, my youngest child was finishing daycare and starting kindergarten. His daycare was located at the site of my office, and I had been dropping him off and picking him up every day. Finally, one of my hiking buddies had been laid off and started a new job at a startup I had never heard of before.

Because of the reorganization, I had to re-evaluate my job. Because of the health of the company, I was willing to look elsewhere. Because my son was done with daycare, I was no longer tied to my current office. Because of all these things, I decided to look far and wide for my new job.

One of the places I looked into was my friend’s startup. The one I only knew about because of him. The one he worked at because he had been laid off. I told him what I was looking for and described my skillset. It turned out that his company was looking for my exact skillset at that exact moment! I interviewed and was hired.

Let’s see all the small changes that could have happened and kept me from my current job. If my colleagues’ project hadn’t taken off, I’d have no need to look for a new job. If my son was younger, I’d still be tied to the daycare. If the company was healthier, I wouldn’t have looked outside it. If I didn’t hike or my friend wasn’t laid off, I’d be completely unaware of my current company. If all of this had happened a few months earlier or later, the company wouldn’t have needed me.

Hiking group poster from 2010. The man in the yellow shirt is the friend who introduced me to my current company.

Do the Random Parts Matter?

At this point you may object: David, there may have been a lot of chance involved, but things would have worked out well for you anyway – the particulars were random, but your being successful was not by chance.

You may be right. But there are two important points to consider. First, my life would have been very different in ways I couldn’t have planned if I hadn’t ended up where I did. I had an acceptable competing offer in hand – I would have ended up at a very large company, instead of a small one. All my experiences would have been very different and I strongly doubt I would have become as well known in my field as I am. I may have been successful, but it would have been a very different version of success. It would have looked like a very different plan.

Second, I have the privilege of having a lot of opportunities. Yes, I have worked hard to provide myself with opportunities, but as covered above, I have benefited from many factors outside my control. Many people do not have all those opportunities or never get the right one.

Increase Your Luck

Hard work and planning matter: they are necessary for success, but not sufficient. So, by all means, celebrate your victories. You earned them. But also acknowledge your luck and be grateful for it. I am.

Finally, consider what you can do to increase your luck. You can put yourself in good situations where things are likely to go your way. I did that by working hard in school. I did that by applying for many jobs so I could compare multiple offers. I also did it when I chose to do something I love – hiking – which happened to introduce me to many people who would become friends.