I like the “Aha !” moment that comes when I solve a mystery,
and performance issues provide a lot of those.

Pierre Laporte,
Senior Staff Software Engineer at Dremio

For context to our readers, Pierre Laporte worked several years as a Performance Engineer at Datastax. Henrik and Matt worked there with him on the Datastax performance team in 2020. Pierre now works at Dremio Corporation, caring about the performance of their Apache Iceberg based big data product.

Henrik:

Hi Pierre, It’s been good to work together again!
I remember when we worked together, that French labor legislation required me, as your manager, to write an annual review in writing. It was translated to French by an official translator, and everything.  So I wrote that you were the most senior performance engineer in the company, working with the largest customers we had, to troubleshoot challenging performance issues that were directly jeopardizing millions of dollars of revenue.

And with that “heroic efforts” introduction, would you like to start by introducing yourself a bit more. Please also tell us something not related to your long career as a performance expert.

Pierre:

Thanks for this opportunity, Henrik!
I have been working on JVM applications for the past 16 years.  First on feature development, then on performance tuning and troubleshooting.  I like the “Aha !” moment that comes when I solve a mystery, and performance issues provide a lot of those.  Aside from that, during the week-ends, I volunteer as a dog trainer in a local association.  It’s all about helping people understand and interact with their dogs better.

Henrik:

At least for me that situation we worked on back then, was the most challenging performance issue I have worked on in my life. Many team members have later told me that the focused effort we did, together, is their best memory of working at Datastax.

I’ve thought about it and maybe one reason is that often the role of a performance engineer is quite lonely. It’s specialized on this very challenging topic, with specialized tooling that nobody else knows how to use, and most of the time your job is to troubleshoot code written by the other engineers. And you do that after the fact, often when the problems are already in production.

How do you think about the role of a performance engineer both now and in the past? 

Pierre:

This is an interesting question.

The role is still the same in that there are far more software engineers than performance engineers.  So by the numbers, performance engineers will inevitably have to troubleshoot code written by others.

That being said, the one difference between now and then is that we have substantially increased the complexity of our technology stacks.  And I do not think a performance engineer can work all alone anymore.  The work involves discussions with subject matter experts on each major part of the stack.

Tell me about your first encounter with change point detection. Where and when was it and what did you think about it?

It was at DataStax.  I had created a server that stored the results of daily/weekly benchmarks, with dashboards of historical data to detect trends.  But this was a manual operation that needed to be performed on a regular basis.  And as one can imagine, each regression candidate could lead to a debate on whether it is real.  This was a time consuming activity.

Then Piotr Kołaczkowski, Guy Bolton King, Shaunak Das and other engineers created Hunter to automate that process based on the data we had.  The funny part of that story is that, at the time, I was busy troubleshooting customer issues and was not involved in internal benchmarks anymore.  I was not even aware that Hunter was being created.  I was very pleased with the result when I saw it.

Making it a systematic, stable process tremendously reduced the time needed to go from issue detection to resolution.

When we launched Nyrkiö last year, we noticed that the initial group of users we got were mostly engineers who had worked either at MongoDB or Datastax, and as they went on to new jobs, they really missed this technology and were happy to hear that they could now use Nyrkiö, to be able to effortlessly find performance regressions also in their new jobs.

Was it like that for you as well? How did you manage during your first year at Dremio? I understand you didn’t have any alternative to Hunter?

Part of my initial work at Dremio was to reduce benchmark results variance, as it was responsible for lots of false positives.  Then, we were in the same state as DataStax before Hunter: manual identification of issues and time consuming discussions to determine whether they were real.

I did not have any alternative to Hunter.  In fact, I had started building a service just like Nyrkiö.  Dremio would probably use this service today if Nyrkiö had not been released.  Fortunately, I got in touch with Matt to discuss my ideas.  That’s when I learned that you were working on a product.  From there, it was an easy call.

You were one of the first users of Nyrkiö Change Point Detection. In fact I believe we contacted you a bit before we announced Nyrkiö to the world. What was your first experience like?

Well, the adoption was tricky, and still is to some extent.  The product lacked features like the ability for all members of an organization to view the results of the benchmarks, Slack notifications for organization change points, etc…

But the core features worked like a charm.  As soon as we sent some pre-existing performance results to Nyrkiö, a regression was flagged.  We had not noticed it.  It triggered an investigation on our end and indeed, it was a real regression.  So our first contact with Nyrkiö immediately delivered value.

Yeah that was great validation for us as well! And probably helped you to stick through some of the initial UX issues. Did we at least get better over time? What is still particularly painful?

Yes definitely.  Last week I demo’ed Nyrkiö to a colleague.  He was very interested in using it.  He opened his laptop, and boom, done, he had access.  We do not have that ease of first use on many applications, to be honest.

What is still particularly painful?

I think detection parameters should be specific to series, and not to tests.  Like, if a metric is highly volatile, we may want to set a % higher threshold to reduce false positives.  But then that can cause false negatives on other metrics.  Overall, finding the right parameters is hard, as it could result in overfitting.

Maybe not particularly but still painful.  Related to the previous point: no enterprise level SAML ability.  Typically, what we would do is add Nyrk.io in Okta and grant every engineer access to it.

Yeah ok I agree with the SAML. It’s on our roadmap for the Enterprise tier for sure. For our first year we took a very focused “Github first” -approach.

(To be continued…)