Follow Our Hike to Become a Performance Assertive QA Team

Zyro‘s QA Lead Asta Dagienė explains the importance of assertive performance testing in QA.

Performance testing is like hiking to a high peak hidden in the clouds. You’re never quite sure where your path will lead you, whether you’ll go up or down, or even if you’re on the right track at all. You must be focused and determined to reach your goal.

This article is designed for performance testing enthusiasts and those interested in exploring the world of assertive testing. In it, I offer insights into my own team’s performance testing map, the process of creating it, and some details from the journey. 

So, what should you take into consideration before starting your journey towards assertive performance testing?

1. Destination. First and foremost, make sure you know what your goal is. 
2. Weather forecast. Double check the factors that will affect your decisions about the tools you use. 
3. Backpack. Make sure you have access to the tools you need to achieve your goal.
4. Waypoints. Plan your route with smaller goals to be agile, adaptable, and able to celebrate small wins while resting.
5. Force majeure. Remember the purpose that drives you to the next point, and always follow the changing context.
6. Stress management. Do not be too strict with your plan; you know your goal, and it’s ok if you reach it another way than you planned.
7. Team. Do not travel alone. A QA is only meant to find issues, not fix them. Share your insights with developers and ask them to join the challenge. Be cross-functional.

Our path to performance testing 

The idea that our team needs to focus on performance more has been around since the day I joined Zyro. The necessity became acute during preparation for our Black Friday promotion. 

Our marketing ambitions for Black Friday were high, so the QA team understood that it was better to be prepared (see item ‘Destination’ in the checklist). 

Since we had done almost nothing related to performance testing, we started with user data digging (see item ‘Weather forecast’ in the checklist). We asked ourselves: 

• How many active user sessions do we have at any given time?
• How much could this increase during a big sale?
• What variations of user journeys are the most crucial and should be tested?

After getting familiar with the initial context, we started choosing tools for scripting the test plan, result storage, and result visualization for analysis (see item ‘Backpack’ in the checklist). Our kit at this stage included tools necessary for backend performance testing:

• JMeter. An open-source tool designed to load test functional behavior and measure performance. 

• InfluxDB. Time series-based data storage that allows you to see changes in performance metrics quite easily. Recommended in pair with Grafana due to simple setup and prepared customizable dashboards easily found on the internet.

• Grafana. A tool that lets you quickly build custom dashboards or importing prepared templates to monitor your synthetic test data right away: no excel sheet manipulation, no human errors. Charts are available during the test run or after it’s finished.

Our initial JMeterr test plan launch and execution was the first significant milestone (see item ‘Waypoints’  in the checklist). 

We clearly understood that measurements would be improved by smooth cooperation in a cross-functional team (see item ‘Team’ in the checklist). But did we have smooth cooperation in a cross-functional team? Of course not. 

The next step, which developers should have taken, was ‘making improvements’, but this had not been properly planned. From this, we realized it was time to initiate a performance focus group including cross-functional members: developers, QA, DevOps, and marketing. 

We also discussed what other small goals ‘waypoints’ we would need to add to our map in order to achieve our goal of making Zyro the fastest builder. The plan we formulated included:

Remembering the importance of UI performance (page load) testing

We had very important user-centric metrics to track and improve here. 

Initially, we had the idea of implementing our in-house tool with a Google lighthouse. After the POC we decided to kill this idea and chose a paid tool, SpeedCurve (‘Stress management’ case from the checklist above).  

SpeedCurve serves us for regression purposes, has a nice deep-dive into metrics, and offers a handy waterfall, where devs usually find useful insights. 

However, over time we found it didn’t fully serve our needs, so we decided to automate the most popular public speed tracking tools (Pingdom, GTMetrics, and Google Pagespeed Insights). Now we use a combination of paid and our public tools automation.

Knowing how fast your competitors are

This is Force majeure factor for our checklist and is a must if you want to know if you are the fastest. So, are we?

After speed focus group dedication, our metrics showed a lot of improvement. The verdict is that in some areas, yes we are the fastest, but in others, we still need to improve. In all cases we are in the top 2-3. 

Checking your vitals and trends

Look from different perspectives.

Your lighthouse score or response time is not the only way to evaluate your performance status. Be curious and learn from different sources (check out this guide on web vitals). Build a dashboard. 

Which data should you track: synthetic or real user experience? This is the main question, and of course, you should read and follow your context, because your project might not even be live. But in general full orchestration gives you better visibility and wider understanding.

Creating assertion channels

Do not make performance data your little secret! You might be surprised what impact awareness can bring. 

We have a dedicated channel and constantly share posts with our team insights on performance data changes. I was surprised how engaged the rest of the team got once they could measure the impact of the change. 

Our company’s ‘bias towards action’ value rocks!

Visiting your performance data dashboards should become a daily habit while drinking your morning coffee. One day, you’ll log on, check our dashboard, and see evidence that you’ve climbed to the peak you were aiming for. 

The bad news is, though, this isn’t your final destination. The main challenge here is not only reaching the peak, but adopting the performance testing lifestyle! If you tick all the boxes on our checklist, you are ready to succeed. 

I wish you an incredible hike with breathtaking views from your peak.