Written by Aukse Zirgule, Zyro‘s Head of QA
Imagine this: your team has spent months polishing a new feature and the big release day is finally here.
Yet, to your disappointment, you notice that your clients use this feature in a completely different way than you’ve expected. They might even hate it and not want to use it at all.
This has happened because you’ve tried to guess your global customers’ behavior based on the best practices on the internet.
Why not let users dictate your functionality instead? It’s important to iterate often and let the users decide what features they need and in which ways they work better.
In this article, I’m going to share why and how we test in production here at Zyro to not just reach our business goals but to make our customers happy.
We don’t follow the traditional QA ways
I was the first QA specialist that joined Zyro, full of enthusiasm and eager to ‘do QA right’ – every new feature had to be fully tested before it went live, releases needed to be planned…
To my surprise, I quickly found out that deploys happened very often and at completely random times at Zyro. New features went live – just like that.
So, I pushed my traditional QA thinking aside and embraced the challenge.
For a startup, it just makes total sense to go the MVP route as opposed to the traditional ‘polished feature’ route that requires a lot of effort and time. That’s something that a startup doesn’t have.
Our product is new, so it is very important to move fast and to know whether we’re going in the right direction.
And what is the right direction, you ask? The answer is simple: the happiness of our customers, as well as their willingness to pay for our product.
Keep in mind that testing in production does not mean that untested code reaches our users. Of course, you need to have a rock-solid infrastructure (more on that below) and guidelines on how exactly you will be testing in production.
Testing in production has a very important advantage – early feedback from customers. It lets us move fast and make improvements to our product quickly.
As you might know, analyzing user experience is important if you want to make your customers happy.
Another important thing is that we can test actual data in order to see the real situation when it comes to our product, and not mock data, which can yield totally different results that are often not reflective of reality.
Plus, we can monitor system performance in real-time scenarios as well.
Here’s how we test in production
It all sounds good in theory, but how do we actually test in production?
First, we have a strong customer success team whose main goal is not to just support the customers, but to help them make their business successful.
Our customer success team lists the most popular feature requests every week. Then, Our UX designers discuss and create new feature flows. QA participates in feature development meetings, identifies risks, and predefines possible test cases.
Second, we have planning meetings, during which we put those features in that week’s sprint and assign a DRI (directly responsible individual) to each of them.
The DRI can be anyone from the team, but the important part is that this person is fully responsible for the feature and its status.
Third, when the feature coding is done, the pull request (PR) is merged and automatically deployed to the staging environment, the DRI reports it to our Review channel. That way, everyone on the team (not just the QA specialist) has the chance to try it out and give feedback.
Finally, if there are no critical bugs, the DRI deploys the feature to live, while QA registers bugs and improvements on GitHub. That’s when testing in production really starts – the QA team members can fully test the feature at this point.
Our end-users get the possibility to test it at the same time and UX designers are able to investigate the user behavior at the same time.
Prepare a solid infrastructure
Sounds risky? It doesn’t need to be.
Sure, in an ideal world deployment would be a zero-risk activity. But ideal conditions are hard to come by.
So, instead of being afraid to deploy to production at any time we want, we have set up a solid infrastructure to support us when we test in production.
Coding workflow. To have safe deploys, we need to weigh up how we code new features and solve bugs. Now, we use a 2 PR approach. It lets us deploy new features and bug fixes to the production environment quickly and at any time. We do not have release days and anyone can deploy new features and bug fixes. This is a perfect example of a very important Zyro value – freedom and responsibility.
Unit tests. The coverage that these types of tests provide is very important. Unit tests are powerful – enough said.
E2E tests. These tests are important but, by creating them, you must know the main user flows to avoid repeatable steps and cut the unnecessary time for a test run. So, you could say that we are guided by the principle that less is more. Do not ignore the importance of reports when implementing E2E tests too. You should be able to see exactly what is failing, as soon as possible.
Performance Monitoring. Performance and speed matter a lot. The tools we use are New Relic, Sentry, Graylog, Kibana, and Uptime Robot. Sometimes we test the main critical flows with JMeter.
User testing sessions. These help us see where we need to make changes and improve our features. We analyze customer behavior and observe how they act via Hotjar. We also do A/B testing.
Customer success team. Customers love help and instant feedback. We couldn’t deploy to the production environment if we didn’t have a customer success team to help our clients.
Quick rollback. Everyone makes mistakes. Even if we conduct a lot of tests, features can fail for reasons that do not depend on us. Failing allows us to improve. The most important part is to know how to recover fast.
Some challenges that we’re facing
Of course, there are pros and cons to everything and testing in production is no different.
Despite mostly being successful, here are some of the challenges that we’re currently facing.
Continuous improvement. When our team was smaller, the challenges were fewer. We were not afraid to click the ‘big red button’ and see all of our new features go live. Since our team has grown fivefold, we’re continuously improving our approaches to QA.
Survive in a very dynamic environment. Do not fail to try. One of our values is a bias toward action and it’s more important to us than never failing or making mistakes. We take action instead of having long discussions about things. Our quick actions help us keep up with competitors and challenges us not to get lost with so many new things going on at the same time.
Bugs in a live environment. No QA methodology is perfect. Our goal is to solve bugs as soon as possible while giving immediate support to our users.
Our product functionality is growing and developing incredibly fast, as is our team. We all know that there will be a lot of new challenges ahead, so we’re already trying to improve our current workflows.
Yet, the most important thing here is to continuously give a lot of attention to our customers and hear what they have to say.
Before I joined Zyro, I had a traditional QA mindset. Every new feature had to be tested in dev/staging environments before going live, automated tests coverage had to be created, the releases had to be planned… The list goes on.
But then I realized that if you want to move fast and make your clients happy with new features every week, you need to find new ways of doing things.
Obviously, not every team can test in production; it all depends on the product or project you’re working on if this QA methodology would work for you.
But don’t forget – the same testing in production strategy (e.g. the one that we use) does not fit all projects either. You need to find the best way to improve the quality of your own product.
And finally, never get complacent. Don’t get used to how things are; push yourself to improve continuously.
I, personally, am thankful to Zyro for a very important lesson I’ve learned: love your customers and give them the chance to choose what is good and what should be improved.
Join the conversation
Your email address will not be published. All fields are required.