Behind Zyro’s easy-to-use frontend is a complex infrastructure that needs to run reliably every single day.
One of the most important people making sure all the parts of the Zyro machine keep turning is our Site Reliability Engineer Paulius.
We caught up with Paulius to learn about how he keeps Zyro safe, what it’s like working in our Backbone team, and how having more freedom in your job comes with its risks.
Could you describe your role at Zyro?
I think my job title is something like SRE, Site Reliability Engineer, but titles don’t matter much at Zyro unless you’re the CEO.
Basically what I’m doing is ensuring that Zyro runs fast and smoothly, and that the backend is as stable and secure as we can make it.
There are two main parts of this job. One side is operational stuff, where I need to take care of day-to-day incidents, like if a builder’s broken and something is not working or I’m just making general updates.
The other part involves the infrastructure. This involves bigger projects like moving or migrating systems to GCP, or finding new ways to carry out tasks more effectively or faster.
I used to be a team of one person, but I joined the Backend team and we became the Backbone team. This works well because I interact most with the Backend people. I now know what’s happening more with the Backend team and I’m bringing together things from both Hostinger and Zyro engineering
What kind of challenges do you face in your role and how do you address them?
So a challenge would be something like a couple of weeks ago when we experienced quite a big DDOS attack. We realized we needed to deal with it in an efficient way.
The challenge became how to prevent attacks like that so it would be both effective and financially feasible. We could have bought a commercial service, but for Zyro it would be pretty expensive because it would cost more per user than users pay us. There are other technical concerns too.
So I designed a solution which we are testing and which will be live next week.
That would be one of the questions in lots of what I do: how can I design the infrastructure in a way that would be both cheap and reliable enough for our users?
Which tools and languages do you need to be familiar with to do your job?
Linux is standard for the people who are doing this work. So basically the tools are standard UNIX tools and we are also using some configuration management tools.
Most of the virtual machines are managed by a tool called Chef, which uses its own language based on Ruby. Also, there is a tool called Ansible, which usually runs ad hoc tasks.
Our infrastructure, which is created on GCP, is defined in another language called Terraform HCL or just basically everyone says terraform.
I think someone would need to have strong Linux or Unix knowledge, at least some experience with networking, with HTTP protocols, also experience with automation. And of course, someone who isn’t afraid to dig deeper and to learn how things work down below and then up to the users, like what users see.
Otherwise, of course, it helps if you have some knowledge about Node.js, which is, for example, what our backend and frontend is being developed with.
What was it like working remotely at Zyro during Covid?
So it was really interesting because I started about a year ago, I think it was in the middle of April.
It was quarantine and everybody was wondering what it would be like. I was doing all the interviews remotely and it was new to me because it was new to everyone. I basically just went to the Kaunas office and just picked up my laptop and that was it.
I started working remotely and when the quarantine relaxed a bit, I just decided that it would be really nice to see people face to face. I started coming every Thursday to the Zyro office physically because I’m living in Vilnius, (about 100 kilometers away from the Kaunas office).
We also got a small office in Vilnius, but most of the engineering team, and especially the Hostinger engineering team which I’m usually communicating with, are in Kaunas.
It still feels like you’re working as a team, at least with people with whom I’m communicating a lot, like the Backbone team and part of Hostinger SRE as a team. This is because the Zyro and Hostinger infrastructures are actually pretty tightly coupled.
I need to communicate with the Hostinger engineering and know in what direction they are moving because we are still part of it. We basically share many parts of the infrastructure with them.
Do you think Zyro is encouraging you still to learn and develop in your role?
When I gave that example about the DDOS issue, when there’s no official solution, we’ve got to find one ourselves. We need to explore and we need to experiment.
When you do that, you are developing yourself.
There was a plan we had to take learning days, to take the whole day for the learning and experimenting, but because I’m responsible for some things which are kind of block for other people, it’s not always working like that.
Instead of that, I usually try to take two half-days. So I first do all the work that needs doing and then study something or just read some documentation.
Do you feel your work has value at Zyro?
There are small things rolling out, not every day, but at least once a week or so. My position is that I’m doing lots of small things that enable other people to do other, bigger things, save some time, or do something more efficiently.
For example, I’m building a pipeline that would enable all of the developers to have their own small version of Zyro to experiment or test independently.
Currently, we have production and staging that are our two main environments. So if there are more than a couple of people working on a couple of features in staging, they might clash. That might introduce some bugs or unwanted behavior when we deploy your code just for production.
We would like to have it so that when people create a feature, they will just have their own small version of Zyro built somewhere in the cloud.
Which of the Zyro values do you think are most important for what you do?
I think the most important value to me is freedom and responsibility, especially when I’m either with the Backbone team or there are decisions that I’m making by myself.
There’s no authority saying, “you need to do this or that,” so I’m free to choose whatever solutions I think are fit for purpose. I’m also responsible for making it work, finding solutions that can be operationalized – easy to operate, easy to monitor, and easy to run at scale.
I have a much wider range of responsibilities with much greater freedom than many developers. If you’re working within existing code, yes, you have freedom, but not as much. My range is wider, but also much riskier.
Is there anything you’d say to someone applying to Zyro?
One of the unique things at Zyro is that, even as quite a small company, we have a big brother Hostinger at our side. As we’re sharing the structure with them, we are also sharing common knowledge, tips, and tricks.
We share infrastructure code and other things with Hostinger which operates worldwide and on a much bigger scale than Zyro so far.
If you want to talk with someone and say, “Oh, I got a good idea. Will it work in the scheme? Will it work in our setup?” Instead of thinking for a day or so, you just go and speak to someone from engineering. You will probably find an expert opinion on most technologies for sure.
Just go to the team Slack and ask. Especially in engineering or infrastructure channels, there are quite interesting discussions on quite deep topics.