How we Built an AI Website Importer at Zyro

Picture this: you want to build a website for your business but like most people out there, you have little-to-no coding experience.

What’s more, you don’t have a heap of spare cash laying around somewhere that you could use to hire someone to build your website for you.

So, what’s your course of action?

A website builder, most likely. Tools like website builders make seemingly impossible tasks simple.

But, not all website builders are created equally. After a while, you might notice that the solution you have found doesn’t match up to the needs of your website as you are trying to develop and grow your business or brand.

And just like that, you’re back at square one: without the experience to make the changes yourself or the money to hire a pro.

What you really need is a new platform, one that has been designed from one with customer obsession in mind and one with all the space you need to grow (we’re talking about Zyro, of course).

That’s exactly why we built Website Importer, a tool that makes switching website builder platforms effortless.

All you would have to do is send us the link and we will port it over to our own platform. Simple, right? 

Now, let us tell you exactly how we developed this tool.

Let’s talk about the web

The web is massive. Well, not literally massive, because it doesn’t weigh a thing, but it is really huge. 

It’s huge in more ways than one but for everyone’s sake we’re going to be talking about one thing: variety. 

Technologies evolve and change at such a rapid pace that cutting-edge innovations become yesterday’s news in the blink of an eye.

The web is constantly shifting shape and this makes the web beautiful, chaotic, and mind-boggling all at the same time.

Now that you have the context, it should be obvious that there are just as many ways to build a website as there are website builders out there.

So, when we are porting your website to our platform, how do we know which bits fit where, and why?

The answer is all about separating the useful information from the white noise.

Extracting Information

One of the main challenges in creating a universal website importer is extracting information. 

Like we said before: the web is just too variable. 

At Zyro, even in the early days, we understood that. So, we said to ourselves: okay, if we can manage to import 80% of all websites reliably, then we’ve done a good job (go easy on us, we’re just getting started).

When it comes to information extraction, we always start with the essentials – the images and texts.

We intend to build on this list later by extracting maps, forms, and video components. It’s not that hard to do. In fact, there are many tools dedicated to website scraping that we could have used to grab a quick win.

But at Zyro, we don’t usually settle for “second best.” After all, we wanted to really capture what any website looks like, no matter how it was created.

That’s why we couldn’t rely on simple scrapers or ready-made libraries. We needed more control.

What next?

We set out to find a headless web browser that we could run on a remote machine that could execute JavaScript code on the original website, take screenshots, and work out what any given website actually looks like to a machine.

English, please. 

Okay, in simple terms, we were working with Python as our programming language of choice. 

Naturally, we wanted something that was written in Python and a solution appeared in the form of Pyppeteer (a Python port of the famous JavaScript headless browser Puppeteer.) 

It had everything we needed and, crucially, it had a really simple interface which is perfect for our use case. 

Since we’re trying to capture both the looks and the content of any given website, we had to write a bunch of JavaScript functions that would return the styles (position, sizes, colors, fonts). Once we had written some custom functions to capture full-page screenshots, we were done with our first step.

So, here’s what we had:

1. A headless browser (Pyppeteer) that allows us to easily write JavaScript functions and capture screenshots.
2. A headless browser, which when paired with custom functions, can extract the styles and content of text, images, and other elements.
3. A remote service that controls all of the above.

Our scheme

Now that you have a general overview of the basic challenges behind building a website importer and how to solve them, let’s take a second to look at how we organized our code for simple maintenance and updates.

Here’s how it works:

1. The end user clicks the “Import a website button” and enters their website URL.
2. The request goes through the frontend and the backend of Zyro. It is validated and our Google Cloud Run instance receives it.
3. Zyro’s Website Importer validates the link again.  The link is sent to the remote browser (which is also a Cloud Run instance) with a bunch of predefined JavaScript functions to execute.
4. The remote browser executes the functions, takes a full page screenshot, extracts further links and these results are then passed back to the website importer.
5. The Website Importer has to decide what to do next. It can either validate the links and send them to the remote browser or move on with further calculations (for now we’ve set a hard limit of a few pages so this defaults to the “move on” phase.)
6. The website importer calculates the extracted information and is presented beautifully on our Zyro grid.

Website Importer endpoint

So far, we’ve mostly been talking about the remote browser. 

However, a lot of things also happen at the Website Importer endpoint and that’s not only the decision whether or not to pass links to the remote browser. 

Like we said, it also makes some calculations. But what are those calculations?

Well, all Zyro websites are made of full-width sections and we use those sections to place elements of all different kinds like texts, images, videos.

This is what the Website Importer endpoint does: it separates the imported website into full width sections. 

Remember, from a code perspective, a section is just a list of elements. 

That’s why the screenshot of a website comes in handy. It allows us to determine what a section actually is by combining the screenshot with the style information and later, by identifying different background colors. 

With this, we can determine which elements belong to which section and create a sorted list.

So, that’s part one of the solution. Now, part two is all about how to place those sections and elements on the Zyro grid itself.

Presenting Information in Zyro Builder

The main problem is translating the sizes and positions of elements from pixels into grid value. We’ve already covered exactly how it is done and how we make imported website information look so good in another post.

Looking ahead, there are still a lot of challenges facing the AI team and our Website Importer.

In fact, as we’re writing this, the Website Importer isn’t even available yet but our problems are getting smaller, we’re closing the gaps and we’re on the verge of releasing the beta version of this feature for new and existing customers. 

The way we see it, the future of the Website Importer is actually not to import a website as it is.

Remember when we were talking about how quickly new developments get old? Well, with the rise of optimization techniques like machine learning, we hope to offer our customers a ‘bigger and better’ experience that can ride the ever-changing waves of the web.

Zyro’s Website Importer is designed to go further. It is designed to be capable of automatically enhancing your website as we port it over to our platform.

Why?

Because Zyro isn’t just about making sure something looks good. It’s about changing the game.