The new internet lacks more use of Websockets and WebRTC on top of stateless HTTP. There, I said it. And yes, that was quite technical. But feel free to keep reading, because I'll first explain it in simple terms before concluding technically with a solution that I hope programmers, businesses, and politicians might pick up on.
Where does a website actually come from?
Let's start at the beginning. But if you already know everything about the internet, feel free to continue from the heading "Are there solutions available?"
Anyway. The internet is everywhere, from refrigerators to Smart-TVs, and from tablets to watches. Innovation in this area is abundant. However, some technologies are still not fully embraced, and that hinders what it could be, or perhaps should be.
What few people know is that every page you view comes from a so-called server, and how that actually works.
The web is stateless: it doesn't remember anything by default
Using the internet can be as simple as visiting a news website or online store. You see a page with information. It comes from a server, simply put, a computer that runs 24/7. However, a server doesn't look like your laptop or desktop computer at home. For example, a server doesn't have a mouse, keyboard, or screen like a typical desktop computer. The computer case of a server is a rack, just like your cutlery drawer in the kitchen.
We often express the height of such a rack in "U", which stands for "Unit". A common height for a server rack is "1U". That's equivalent to 44.45 millimeters. Anyway. Many of these servers are needed for all those websites in the world. That's why racks, which can sometimes accommodate up to 48U, were invented. That's 48 servers in 1 rack. And many of these racks are then placed in a suite. And many of these suites are then found in a data center. A data center is a building specifically for servers. Thousands of 1U servers can be housed in such a building in this way.
Many of these servers are so-called webservers. A web server is a server specialized in delivering the "web" as we know it. Websites. When you type a web address into your computer or phone, your device eventually reaches one of these webservers somewhere in a data center through clever routes (DNS). And that server formulates a response that comes all the way back to you.
In the case of webservers, that response generally comes in the form of text. Often a combination of three computer languages. Namely; HTML, CSS, and Javascript. Your internet application, also known as browser, knows exactly what to do with that response. It can understand and transform each of those three languages into the webpage you ultimately see on your device.
The importance of sessions for webservers
A web server is a simple machine. When you request a webpage, it provides the answer. Your device translates that code text into a nice page that you understand and know.
If you then click on a link on that page, you'll end up on a new page. The whole process starts over again from scratch. The new address appears in your browser's bar, your browser follows routes through cables, and you eventually end up back at a server that provides an answer.
The server knows nothing about you or your device. It doesn't even know that you were just on the homepage and clicked on a link there. Your request for the second page is for the web server as if any other random person just happened to ask for that page.
Minimization of unnecessary data transfer
And then came sessions. Sessions have had a huge impact on the internet. Sessions are technically difficult and challenging to do safely as a developer, but the operation is very simple. Sessions can exist because cookies exist. You've probably heard of those before. Cookies are small text files stored on your device.
Webservers that use sessions instruct your device when opening a page to also place a cookie if it didn't already exist on your device. That cookie is a text file and contains, for example, a unique code. A so-called session code. And your device will send this cookie with code back to the web server every time you visit the same website.
Because the web server now not only receives the request for a page but also a session code stored as a cookie on your device, it can recognize you. It now knows that you were there before. After all, you've already had a session code.
Through session codes in cookies, you can log in. Because if you log in on the homepage and then go to another page, your browser will send the session code to the server. The server remembers that you were logged in and will also show on the new page that you are still logged in.
Another example is the shopping cart. If you add something to your shopping cart, the server remembers that you did this. And if you visit another page and your session code goes back to the server, the server knows that it should show the same items in the shopping cart on the new page.
Heading towards fully real-time communication
But besides your shopping cart and your account page, the rest of the page is still the same for everyone. It's the same code response that your browser transforms into the page you see. And there's a lot of waste there.
Imagine an online store has 10 products. And you visit that page. The code for all 10 products comes from the web server to your computer.
If the owner of the online store adds an 11th product at that moment, nothing happens on your device. When you requested the page, there were only 10 products. The page on your screen is therefore a snapshot of when you opened it. A kind of photo from a few minutes ago.
Now, if you refresh the page, the completely new code from the web server comes to your device. And because there are now 11 products, you'll see 11. And all the code for all 11 came to you during the refresh.
What is much more efficient, of course, is to only get the 11th product. After all, your device already knew about the previous 10. So that code wasn't needed anymore.
A deeper understanding and use of real-time capabilities
But what if the owner doesn't add an 11th product, but for example changes the price of the 10th product? In today's internet, you only see that price change when you refresh the page. And you get the code from the web server for all 10 products, even though only the price of the last product is different.
Again, waste. Because your device already knew all the information about those products. It just didn't know that the price of the 10th product had changed.
Conceptually, it makes more sense to exchange only the changed data between your computer and the web server. Whether it's an 11th product or just the new price of the 10th product.
Although a web technique called "XMLHttpRequest" or sometimes abbreviated to "XHR" helps with this, it's not perfect yet. With XHR, a part of the page can be updated while the rest of the code remains the same and therefore doesn't need to be sent over the network again.
But with XHR, your device has to ask the server for a piece of new information. It's one-way traffic. And this often happens based on a countdown clock on your device. For example, this clock starts counting down from 10 to 0 every 10 seconds. And every time it reaches 0, it asks the server for new information. Not the entire page, but a part. And then the countdown starts again from 10. This technique is called "long polling".
Are solutions already available?
Long polling is not efficient. For example, it always happens every 10 seconds, whether something has changed or not. And although it only requests a part of the information from the web server, it can spend hours asking the server for the most up-to-date information every 10 seconds when nothing has changed. This is especially annoying for mobile devices, which unnecessarily drain the battery because of this.
For this reason, there has been a new concept for a long time. Instead of your device always asking the server for an answer, whether it's when opening the page or while you're on that page with long-polling, it's much smarter if the server just lets your device know when something has changed. And until then, your device doesn't need to do or ask anything from the web server.
Two techniques that help with this are Websockets and WebRTC. The first establishes a connection between your device and the web server that works in two directions, bidirectional, unlike the one-way traffic before. And with WebRTC, a bidirectional connection can even be made between you and a completely different device, without the web server in between. This latter is especially useful for chat, voice, or video between two internet users. Directly. So the server is no longer unnecessarily used as an intermediary.
The crucial final push is missing
With bidirectional traffic, information can arrive on your device as soon as something has changed. And only the data that has changed. This is efficient and saves not only unnecessary traffic but also energy on both sides.
That sounds like it's solved, right? Not quite. Unfortunately. Although XHR is now well supported and widely used, Websockets and WebRTC are not. There are various reasons for this, both technical, bureaucratic, and commercial.
Most of the world's webservers use PHP. And PHP does not offer websockets as standard. PHP can be extended with this, but this mainly happens in internal test setups where you as a developer have all the say. In practice, hosting companies hardly offer any PHP extension for websockets.
Isn't NodeJS a solution?
Then there is NodeJS hosting, an alternative to PHP hosting. It can even be used together, with NodeJS acting as a reverse proxy for PHP, while NodeJS also handles Websocket communication. But this is rarely offered and can sometimes be very expensive compared to PHP hosting. Additionally, popular website systems are often not made for NodeJS. Using a second server with NodeJS alongside also means unnecessary costs and undermines the efficiency because this NodeJS server now has to communicate with the PHP server "through the back door". Often on a request or long-polling basis. This undermines exactly the advantage of the whole concept.
Aside from the technology, it's also important for developers to think differently. Whereas the web has been working and thinking in the current stateless way for decades, this way of working and thinking is completely different. Many programmers will even immediately say, "Websockets? Isn't that for chat or something?" While the bidirectional method that allows the web server to "push" goes much further than that. In fact, it can drastically reduce communication via networks worldwide and thereby also the energy lost with it. It can literally save dizzying amounts of money worldwide, a multiple of billions in the global IT industry. But even just the energy saving alone can have an unprecedented impact on the world and the future of IT.
Bidirectional with push is the future
A bidirectional internet, where every web server can push everything to the client, is like switching from coal locomotives to green hydrogen trains when it comes to energy. And like switching from medieval bloodletting to modern neurosurgery when it comes to technology. It's actually quite crazy that it hasn't been embraced better yet.