Launch of Wegstatus 3.0

In the past year, major changes have been made to the Wegstatus platform, most of which are not visible or noticed by users.

In this blog, we will give a glimpse into the what and why of these developments.

The Wegstatus platform processes multiple data streams that are offered in different ways. A distinction is made between:

• Static data

• Current data

For static information, think of information that describes the Dutch road network. Current data contains information about the current situation on the road, such as the location of an accident, a lane closure or a bridge opening. With this type of data, it is important that it is processed quickly, efficiently and reliably. If this does not happen, the data may no longer be relevant when it reaches the users.

In Road Status 2.0, there are a number of challenges with regard to data processing. Periodically, a pull is performed, say every five minutes, at a data source to ask for the situation at that moment. 

The following questions can be asked here:

• Are events missed that fall between two pull moments?

• In the worst case, a current situation will not be processed until five minutes later. And is it still relevant then?

• How many situations are processed twice?

For example, when a bridge opens and closes between two pull moments, this can have happened unnoticed without registration of Road Status.

When a pull is done periodically to get the current situation, it can happen that situations are processed that were also part of the current dataset in the previous pull. Each time, the entire dataset is processed to determine whether the situation is already known or new.

This can be done faster and more efficiently. The digitalization of the world ensures that more and more data is becoming available and has to be processed, the platform should actually grow with this. The existing structure of Road Status 2.0 makes growth and further development a difficult and risky process.

That is why we started thinking about a new infrastructure in which scalability and efficiency are central.

Event-driven and scalability

It was previously mentioned that part of the current situation that is pulled is already known, processing this is a waste of time and computing power.

When the data source can send the situation at the moment that it has changed, then the following always applies:

• The situation is new and current. This makes every processing useful.

• The situations are processed without delay in the Road Status platform.

One consequence is that there are more peaks and troughs in data processing. It is expected that more situations will have to be processed during the morning rush hour than at night.

Growing, or scaling, of the application means that the system can handle changing demand. One cause is that more data can be processed, but also that the number of users or services offered is growing.

One option is to add faster CPUs and more memory to the system on which the Wegstatus platform is running, the so-called vertical scaling. This cannot be done infinitely and is expensive, because you would have to buy new hardware again and again.

Adjustments to the structure of the software and the infrastructure can also contribute to this. When such a dataset with current data is processed, this can be done 1-by-1, but this could also be done with several at the same time.

This is horizontal scaling. In this case, applications or processes of the Road Status platform are executed multiple times. In concrete terms, this allows more situations on the road to be processed in the same time, which means that notifications reach the user faster.

Figure 2: Parallel processing

Google Cloud

The Wegstatus platform consists of multiple applications and if multiple copies of these are also operational, this has consequences for maintenance and complexity. In order to be able to implement these adjustments, the services of Google Cloud were chosen. 

A configuration of standard services and customization was created here. Standard services are easy to use and maintenance is the responsibility of Google Cloud.

In order to keep the various applications of the Wegstatus platform manageable, Google Kubernetes Engine was chosen. This is a tool that (with the correct customization settings) ensures that the applications are operational with the correct number of copies in order to be able to quickly process the current situations at that moment.

With this adjustment, the integration between development and operation has become stronger. Step by step, work is being done on a method in which continuous adjustments in the software can be brought to the users.

Continuation

Many functions have been rewritten and improved, so that the task can be performed more efficiently or to improve the adaptability of the component. All of this can have an impact on the functionalities and requires careful checking whether the right thing is still being done after the adjustment.

In the coming period, work will be done on further migration of functions from Wegstatus 2.0 to 3.0. The existing functions are faster and more efficient, and larger data flows can be processed. This new architecture ensures that the Wegstatus platform is ready for the future and makes it possible to add new functions more easily.