SURF Orchestration ♫ is music to our ears

A conductor of a symphonic orchestra always has a number of fixed instruments in his orchestra: the wind section, strings, the bass section and percussion. The SURF orchestrator also has a number of fixed systems that are part of the delivery of a SURFnet8 or NetherLight network service: CRM for customer-related data, IMS for the technical registration of services, JIRA for project tracking, IPAM for IP address management and NSO for the activation of the network services. Since the start of this project, thousands of workflow symphonies have already been performed, and these workflows have resulted in the delivery, adjustment or removal of network services. And the number of workflows is continuously increasing.

A solid foundation – a fixed music sheet

Services can only be delivered, adjusted and removed easily, consistently and quickly if a number of basic elements are well balanced.  The automation project at SURF has paid a lot of attention to this from day one. It has examined the entire data architecture in order to ensure a single source of truth for all data. This means that we know which data is stored where and can trust that it is the truth. Services are built and administered in relevant systems in a clear and strict way. When an update becomes available, this automatically evolves to a new version. We have also gained experience with many of the processes, and we have optimised them into lifecycle processes where necessary and where possible. Delivery, adjustment and termination of services is always based on a fixed recipe: a workflow. It is the sheet music the entire symphony plays to.

Beautiful melody

Our digital conductor makes sure that all the right notes are played based on the score: all data is constantly monitored. If any data turns out to be incorrect – for example because a manual adjustment has been made to one of the systems (you can imagine how quickly this can make the music sound off-key) – the composers (SURF) will be informed. They can rectify the inconsistency to restore the beautiful melody: the data is once again consistent with reality.

Ready for service – music to our ears

If all physical components (patch panels, fibres and hardware) are present and connected at a certain location, the network services on the SURFnet8 network can be delivered, adjusted or removed completely and accurately within a few minutes. This also means that a delivered service can be used immediately. The service is also immediately visible in the SURF Network Dashboard,  which can be requested via the Network Dashboard API. Statistics are gathered from the first second. The service is also directly included in the impact analysis in terms of malfunctions and works, and its IP addresses and series – including (reverse) DNS – are well managed and published.

Looking forward – lead sheets

The solid automated base will make it easier to facilitate and integrate new, innovative functionalities in the future. Examples of this are the addition of functionality and technologies to existing products and the creation of new products with building blocks of existing products. This allows us to respond to the needs of education and research more quickly. A new musical score can easily be put together and in some cases, certain building blocks from another score can be reused. A good example of this is Firewall-as-a-Service, for which we use part of the SURFnet8 building blocks.

Keeping track of developments

Another advantage of this complete solution is that we have become less dependent on suppliers. Modularity and standardisation have made us more flexible than ever before in choosing the functionality that is suitable for SURF. Together with our partners at the institutions, SURF keeps track of the relevant developments. If there is an interesting development, we share our knowledge on the subject and/or we integrate it into our environment. The musical score can be rewritten to a newer version that suits these new times. A symphonie fantastique!

International collaboration

We can keep our maintenance, quality and security at the appropriate levels by making the orchestrator core code open source in the near future and by working together with (international) partners. We hope that more and more networks will use an orchestrator in the future, and we are anticipating interoperability between the domains (for example, through the use of NSI via NetherLight). We can see this happening slowly on an international level, and this can also be interesting on campus. For example, this makes it possible to deliver a network service with certain quality requirements all the way to the researcher.

Machine learning

Structured data that is made clear on a central level also offers more possibilities for applying machine learning. For example, my colleague Caspar recently researched how we can make predictions based on statistics from SURFinternet connections. In the future, this will enable us to automatically expand a connection's capacity based on these analyses. We will also be able to automatically combine certain necessary building blocks to make it easier for researchers to do their work. Just like Aimée, the assistant-composer with artificial intelligence supporting the extensive automatic creation of symphonies ... :-).

* illustrations by André Slob, taken from the presentation by my colleague Ruben van den Brink