SURF Orchestration ♫ is music to our ears
The delivery of a network service to a customer entails far more than just one activity. When the activities are harmoniously combined, all of the administrative and technical components will strike the right note at the right time to achieve the desired situation. We call this orchestration. Since the end of 2016, SURF has been developing its own digital conductor, who provides, adapts or discontinues network services at the request of an authorised person or process. Why is this solution important for SURF's member institutions? How can orchestration help education and research? This blog post will tell you more about this subject.
A conductor of a symphonic orchestra always has a number of fixed instruments in his orchestra: the wind section, strings, the bass section and percussion. The SURF orchestrator also has a number of fixed systems that are part of the delivery of a SURFnet8 or NetherLight network service: CRM for customer-related data, IMS for the technical registration of services, JIRA for project tracking, IPAM for IP address management and NSO for the activation of the network services. Since the start of this project, thousands of workflow symphonies have already been performed, and these workflows have resulted in the delivery, adjustment or removal of network services. And the number of workflows is continuously increasing.
SURF devises, designs, builds and manages the Dutch network for education and research, and has in-house knowledge and expertise to make choices between different technologies and to direct their development. Automation & Orchestration is an important pillar: it takes the next step in the maintenance of a state-of-the-art network and the associated services. This development also opens up new opportunities, such as the integration with computing and storage facilities and other networks.
A solid foundation – a fixed music sheet
Services can only be delivered, adjusted and removed easily, consistently and quickly if a number of basic elements are well balanced. The automation project at SURF has paid a lot of attention to this from day one. It has examined the entire data architecture in order to ensure a single source of truth for all data. This means that we know which data is stored where and can trust that it is the truth. Services are built and administered in relevant systems in a clear and strict way. When an update becomes available, this automatically evolves to a new version. We have also gained experience with many of the processes, and we have optimised them into lifecycle processes where necessary and where possible. Delivery, adjustment and termination of services is always based on a fixed recipe: a workflow. It is the sheet music the entire symphony plays to.
One of our developers, René Dohmen (in the middle at the top and in closeup from 3:11 to 4:07 in the video), recorded a beautiful piece of music together with his big band during the coronavirus lockdown. This is a great example of how the conductor makes sure that the right people play the right instrument at the right time and work together to achieve the right outcome!
Our digital conductor makes sure that all the right notes are played based on the score: all data is constantly monitored. If any data turns out to be incorrect – for example because a manual adjustment has been made to one of the systems (you can imagine how quickly this can make the music sound off-key) – the composers (SURF) will be informed. They can rectify the inconsistency to restore the beautiful melody: the data is once again consistent with reality.
Ready for service – music to our ears
If all physical components (patch panels, fibres and hardware) are present and connected at a certain location, the network services on the SURFnet8 network can be delivered, adjusted or removed completely and accurately within a few minutes. This also means that a delivered service can be used immediately. The service is also immediately visible in the SURF Network Dashboard, which can be requested via the Network Dashboard API. Statistics are gathered from the first second. The service is also directly included in the impact analysis in terms of malfunctions and works, and its IP addresses and series – including (reverse) DNS – are well managed and published.
Looking forward – lead sheets
The solid automated base will make it easier to facilitate and integrate new, innovative functionalities in the future. Examples of this are the addition of functionality and technologies to existing products and the creation of new products with building blocks of existing products. This allows us to respond to the needs of education and research more quickly. A new musical score can easily be put together and in some cases, certain building blocks from another score can be reused. A good example of this is Firewall-as-a-Service, for which we use part of the SURFnet8 building blocks.
Keeping track of developments
Another advantage of this complete solution is that we have become less dependent on suppliers. Modularity and standardisation have made us more flexible than ever before in choosing the functionality that is suitable for SURF. Together with our partners at the institutions, SURF keeps track of the relevant developments. If there is an interesting development, we share our knowledge on the subject and/or we integrate it into our environment. The musical score can be rewritten to a newer version that suits these new times. A symphonie fantastique!
We can keep our maintenance, quality and security at the appropriate levels by making the orchestrator core code open source in the near future and by working together with (international) partners. We hope that more and more networks will use an orchestrator in the future, and we are anticipating interoperability between the domains (for example, through the use of NSI via NetherLight). We can see this happening slowly on an international level, and this can also be interesting on campus. For example, this makes it possible to deliver a network service with certain quality requirements all the way to the researcher.
Structured data that is made clear on a central level also offers more possibilities for applying machine learning. For example, my colleague Caspar recently researched how we can make predictions based on statistics from SURFinternet connections. In the future, this will enable us to automatically expand a connection's capacity based on these analyses. We will also be able to automatically combine certain necessary building blocks to make it easier for researchers to do their work. Just like Aimée, the assistant-composer with artificial intelligence supporting the extensive automatic creation of symphonies ... :-).
* illustrations by André Slob, taken from the presentation by my colleague Ruben van den Brink