Mar 7, 2017

Aeon-ZTPS: Bootstrap a Multi-Vendor Network

system-fail-situation-network.jpgWhether you have a data center, campus, or remote branch deployment, bootstrapping new equipment is not fun.  The job is very time consuming and highly error prone when done manually. People want a “human free” solution — commonly called zero-touch-provisioning.  The basic concept is simple: power-on equipment from factory-reset, get an IP address from DHCP, get the right version of network operating system (NOS) installed, and finally apply the device specific configuration.  Simple in theory, not so great in practice.  Here’s why:

  • No standard process. Each network equipment vendor / NOS has a different mechanism. Each mechanism has its own set of special requirements and implementation challenges.  Traditional networking vendor NOS, like Cisco NX-OS has POAP and Arista EOS has Arista-ZTP.  More modern networking hardware use the Open Network Install Environment (ONIE) for initial NOS install. ONIE is only part of the overall bootstrap solution.  Cumulus Linux, for example, has an Auto-Provisioning ZTP process that runs post ONIE for device configuration.
  • No process visibility.  Each of these processes generally involve a “script” that you must create, and this script is downloaded and executed as part of the NOS specific ZTP process.  How do you determine how the bootstrap process is proceeding?  More often when it fails, what visibility (e.g. logs) exists to help you troubleshoot the issue?  Some NOS bootstrap processes can take tens of minutes.   During this “blackout period” you are left wondering what’s going on.  Any visibility that you want, you have to explicitly code into your scripts.
  • No one size fits all.  Each piece of network equipment may need a different NOS or version of NOS perhaps depending on device model, its role of function within the network, or other specific criteria.  Each device configuration is going to be unique.  Attempting to mash all these different factors into a single script you have to write creates a number of complex dependencies.

Because you are responsible for writing the “ZTP script”, you assume a huge burden to deal with a lot of complex tasks around the mechanics of the bootstrap process — rather than focusing exclusively on the aspects specific to the network services you need to deploy. There is not a lot of value to write the “install the OS on this device” code, for example.  But there is a lot of value for you to template build specific configurations for specific device network services because that is what makes the ZTP process unique for your network.

In my travels working in network automation I got the opportunity to talk with a number of companies that had to implement a solution to this problem — data centers, campus, branch offices — many networks; same problem.  Their solutions were very specific to their company, and could not be open-sourced or made to be more general purpose.  This common use-case of network bootstrapping does lend itself to the need for an open-source project that provides the framework mechanics.

What is Aeon-ZTPS?

Aeon-ZTPS is an open-source project that gives you an out-of-the-box server / virtual-machine to solve the problem of bootstrapping a multi-vendor network.  The project is designed to run on the popular hypervisors: VirtualBox, ESXi, and KVM.  This project is sponsored by Apstra, Inc. because many of our customers need a universal ZTP server to manage a hybrid network.  The Apstra Customer Enablement team that I head up is responsible for managing the project.   Full documentation is available on ReadTheDocs.

The server provides a GUI, runs a web-service providing a REST API, uses a concurrent process-task queuing system, and maintains a database backend.  The web-based GUI (that uses the REST API) gives you a heads-up display on bootstrap progress.  For example:


The system maintains a collection of logs in /var/logs/aeon-ztp providing explicit information on each action and stage of the bootstrap process.  Much of the information found in the logs is also available via the GUI and API.

The Aeon-ZTPS can also be setup to also provide ISC-DHCP server; and comes equipped with example ISC-DHCP server config setup for each of the provided vendors.  More commonly, however, the DHCP server is a separate server, so all you will need to do is setup the proper DHCP options and point them to the Aeon-ZTPS server.  There are no device-unique requirements in any of the DHCP setup; i.e. no need to have serial-numbers or MAC-addresses configured or coordinated in the DHCP config!

The Aeon-ZTPS server ships with the vendor specific boot-scripts that enable remote management and trigger a registration function.  All you need to do is to configure a few files and provide your “finally” script as described in the documentation.  You are also responsible for copying the vendor specific files (e.g. NOS binaries) onto the Aeon-ZTPS since those files must come directly from your vendors.

How does it Work?

The core concept of the Aeon-ZTPS approach is to do as little as possible during the device factory bootup execution; i.e. the part that is vendor specific (POAP vs. Arista-ZTP vs. AutoProv, etc).  The vendor required boot-script simply needs to enable remote management and notify the Aeon-ZTPS server that it is ready to be bootstrapped.  The Aeon-ZTPS project currently provides these vendor specific scripts for Cisco, Arista, and Cumulus.  Additional equipment / NOS support can be easily added.  Once the Aeon-ZTPS is notified (registered), then the process to perform NOS upgrade and basic configuration can be performed in a controlled environment that provides step-by-step visibility via log files, a REST API, and a GUI.  The overview of the complete process is shown:


The final stage in the process is to call a User provided script (oh no!), but in this case the script is called after the devices has the correct NOS version and some basic configuration. In this way, the User can focus on the specific device setup they need to accomplish, rather that all of the other bits (like NOS upgrade).  The User can choose any method they want to do the device specific configuration.  For example, the “finally” script could run an Ansible playbook, install software agents like Puppet, Chef, or the Apstra Operating System (AOS®) device agent.


Chances are you are responsible for managing a complex network.  You have to deal with this bootstrapping problem.  You know you need to do more network automation.  You can get started now using the Aeon-ZTPS project to deal with the drudgery of the bootstrap mechanics and allow you to focus on what you do best.   I hope this project helps you along your automation journey.  We look forward to your feedback and suggestions via the github repository.  The project is still in its early field use and we appreciate your time and community engagement!

Read our white paper on Understanding IBA icon-arrow-right-double-24.png