This post was originally published on the Packet Pushers’ Ignition site on March 2, 2020.
Perhaps you haven’t found the motivation to start learning how to interact with infrastructure as code. Fair enough. “Infrastructure as code” sounds like a gimmick, and if you’ve got some of that hard-won experience, you’re skeptical that IaC is worthwhile. We all have plenty of work to do.
If your pain tolerance for the CLI is high enough, learning new technology seems like more trouble than it’s worth. Why learn something else if what you already know is good enough?
CLI Output Is Not Structured Data
The CLI does not offer structured data. Instead, you’re getting formatted data–unstructured data that is displayed on a screen in a human-readable way. Maybe to your trained eye the data seems structured.
For example, you know that a number is numerical type of data and a bunch of text is string type of data. You can infer structure from the columnar format the data is presented in, since you know how to read the column header and the values listed underneath.
Take a look at this familiar block of formatted but programmatically unstructured data.
That data looks nice enough to your eye, your eye being the intended audience. That pretty output doesn’t help you programmatically, though. The data your brain knows how to cope with is not structured in a way that’s useful for machines. This is such a well-known issue that the Python libraries Netmiko and NAPALM address it somewhat, normalizing and structuring much of the data that is scraped from an interactive CLI session.
Maybe you’re thinking that all my fussing about structured data is just another windmill to tilt at. Surely, you can wrestle CLI output into usefulness, right? We all know of clever Excel pasting techniques and fancy text manipulation that can whip screen-scraped data into shape. Hey, I’ve gotten awfully good over the years with my Find/Replace to turn something ugly I copied from the screen into a CSV I could import into Excel. But this is not the way forward. We have better tools now.
Structured Data
Sane network automation demands structured data. For our purposes, I’m defining structured data as data that is sent back to you in a delimited, normalized fashion from a device. Data you can manipulate. Data you can make use of.
Two examples of structured data include SNMP and API responses to GET queries.
SNMP data is structured. You can read exactly how every OID is to be interpreted via the MIB definition. If you’re in the Cisco ecosystem, Cisco offers a free, unauthenticated MIB Locator tool. From this tool, you can download any MIB file describing the SNMP OIDs supported by a variety of devices running IOS family code. Those MIB files, cryptic and frustrating though they are to parse, explain the structure of the data that will be returned to an SNMP GET request when a specific OID is polled.
API data is structured. You can read exactly how the API data is structured in the API’s documentation. For instance, once you’ve installed your own copy of NetBox, you can find the API documentation in /api/docs/. You’ll discover that the NetBox API returns data as JSON. You’ll also find the various key-value pairs contained in the JSON objects returned, depending on your query.
Questions & Answers
A caveat for both SNMP and API data is that they aren’t always documented. This is a huge deal when performing acts of network automation. Why? Only from documentation can you determine how to programmatically format a question and what sort of answer to expect back. In that context, even cryptic documentation is better than no documentation.
In the case of SNMP, a GET operation is how you perform a polling operation. The GET will be of a specific OID in the form of a long, hierarchical, dotted SNMP number in forming the question. The answer will be an integer. Or an IP address. Or a string. Or a 64-bit counter. Etc.
In the case of an API, there’s a good chance the question is a RESTful operation against a specific URL. The answer will come back as JSON. Or XML (yuck). Or something else.
The Usefulness Of Structured Data
Structured data is useful. For instance, I know that when I perform a RESTful GET against the NetBox API /dcim/device-types/{id}/ URL, I’m going to get back a JSON object that contains, among several other key-value pairs, a key called display_name containing a string value. Maybe I need that value.
Have a look at this pretty-printed JSON I get from a NetBox API call using curl.
Fantastic! In a script, I can reference whatever the object is. What could I do with this data now that it’s easily referenced?
- I could store it in a database to check on later.
- I could print it on the screen as part of a report.
- I could use it in a calculation to determine link utilization.
- I could graph it on a web page to detect anomalies.
- I could send it to a Slack channel as part of an ops team alert.
The number of use cases is as unique as your requirements, and that’s really my point. Structured data opens the door for everything else you want to do related to network automation. When your network data is structured, the wall of obscurity surrounding the questions, “Is this device doing what I want it to?” and “Are things going okay?” comes down.
To me, this is the way forward.