Streaming Telemetry – Resistance Is Futile

Why Streaming Telemetry?

Network management is that area of networking that has been lying in a dormant state for the past 3 decades, begging for improvement. Telemetry seems to be the ray of hope that is finally bringing the long-awaited KaiZen to this field.

Imagine a beautiful day walking down a street, passing by jeweler’s shop when a masterpiece on display catches your eye. What you are looking at is a piece of metal that started its journey in some mine halfway across the globe. It had to be extracted from deep down within the earth and sieved through tons of mud before being sorted and cleared to be made into a piece of some value. It is then passed through the hands of an experienced and skilled craftsman and eventually metamorphosed into a piece of art.

We have good craftsmen waiting for the gold nugget to arrive for them to start working, we are currently supplying them pieces of iron in the form of SNMP and Syslog collected data over a time interval of 5-15min. Telemetry is a gold mine which would help produce this long-awaited rich granular data source.

Last decade has seen significant improvement in facial, handwriting & speech recognition. thanks to the ton of labeled and raw data set provided by internet user for the algorithm to be trained. Same needs to be done for networking device and this is where raw telemetry data can fill the void.

Juniper has been supporting streaming telemetry for quite a while and new sensor and feature are added every new release.

We are going to proceed forward with certain assumption of readers basic knowledge on prerequisite below topic. For benefit of user who would like to read up on this topic we are including some reference to external blog covering this topic in more details. (Please check the reference section)

Streaming JUNOS Telemetry

The traditional model for monitoring the health of a network is based on a so-called “pull” model, using SNMP and CLI to periodically poll network elements. These methods have inherent scalability limitations and are resource intensive, particularly when polling a large number of metrics at a very high frequency.

Telemetry flips this around entirely and eliminates the need for polling, by relying instead on a “push” model to asynchronously deliver the telemetry data as a stream to a downstream collector. This approach is much more scalable and supports the monitoring of thousands of objects in a network with granular resolution.

Currently, Junos exposes telemetry data over UDP and gRPC transports as part of Junos Telemetry Interface (JTI). Anyone willing to consume telemetry from Junos devices can write a collector that listens for incoming UDP telemetry stream packets from Junos device or open a gRPC connection to the device and receive streaming telemetry. This would bring in a requirement for customers to either write and maintain a collector or use an existing one that can receive JTI data.

picture1

picture2

Fig1: Streaming telemetry in Junos

Supported Transports, Encodings and Data Models:

picture3

Sample Protobuf

picture4

Sample Openconfig

picture5

Once we enable telemetry on device their need to be application that is able to consume this data, one option would be write an application that could accept this data for future decoding, processing and storing. That might not be a practical solution unless you got army of developer. Not to worry Juniper has contributed plugin for few widely used application like Telegraf and FluentD, which could be used for consuming this streaming telemetry data. What if even that would be bit too much to bite on, not to worry there is a Juniper initiated opensource project on GitHub called “OpenNTI” which is combination of other open source tool made into a complete package.

screen shot 2019-01-25 at 3.19.47 pm

Fig3: Typical telemetry deployment.

As the big push arises in automating the whole network end to end with the idea of rapidly deploying services, AT&T took a leap and developed an automation framework called ECOMP which was later open sourced and called as ONAP. In subsequent part of this blog we would cover ONAP and then integration of juniper telemetry in ONAP ecosystem.

See you on the next part…if you like the blog and want to continue on journey of this telemetry packet” To boldly go where no telemetry packet has gone before”.

“Beam me out, Scotty”

Reference:

Data format:

  • JSON https://www.json.org/
  • Protobufs https://developers.google.com/protocol-buffers/docs/overview
  • Openconfig http://www.openconfig.net/

Transport of data:

  • gRPC https://blog.netsil.com/http-2-and-grpc-the-next-generation-ofmicroservices-interactions-aff4ffa6faed

Application Eco-system:

  • Logstash     https://github.com/elastic/logstash
  • Fluentd       https://github.com/fluent/fluentd
  • Telegraf      https://www.influxdata.com/time-series-platform/telegraf/

Leave a Reply

Your email address will not be published. Required fields are marked *