LAN Physical to Overlay

It is terribly difficult to express the design elements of VxLAN technology in a single drawing.  As seen in previous blogs, the elements of the physical construct of an ECMP (Equal-Cost Multi-Path) spine and leaf is astonishingly complex.  These become represented in abstracted views of the total in graduations from quite simple to piece-by-piece elements in order to express their true nature.

Part of the problem is associated with these complexities, but having put together multiple descriptive models, it really is because the OVERLAY doesn't look anything like the physical design.  Networkers got used to broadcast domains being associated with a wire.  Then being associated with an L2 abstraction.  Now networkers need to figure out how to represent the multiple levels of networking with additional L2-in-L3 tunnels.

It's also not as if this hasn't happened before.  VPN drawings look much like VxLAN drawings, but in a VPN drawing, there may be one or a couple, in VxLAN there could be a lot more.  Ultimately, the issue becomes how complex the drawing is to layer in all of the information.

Starting from the view of a Modern Platform for Enterprise (as opposed to public cloud), networkers need to connect particular security elements, network services, control plane services, backup, networking and platforms.  Architecturally, the model looks very much like this.

Figure 1.  Modern Platform

DC LAN (red) plays a dominant role in this platform architecture, providing connectivity from any element to any element within the construct.  Be it a physical device, a virtual machine on a hypervisor or a container on an operating system.

Figure 2.  Any to Any

It's not as simple as that though.  The network attributes are overlaid on top of hardware, software and logical (or abstracted) networking mechanisms in this model.  In this extremely simplified model, use case 1 is where the hypervisor communicates with another hypervisor (green) and use case 2 where the hypervisor communicates with a bare metal object.  A third use case exists where a container communicates with hypervisor or hardware, but it's not used frequently yet in the Enterprise.

The VxLAN (Virtual eXtensible LAN) kernel module of the hypervisor (in this case VMware ESX) communicates through it's vSwitch to the physical medium, a network adapter on the physical machine.  This is then passed to the physical switch to be passed in accordance to the VNI (Virtual Network Identifier) associated with the packets.  In case 1, it is then picked up by the network card on the second physical host, working though the vSwitch to get to the virtual machine.  In case 2, the endpoint is a VTEP (Virtual Tunnel EndPoint) where it is de-encapsulated from L3 to L2 to arrive at the physical server.

In both cases, two sets of interacting Control Planes, VMware NSX and Arista CVX provide the path information to instantiate the VxLAN tunnel.

Figure 3.  VxLAN Delivery (gratuitous re-use from earlier blog)

Now, there needs to be a bold red blinking sign that states the network still exists, both in its physical and logical form.  Herein lies the Spine-and-Leaf.  It is similar to a Clos design (as in Charles Clos) that does have some oversubscription at different levels.  Physically it utilizes the spine as a one hop transport route between all leaf nodes.

The concept though is quite simple.  Scale out horizontally as large as possible.  When that is exceeded, add another spine and make a 2 hop transport route.  The Leaf nodes come in two basic flavors, one that provides transport access to hosts, the other leaf flavor provides Services.  Services Leaf are used to establish specific service functions.  The two examples shown here are WAN access with perimeter security Firewall and IP protocol based storage.

Moving from Brown to Green in this model should be relatively easy.  Once the Spine and Leaf is established, legacy networks may be connected at a Leaf node to provide access utilizing the switch VTEP.  It's an extra hop, but the latency in this network type is extremely low.

Figure 4.  Spine and Leaf (high level)

Combining Figure 3 and Figure 4 is terribly problematic.  It may not even be useful at large scale due to the enormous details necessary to show it in its entirety.  

What networkers can do is abstract the drawing a bit.  Pull it up from the physical layer.  The view in Figure 5 is just such a drawing.

Utilizing Figure 4 as the groundwork (the physical) and "pulling up" the abstractions and overlays that reside on it.  

In this scenario, the spine and each leaf are BGP Autonomous Systems.  The first layer above the physical equipment is an IP network routed solely within the confines of the spine and leaf (Blue routers).  Additionally, a VRF is run on the same structure to propagate a management network (Red routers) to all switches.  It also acts as the distribution layer for any management switches deployed within the Leaf cabinets.

The VNI framework is then "pulled up" to the final level.  This is the tunnel path.  

Below the physical switch VTEP, an OVSDB (OpenVSwitch DataBase) is used to manage the interactions of all platform systems that are virtualized with, in this case, VMware NSX.

At and above the physical switch are the VTEP, managed by another OVSDB for all physical element connections not associated with VMware NSX.  The control plane in this model is Arista CVX.

As in Diagram 3, the OVSDB (VMware) communicates with the OVSDB (Arista) to manage the entirety of the tunnel formation.

Figure 5.  An high level view of the network from physical to overlay

In light of creating a view of the network that expresses as much as possible with the least amount of complexity, this drawing has proven to be quite good at showing as much as possible without making it impossible to follow.

If you use this model to describe your network, please do let me know how it turned out.

I'm interested in any other way you may have to show this type of information graphically.  Please tweet them to @abusedbits or link them to my Google+.