Over the past decade, as the size of the network has grown, we have seen a shift from the traditional three-tier network architecture to the flatter, wider Spine-Leaf architecture. With its fully meshed connectivity, the Spine-Leaf architecture gives us the predictable high-speed network performance we desire and the reliability in our network switch fabric.

In addition to its advantages, the Spine-Leaf architecture presents challenges in terms of structured cabling. In this article, we will examine how to construct and expand a four-directional spine and gradually develop to a larger spine (such as the 16-spine spine) and maintain linear velocity switching capability and redundancy during growth. We will also explore the advantages and disadvantages of the two methods in building the main distribution areas of our structured fiber optic cable; one method using traditional fiber optic patch cords and the other using optical grid modules.

A brief history

Since the advent of the Local Area Network (LAN) protocol in the 1980s, Ethernet has been the driving force behind data center and Internet development with its simple algorithms and low manufacturing costs. The Ethernet switch looks at every packet it receives before switching. It only opens the outer envelope to read the address of layer 2 without worrying about reading the IP address. This allows Ethernet switches to move packets very quickly.

Despite the high efficiency of Ethernet, as the size of the network increases, it also has some shortcomings. In a network of multiple Ethernet switches, a technique called Spanning Tree Protocol (STP) is used to prevent broadcast packets such as Address Resolution Protocol (ARP) requests from flooding and circulating in the network. STP blocks redundant links to prevent loops from occurring in the network. A network running on STP technology uses a redundant link as a failover when the primary link fails. This provides flexibility to the infrastructure at the expense of half the utilization of available bandwidth.

Before we encountered new problems, we built the network for a long time with the expansion tree logic. The first problem is that our dual-core network is limited and there is no room for growth (in order to serve more and more customers, our network needs to grow accordingly). The second issue is the delay. If we have a large network, we usually divide them into smaller networks, which we call virtual LANs (vlans). This will result in different types of data traffic with different delays.  Traffic through a Layer 2 network in a single VLAN has different delays than traffic between different VLANs through a Layer 3 core.

▲ A typical three-tier network that supports the extended tree protocol. Redundant links are blocked to prevent network loops.

Introduction to the Spine-Leaf architecture

Modern e-commerce, social media, and cloud applications mostly use distributed computing to serve customers. Distributed computing means that the server talks to the server and works in parallel to create a dynamic web page and answer customer questions; it requires the same latency.  Waiting for the results will make the customer unsatisfied. We need a network architecture that grows evenly and provides uniform latency for modern applications.

The solution to these problems comes from a network architecture, the “Spine-Leaf architecture” that is said today. This idea has been around since the first introduction of multi-level circuit-switched networks (also known as the Clos network) by Charles Clos in 1952. The backbone of this network architecture is called the Spin, and each Leaf is connected from the Spin to a further extended network resource. Simply add more Spin or Leaf switches and the network grows evenly without changing network performance.

Compared with the traditional 3-layer architecture, the Spin portion of the network grows horizontally, constraining the number of layers in the network. For example, with two-way Spin, we can build a network that supports up to 6,000 hosts. With four-way Spin, we can build up to 12,000 hosts on the network. With 16-way Spins, we can have more than 100,000 10-GbE hosts.

Second, all Leaf switches are connected to each available Spin switch in the fabric. This fully meshed architecture allows any host connected to the Leaf to connect to other hosts using only two hops, a switch-to-switch connection. For example, Leaf1 to Leaf1 and Leaf1 to Leaf10. Because the entire spine layer is built redundantly (in the case of Spin or Leaf switch downtime), alternate paths and resources can be used automatically.

The basic rules for establishing a Spin-Leaf network are as follows:

The main building blocks are network Leaf switches and network spin switches.

All hosts can only connect to the Leaf switch.

The Leaf switch controls traffic between servers.

The Spin switch switches traffic forward along the best path between Layer 2 or Layer 3 Leaf sub-switches.

The upstream port count on the Leaf switch determines the maximum number of spin switches.

The number of Spin switch ports determines the maximum number of Leaf switches

These principles affect how switch manufacturers design their devices.

Look closely at a spin switch. If we look at a typical Spin switch, at first glance we notice multiple expansion slots, for example 4 or 8 accept different line cards for connecting to the Leaf switch uplink.

▲ In a Spin-Leaf network architecture, the Leaf switch controls traffic between servers, while the Spin switch forwards traffic along the best path between Leaf switches.

A fabric called 16-way Spin can be extended to support more than 100,000 10 Gigabit Ethernet hosts.