Configuring Affinity between vSmart and vEdge Devices

Last updated
Save as PDF

One way to manage network scale is to configure affinity between vSmart controllers and vEdge routers. To do this, you place each vSmart controller into a controller group, and then you configure which group or groups a vEdge router can establish control connections with. The controller groups are what establishes the affinity between vSmart controllers and vEdge routers.

Configure the Controller Group Identifier on vSmart Controllers

To participate in affinity, each vSmart controller must be assigned a controller group identifier:

vSmart(config#) system controller-group-id number

The identifier number can be from 0 through 100.

When vSmart controllers are in different data centers, it is recommended that you assign different controller group identifiers to the vSmart controllers. Doing this provides redundancy among data centers, in case a data center becomes unreachable.

For vSmart controllers in the same a data center, they can have the same controller group identifier or different identifiers:

If the vSmart controllers have the same controller group identifier, a vEdge router establishes a control connection to any one of them. If that vSmart controller becomes unreachable, the router simply establishes a control connection with another one of the controllers in the data center. As an example of how this might work, if one vSmart controller becomes unavailable during a software upgrade, the vEdge router immediately establishes a new TLOC with another vSmart controller, and the router's network operation is not interrupted. This network design provides redundancy among vSmart controllers in a data center.
If the vSmart controllers have different controller group identifiers, a vEdge router can use one controller as the preferred and the other as backup. As an example of how this might work, if you are upgrading the vSmart controller software, you can upgrade one controller group at a time. If a problem occurs with the upgrade, a vEdge router establishes TLOCs with the vSmart controllers in the second, backup controller group, and the router's network operation is not interrupted. When the vSmart controller in the first group again become available, the vEdge router switches its TLOCs back to that controller. This network design, while offerring redundancy among the vSmart controllers in a data center, also provides additional fault isolation.

Configure Affinity on vEdge Routers

For a vEdge router to participate in affinity, you configure the vSmart controllers that the router is allowed to establish control connections with, and you configure the maximum number of control connections (or TLOCs) that the vEdge router itself, and that an individual tunnel on the router, is allowed to establish.

Configure a Controller Group List

Configuring the vSmart controllers that the router is allowed to establish control connections is a two-part process:

At the system level, configure a single list of all the controller group identifiers that are present in the overlay network.
For each tunnel interface in VPN 0, you can choose to restrict which controller group identifiers the tunnel interface can establish control connections with. To do this, configure an exclusion list.

At a system level, configure the identifiers of the vSmart controller groups:

vEdge(config)# system controller-group-list numbers

List the vSmart controller group identifiers that any of the tunnel connections on the vEdge router might want to establish control connections with. It is recommended that this list contain the identifiers for all the vSmart controller groups in the overlay network.

If, for a specific tunnel interface in VPN 0, you want it to establish control connections to only a subset of all the vSmart controller groups, configure the group identifiers to exclude:

vEdge(config-vpn-0-interface)# tunnel-interface exclude-controller-group-list numbers

In this command, list the identifiers of the vSmart controller groups that this particular tunnel interface should never establish control connections with. The controller groups in this list must be a subset of the controller groups configured with the system controller-group-list command.

To display the controller groups configured on a vEdge router, use the show control connections command.

Configure the Maximum Number of Control Connections

Configuring the maximum number of control connections for the vEdge router is a two-part process:

At the system level, configure that maximum number of control connections that the vEdge router can establish to vSmart controllers.
For each tunnel interface in VPN 0, configure the maximum number of control connections that the tunnel can establish to vSmart controllers.

By default, a vEdge router can establish two OMP sessions for control connections to vSmart controllers. To modify the maximum number of OMP sessions:

vEdge(config)# system max-omp-sessions number

The number of OMP sessions can be from 0 through 100.

A vEdge router establishes OMP sessions as follows:

Each DTLS and and each TLS control plane tunnel creates a separate OMP session.
It is the vEdge router as a whole, not the individual tunnel interfaces in VPN 0, that establishes OMP sessions with vSmart controllers. When different tunnel interfaces on the router have affinity with the same vSmart controller group, the vEdge router creates a single OMP session to one of the vSmart controllers in that group, and the different tunnel interfaces use this single OMP session.

By default, each tunnel interface in VPN 0 can establish two control connections. To change this:

vEdge(config-vpn-0-interface)# vpn 0 interface interface-name tunnel-interface max-control-connections number

The number of control connections can be from 0 through 100. The default value is the maximum number of OMP sessions configured with the system max-omp-sessions command.

When a vEdge routers has multiple WAN transport connections, and hence has multiple tunnel interfaces in VPN 0, the sum of the maximum number of control connections that all the tunnels can establish cannot exceed the maximum number allowed on the router itself.

To display the maximum number of control connections configured on an interface, use the show control local-properties command.

To display the actual number of control connections for each tunnel interface, use the show control affinity config command.

To display a list of the vSmart controllers that each tunnel interface has established control connections with, use the show control affinity status command.

Best Practices for Configuring Affinity

In the system controller-group-list command on the vEdge router, list all the controller groups available in the overlay network. Doing so ensures that all the vSmart controllers in the overlay network are available for the affinity configuration, and it provides additional redundancy in case connectivity to the preferred group or groups is lost. You manipulate the number of control connections and their priority based on the maximum number of OMP sessions for the router, the maximum number of control connections for the tunnel, the controller groups a tunnel should not use.
A case in which listing all the controller groups in the system controller-group-list command provides additional redundancy is when the vEdge router site is having connectivity issues in reaching the vSmart controllers in the controller group list. To illustrate this, suppose, in a network with three controller groups (1, 2, and 3), the controller group list on a vEdge router contains only groups 1 and 2, because these are the preferred groups. If the router learns from the vBond controller that the vSmart controllers in groups 1 and 2 are up, but the router is having connectivity issues to both sites, the router loses its connectivity to the overlay network. However, if the controller group list contains all three controller groups, even though group 3 is not a preferred group, if the router is unable to connect to the vSmart controllers in group 1 or group 2, it is able to fall back and connect to the controllers in group 3.
Configuring affinity and the order in which to connect to vSmart controllers is only a preference. The preference is honored whenever possible. However, the overarching rule in enforcing high availability on the overlay network is to use any operational vSmart controller. The network ceases to function only when no vSmart controllers are operational. So it might happen that the least preferred vSmart controller is used if it is the only controller operational in the network at a particular time.
When a vEdge router boots, it learns about all the vSmart controllers in the overlay network, and the vBond orchestrator is continuously communicating to the router which vSmart controllers are up. So, if a vEdge router cannot reach any of the preferred vSmart controllers in the configured controller group and another vSmart controller is up, the router connects to that controller. Put another way, in a network with multiple vSmart controllers, as a last resort, a vEdge router connects to any of the controllers, to ensure that the overlay network remains operational, whether or not these controllers are configured in the router's controller group list.
The controller groups listed in the exclude-controller-group-list command must be a subset of the controller groups configured for the entire router, in the system controller-group-list command.
When a data center has multiple vSmart controllers that use the same controller group identifier, and when the overlay network has two or more data centers, it is recommended that the number of vSmart controllers in each of the controller groups be the same. For example, if Data Center 1 has three vSmart controllers, all with the same group identifier (let's say, 1), Data Center 2 should also have three vSmart controllers, all with the same group identifier (let's say, 2), and any additional data centers should also have three vSmart controllers.
When a data center has vSmart controllers in the same controller group, the hardware capabilities—specifically, the memory and CPU—on all the vSmart controllers should be identical. More broadly, all the vSmart controllers in the overlay network, whether in one data center or in many, should have the same hardware capabilities. Each vSmart controller should have equal capacity and capability to handle a control connection from any of the vEdge routers in the network.
When a router has two tunnel connections and the network has two (or more) data centers, it is recommended that you configure one of the tunnel interfaces to go to one of the data centers and the other to go to the second. This configuration provides vSmart redundancy with the minimum number of OMP sessions.
Whenever possible in your network design, you should leverage affinity configurations to create fault-isolation domains.

Additional Information

High Availability Configuration Examples
High Availability Overview