High Availability Configuration Examples
This article provides examples of configuring high availability, specifically, of configuring affinity between vSmart controllers and vEdge routers.
Configure Affinity to vSmart Controllers in a Single Data Center
In an overlay network with a single data center that has multiple vSmart controllers, if you want the vEdge routers to establish a single control connection to one of vSmart controllers, there is no need to configure affinity because this situation is the default behavior.
However, if you want the vEdge routers to establish control connections to more than one vSmart controllers, to provide redundancy in case one of the controllers becomes unavailable, you configure affinity. You generally place the vSmart controllers in the same controller group.
Let's say that all the vSmart controllers use the same controller group identifier, 1. You configure the identifier on all three controllers as follows:
vSmart(config)# system controller-group-id 1
To verify the configuration, use the show running-config command:
vSmart# show running-config system system description "vSmart in data center 1" host-name vSmart gps-location latitude 37.368140 gps-location longitude -121.913658 system-ip 172.16.255.19 site-id 100 controller-group-id 1 organization-name "Viptela Inc" clock timezone America/Los_Angeles ...
We want the three vEdge routers to establish two control connections to two of the three vSmart controllers. We do this for purposes of redundancy, in case one of the controllers becomes available. Because all the vSmart controllers are in the same controller group, we cannot specify or influence which of the two controllers the vEdge routers connect to. The configurations on all three routers are effectively identical. We show here the configuration for router vEdge-1.
First, configure the available vSmart controller groups. This scenario has just one group:
vEdge-1(config)# system controller-group-list 1
By default, a vEdge router can establish two control connections. Because we want each vEdge router and each tunnel interface to connect to two vSmart controllers, no configuration is required here. However, if you want to explicitly configure the these parameters, you configure the maximum number of OMP sessions at the system level and the maximum number of control connections per tunnel:
vEdge-1(config)# system max-omp-sessions 2
vEdge-1(config)# vpn 0 interface ge0/2 tunnel-interface
vEdge-1(config-tunnel-interface)# max-control-connections 2
Here are the relevant configuration snippets from vEdge-1:
vEdge-1# show running-config system system host-name vEdge-1 gps-location latitude 43.0 gps-location longitude -75.0 system-ip 172.16.255.11 site-id 100 max-omp-sessions 2 controller-group-list 1 organization-name "Viptela Inc" ... vEdge-1# show running-config vpn 0 ... interface ge0/2 ip address 10.0.5.11/24 tunnel-interface encapsulation ipsec color lte max-control-connections 1 no allow-service bgp allow-service dhcp allow-service dns allow-service icmp no allow-service sshd no allow-service netconf no allow-service ntp no allow-service ospf no allow-service stun ! no shutdown ...
To display the control connections with the vSmart controllers, use the show control connections command. The last column, Controller Group ID, lists the vSmart controller group that the vEdge router is in.
vEdge1# show control connections
PEER PEER CONTROLLER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC GROUP
TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE UPTIME ID
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart dtls 172.16.255.19 100 1 10.0.5.19 12446 10.0.5.19 12446 lte up 0:00:00:53 1
vsmart dtls 172.16.255.20 200 1 10.0.12.20 12446 10.0.12.20 12446 lte up 0:00:00:22 1
To display the maximum number of control connections allowed on the router, use the show control local-properties command. The last column of the last line of the output lists the maximum controllers. The following is the abbreviated output for this command:
vEdge-1# show control local-properties
personality vedge
organization-name Viptela Inc
certificate-status Installed
root-ca-chain-status Installed
certificate-validity Valid
certificate-not-valid-before Mar 10 19:50:04 2016 GMT
certificate-not-valid-after Mar 10 19:50:04 2017 GMT
...
PUBLIC PUBLIC PRIVATE PRIVATE MAX
INDEX INTERFACE IP PORT IP PORT VSMARTS ... LAST CONNECTION CONTROLLERS
--------------------------------------------------------------------------------------------------------
0 ge0/2 10.0.5.11 12386 10.0.5.11 12386 2 ... 0:00:07:54 2
Two commands display information about the control connections established by the affinity configuration. To see, for each interface, which controller groups are configured and which the interface is connected to, use the show control affinity config command:
vEdge-1# show control affinity config
EFFECTIVE CONTROLLER LIST FORMAT - G(C),... - Where G is the Controller Group ID
C is the Required vSmart Count
CURRENT CONTROLLER LIST FORMAT - G(c)s,... - Where G is the Controller Group ID
c is the current vSmart count
s Status ✓ when matches, χ when does not match
EFFECTIVE
REQUIRED
INDEX INTERFACE VS COUNT EFFECTIVE CONTROLLER LIST CURRENT CONTROLLER LIST EQUILIBRIUM
------------------------------------------------------------------------------------------------------
0 ge0/2 2 1(2) 1(2)✓ Yes
The command output above shows that affinity is configured on interface ge0/2.
- The Effective Required vs. Count column shows that the interface is configured to create two control connections, and, in fact, two control connections have been established. You configure the number of control connections for the tunnel interface with the max-control-connections command.
- The Effective Controller List column shows that affinity on the interface is configured to use vSmart controller identifier 1 and that the router supports two OMP sessions. You configure the affinity controller identifiers with the controller-group-list command (at the system level) and, for the tunnel interface, the exclude-controller-group-list command.
- The Current Controller List column lists the actual affinity configuration for the interface. The output here shows that the interface has two control connections with vSmart controllers in group 1. The check mark indicates that the current and effective controller lists match each other. If, for example, the tunnel had established only one TLOC connection to a vSmart controller, this column would show "1(1)X".
- The Equilibrium column indicates that the current controller lists matches what is expected from the affinity configuration for that tunnel interface.
To determine the exact vSmart controllers that the tunnel interface has established control connections with, use the show control affinity status command:
vEdge-1# show control affinity status
ASSIGNED CONNECTED CONTROLLERS - System IP( G),.. - System IP of the assigned vSmart
G is the group ID to which the vSmart belongs to
UNASSIGNED CONNECTED CONTROLLERS - System IP( G),.. - System IP of the unassigned vSmart
G is the group ID to which the vSmart belongs to
INDEX INTERFACE ASSIGNED CONNECTED CONTROLLERS UNASSIGNED CONNECTED CONTROLLERS
------------------------------------------------------------------------------------------------------------------------
0 ge0/2 172.16.255.19( 1), 172.16.255.20( 1)
The command output above shows that interface ge0/2 has control connections to two vSmart controllers, 172.16.255.19 and 172.16.255.20, that both controllers are in group 1, and that both controllers are in one of the groups configured in the controller group list. If the interface were connected to a vSmart controller not in the controller group list, it would be listed in the Unassigned Connected Controllers column.
When a data center has multiple vSmart controllers, you can configure them to be in different controller groups. For example, if you configure them to be in two different controller groups, you could each vEdge router to establish two control connections, one to each of the groups. While this configuration design is similar to what we discussed in the previous section, providing redundant control connections to the vSmart controllers, on subtle difference is that it provides fault isolation between the two vSmart controller groups in the data center.The configuration for this scenario is almost identical to the configuration when vSmart controllers are two data centers. The only difference is that here, two vSmart controller groups are collocated in the same data center. See the configuration example in the next section.
Configure Affinity to vSmart Controllers in Two Data Centers
You can use affinity to enable redundancy among data centers, for a network design in which multiple vSmart controllers are spread across two or more data centers. Then, if the link between a vEdge router and one of the data centers goes down, the vSmart controllers in the second data center are available to continue servicing the overlay network. The figure below illustrates this scenario, showing three vSmart controllers in each of two data centers. Each of the three vEdge routers establishes a TLOC connection to one controller in the West data center and one in the East data center.
You configure the three vSmart controllers in DC1-West with controller group identifier 1:
vSmart-DC1(config)# system controller-group-id 1
The three vSmart controllers in DC2-East are in controller group 2:
vSmart-DC2(config)# system controller-group-id 2
We want all the vEdge routers to have a maximum of two OMP sessions, and we want each tunnel interface to have a maximum of two control connections and to not exclude any controller groups. So the only configuration that needs to be done on the vEdge routers is to set the controller group list. We want vEdge routers in the west to prefer vSmart controllers in DC1-West over DC2-East:
vEdge-West(config)# system controller-group-list 1 2
Similarly, we want vEdge routers in the east to prefer DC2-East:
vEdge-East(config)# system controller-group-list 2 1
The software evaluates the controller group list in order, so with this configuration, the vEdge-West routers prefer vSmart controller group 1 (which is the West data center), and the vEdge-East routers prefer vSmart controller group 2.
You can fine-tune the controller group preference in other ways:
- Set the maximum number of OMP sessions allowed on the router to 1 (system max-omp-sessions 1). To illustrate how this works, let's look at a vEdge-West router. The router has only one tunnel interface, and that interface creates one control connection to vSmart controller list 1. If all the vSmart controllers in this group become unavailable, or if the connection between the router that the DC1-West data center goes down, the tunnel interface establishes one control connection to vSmart controller list 2, because this group is listed in the system controller-group-list command. If all vSmart controllers in both controller groups, or the connections to them, become unavailable, and if the vBond orchestrator also indicates that all these vSmart controllers are unreachable, the tunnel interface establishes a control connection to any other vSmart controller in the overlay network if other controllers are present.
- Set the maximum number of control connections that the tunnel interface can establish to 1 (vpn 0 interface tunnel-interface max-control-connections 1). Because the software evaluates the controller group list in order, for a vEdge-West router, this configuration forces the tunnel interface to establish a control connection to vSmart controller group 1. Again, if this controller group or data center becomes unreachable, the tunnel establishes a control connection with controller group 2, because this group is configured in the system controller-group-list command. And if neither controller group 1 or 2 is available, and if another vSmart controller is present in the network, the tunnel interface establishes a control connection with that controller.
- Exclude the non-preferred vSmart controller group for a particular tunnel. For example, for a vEdge-West router to prefer controller group 1, you configure vpn 0 interface tunnel-interface exclude-controller-group-list 2. As with the above configurations, if this controller group or data center becomes unreachable, the tunnel establishes a control connection with controller group 2, because this group is configured in the system controller-group-list command. And if neither controller group 1 or 2 is available, and if another vSmart controller is present in the network, the tunnel interface establishes a control connection with that controller.
Configure Redundant Control Connections on One vEdge Router
When a router has two tunnel connections and the network has two (or more) data centers, you can configure redundant control connections from the vEdge router to vSmart controllers in two of the data centers. It is recommended that do this using the minimum number of OMP sessions—in this case, two. To do this, you configure one of the tunnel interfaces to go only to one of the data centers and the other to go only to the second. This configuration provides vSmart redundancy with the minimum number of OMP sessions.
On the vEdge router, define the controller group list and configure the maximum number of OMP sessions to be 2:
vEdge(config-system)# controller-group-list 1 2
vEdge(config-system)# max-omp-sessions 2
For one of the tunnels, you can use the default affinity configuration (that is, there is nothing to configure) to have this tunnel prefer a vSmart controller in group 1. You can also explicitly force this tunnel to prefer vSmart controller group 1:
vEdge(config-tunnel-interface-1)# max-control-connections 1
You do not need to configure exclude-controller-group-list 2, because the software evaluates the controller group list in order, starting with group 1. However, you could choose to explicitly exclude vSmart controller group 2.
Then, on the second tunnel, configure it to prefer a vSmart controller in group 2. As with the other tunnel, you limit the maximum number of control connections to 1. In addition, you have to exclude controller group 1 for this tunnel.
vEdge(config-tunnel-interface-2)# max-control-connections 1
vEdge(config-tunnel-interface-2)# exclude-controller-group-list 1
Additional Information
High Availability Overview
Using Affinity to Manage Network Scale