NV3458 BGP load sharing dual ISPs

[Anonymous]

I am trying to setup 2 ISP 10mb connections that will load balance all traffic inbound & outbound.   The router is not setup as a firewall just a BGP router.  It is working at this time but all traffic only goes to one ISP unless the connection goes down, then it will go to the other ISP.

[Anonymous]

Thank you for asking this question in the support community.  The information about load sharing when multihomed to multiple service providers can be found in the Configuring BGP in AOS guide, Example 4 on page 52.

AOS allows multiple equal cost routes to be used for the purposes of load sharing outbound traffic.

The need for load sharing is typically found in BGP applications where an AOS device is multihoming with multiple connections to different service providers. The BGP protocol does not provide support for load sharing. Therefore, BGP will always export the single best path for a given prefix to the IP route table. However, there are methods that can be implemented that will allow multiple BGP-derived routes to be imported into the IP route table. Aside from the BGP-specific configuration, load sharing must be globally enabled on the AOS device to allow the presence of multiple equal cost routes in the IP route table.

The example in the guide illustrates load sharing across multiple links where the customer’s router is multihomed to two different service providers. Each service provider is advertising a default route to the AOS device. The default routes contain equal BGP attributes, therefore one route is no more desirable than the other according to the BGP selection process. The objective is to ensure that outbound traffic from the customer’s network is load balanced (load shared) between the two Internet connections.

Several configuration steps are needed to allow BGP load sharing to take place. The ip load-sharing per-destination (for IPv4 routing) or the ipv6 load-sharing per-destination (for IPv6 routing) must be enabled in Global Configuration mode. This command allows duplicate routes to exist in the IPv4 or IPv6 routing table. The command maximum-paths 2 is issued in BGP AF Configuration mode to allow up to two equal cost routes from BGP to be exported to the routing table. In order for multiple BGP routes to the same destination to be candidates for load sharing, they must be equal cost and share the same AS number in the AS path attribute. Since each service provider in this example owns their own unique AS number that is added on to the BGP path attribute, an arbitrary AS number must be prepended to each eBGP neighbor’s BGP advertisement. This AS number enables the BGP routes to become candidates for load sharing since the routes now appear to originate from the same AS. This is accomplished with the route map (LOAD-SHARE) applied inbound to each eBGP neighbor. It matches the prefix list (DEFAULT) that only allows the default route, which automatically filters any other advertised routes. The route map then prepends the same arbitrary AS path (65009) to the default route prefix learned by each neighbor. The prefix list (PUBLIC-BLOCK) is used to advertise only the customer public block outbound to both service providers. This prefix will prevent any routes learned by the AOS device using BGP from one service provider from being advertised to the other service provider. If default routes are only learned from the service providers, then the potential of becoming a transit AS is not an issue. However, it is good practice to use outbound prefixes as a preventative measure for multihoming setups.

I have attached a sample configuration.  I hope this makes sense, but please do not hesitate to reply to this post with additional questions or information.  I will be happy to help in any way I can.

Levi

[Anonymous]

Levi,

I have setup the config as per the example and now have all incoming traffic on one ISP and all outgoing on the other ISP.  The main difference between the two is one has 3 hops configured and the other is one hop.

Here is my SHOW IP BGP output:

   Network             NextHop         Metric LocPrf      Path

*> 0.0.0.0/0           65.xxx.xxx.137  0                  209 i

   0.0.0.0/0           66.xxx.xxx.105                      33581 19108 i

*>o67.xxx.xxx.0/24     0.0.0.0                             i

and from SHOW RUN:

!

!

ip prefix-list PUBLIC-BLOCK seq 10 permit 67.xxx.xxx.0/24

ip prefix-list DEFAULT seq 10 permit 0.0.0.0/0

!

route-map LOAD-SHARE_Sxxxlink permit 10

  match ip address prefix-list DEFAULT

  set as-path prepend 33581

route-map LOAD-SHARE_CxxxLINK permit 10

  match ip address prefix-list DEFAULT

  set as-path prepend 33581 33581 33581

  set community 13xxxx04

!

!

router bgp 33581

  no auto-summary

  no synchronization

  bgp log-neighbor-changes

  maximum-paths 2

  network 67.xxx.xxx.0 mask 255.255.255.0

  neighbor 65.xxx.xxx.137

    no default-originate

    ebgp-multihop 3

    prefix-list PUBLIC-BLOCK out

    route-map LOAD-SHARE_CxxxLINK in

    send-community standard

    soft-reconfiguration inbound

    update-source vlan 67

    remote-as 209

  neighbor 66.xxx.xxx.105

    no default-originate

    prefix-list PUBLIC-BLOCK out

    route-map LOAD-SHARE_Sxxxlink in

    send-community standard

    soft-reconfiguration inbound

    update-source vlan 67

    remote-as 19108

!

Jim S.

[Anonymous]

Jim:

It appears you are on a firmware version prior to 18.03.01.00.  When you get a chance, will you reply with what firmware version the ADTRAN unit is running?  Also, one thing you will need to change in the route-maps' configuration is the set as-path prepend 33581 command.  You will need to change this to an arbitrary AS number, not the same as your AS.  For example, set as-path prepend x (where "x" represents an arbitrary number which is not your AS number, nor already in your AS path). 

After you make that change, you may need to clear the BGP process.  If you are on pre 18.03.01.00 firmware, which it appears you are, then the command will be clear ip bgp *.  If you are on 18.03.01.00 or above the command is clear bgp ipv4 *.

If this does not resolve the outbound load-sharing problem, please reply and attach the complete configuration, as well as the output from the show ip bgp / show bgp ipv4 command (depending what firmware version you are on).

Levi

[Anonymous]

Levi,

Changing the AS-PATH PREPEND fixed the traffic from the ISP so I see inbound traffic from both.     but it is still only send traffic out to one ISP.

Firmware is 18.02.01.00.E but I can upgrade to 18.03.01.00.E this weekend if that would help.

BGP-Internet#sh ip bgp
BGP local router ID is 216.xxx.xxx.235, local AS is 33581.
Status codes: * valid, > best, i - internal, o - local
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network             NextHop         Metric LocPrf      Path
*> 0.0.0.0/0           66.xxx.xxx.105                      133581 19108 i
*  0.0.0.0/0           65.xxx.xxx.137  0                  133581 133581 133581 209 i
*>o67.xxx,xxx.0/24     0.0.0.0                             i
Total RIB entries = 3

Is there anyway I can send you my config privately?

[Anonymous]

:

I sent you a personal message that you can reply to with the complete configuration.

Levi

[Anonymous]

:

Please change the route-map to the example below.  The first thing that should be changed is the AS-path should be the same, so BGP will see them as equal AS hop routes.

Levi

route-map LOAD-SHARE_CxxxLINK permit 10

  match ip address prefix-list DEFAULT

  set as-path prepend 33581

  set community 13xxxx04

[Anonymous]

Levi,

I changed the route-map to "set as-path prepend 133581". The only change
was the outbound traffic now goes out the CxxxLink connection.

I tried replying to your email but it comes back with a 553 -address unknow.  I used the recommended reply address show but it is the same as the one from the forum updates.

Thanks for your help!

Jim

[Anonymous · Accepted solution]

Levi replied:       With the current configuration, there must be something that is winning in the BGP selection process.  Therefore, something is not equal that we are receiving from the ISP's router.  Please, reply with this output.  Also, since you changed the route-map as-prepend have you cleared the BGP process yet?

Here is result screen.    I did a "clear ip bgp * " after i changed the route-map.  The traffic is still coming in from both ISPs but only going out one.

BGP-Internet#sh ip bgp
BGP local router ID is 216.xxx.xxx.235, local AS is 33581.
Status codes: * valid, > best, i - internal, o - local
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network             NextHop         Metric LocPrf      Path
*  0.0.0.0/0           66.xxx.xxx.xxx                       133581 19108 i
*> 0.0.0.0/0           65.xxx.xxx.xxx  0                  133581 209 i
*>o67.xxx.xxx.0/24     0.0.0.0                             i
Total RIB entries = 3
BGP-Internet#

Levi replied:    Thank you for replying with the requested information.  The problem appears to be with the MED.  The Internet Engineering Task Force (IETF) decision regarding BGP MED assigns a value of infinity to the missing MED, making the route that lacks the MED variable the least preferred.  Since one ISP is sending a MED of 0 and the other is sending "infinity" we might be able to adjust this in the inbound route-map from the ISP sending the "infinity" with the command set metric 0. 

Please, add that to the route-map and clear the BGP process.  Then reply again with the output from the show ip bgp command.

Levi: That fixed it!

BGP-Internet#sh ip bgp
BGP local router ID is 216.xxx.xxx.235, local AS is 33581.
Status codes: * valid, > best, i - internal, o - local
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network             NextHop         Metric LocPrf      Path
*> 0.0.0.0/0           66.xxx.xxx.xxx   0                  133581 19108 i
*> 0.0.0.0/0           65.xxx.xxx.xxx   0                  133581 209 i
*>o67.xxx.xxx0/24     0.0.0.0                             i
Total RIB entries = 3
BGP-Internet#

Thank You, Levi