Multi-VRF with NAT on NV3448 with Eth0/1 as WAN - Need same subnet on multiple internal VLANs

[Anonymous]

I have a NV3448 deployed in a small data center hosting scenario where I need to provide the same private subnet to multiple hosted customers.  The NV3448 provides firewalling and NAT svcs for the hosted virtual clients while keeping all clients seperated via VLAN interfaces throughout (ie. router, switch and virtual switch).  My attempts to implement Multi-VRF have been unsuccessful thus far.  My NAT rule doesn't seem to be working.

The basic configuration is NV3448 (ADTRAN, Inc. OS version R10.5.3.E) with VLAN 106 (Client A) and VLAN 206 (Client B) for two internal client interfaces.  Both clients are required to use the same private subnet of 10.1.0.0 /16.  Utilizing different Public IPs for NAT'd traffic.

1. Created vrf RED

vrf RED route-distinguisher as-2byte 2:2

2. Turned on ip firewall for vrf RED

ip firewall

ip firewall vrf RED

3. Created multiple VLAN interfaces

vlan 106

     name "ClientA"

!

vlan 206

     name "ClientB_VRF RED"

!

interface vlan 106

  description ClientA_NET

  ip address  10.1.1.45  255.255.255.0

  ip mtu 1500

  ip access-policy ClientAVMNet_VLAN106

  no shutdown

!

interface vlan 206

  description VRF_RED_ClientB_NET

  vrf forwarding RED

  ip address  10.1.1.45  255.255.0.0

  ip mtu 1500

  ip access-policy RED_ClientB_206

  no shutdown

4. Created ACLs

ip access-list extended ClientA_NAT

  remark Many:1 NAT ClientA_VM_Net

  permit ip 10.1.1.0 0.0.0.255  any

!

ip access-list extended RED_ClientB_NAT

  remark Many:1 NAT ClientB_VM_Net

  permit ip 10.1.0.0 0.0.255.255  any

5. Created ACPs

ip policy-class ClientA_106

  nat source list ClientA_NAT address x.x.x.1 overload

!

ip policy-class RED_ClientB_206

  nat source list RED_ClientB_NAT address x.x.x.2 overload

6. WAN Interface Config

interface eth 0/1

  ip address  x.y.y.202  255.255.255.252

ip address range  x.x.x.1  255.255.255.128  secondary

  ip access-policy WAN

  no shutdown

!

interface loop 2

  description RED_VRF_WAN

  vrf forwarding RED

  ip address  x.x.x.2  255.255.255.255

  ip mtu 1500

  {ip access-policy WAN_RED_VRF} - not currently added, but should add correct?

  no shutdown

7.  Default Routes

ip route 0.0.0.0 0.0.0.0 x.y.y.201

ip route vrf RED 0.0.0.0 0.0.0.0 loop 2

*****************************************************

I have tried various options to this configuration.  I've read through the Adtran "Configuring Multi-VRF in AOS" dated May 2013.  On the NAT, I tried it to the interface (Loop 2) and also to the IP Address...same results.  When done to the IP address, you have to add in the additional vrf RED classifier, but it still didn't make any difference.  I do not have the "ip firewall check reflexive-traffic" enabled, hence why I added the loop back interace in.

Testing results:

-From switch sourced on VLAN206, I'm able to ping NV3448 VLAN206 IP.

-I'm able to ping the Loop 2 interface when sourced with 10.1.1.45 by adding vrf RED addition.

-I'm NOT able to ping Loop 2 interface from the switch when sourced on VLAN 206, nor do I see any activity in the ip policy-session vrf RED

-I'm obviously not able to ping external public IPs when sourced from 10.1.1.45 on the vrf RED.

Command Syntax Used for testing:

ping 8.8.8.8 vrf RED source 10.1.1.45

show ip policy-sessions vrf RED

I ommited several configuration sections that I didn't think were pertinent the Multi-VRF configuration.  I'm seeking clarification on the following:

1. Why isn't my NAT statement working for the vrf RED network?

2. In addition to creating the interface loop 2 as shown above, is there anything else I need to do with it?

3. When do you put the loop 2 interface in the vrf and when would you not need to?

4. If I'm not seeing the ping traffic on the vrf RED policy sessions then where is it going when I ping the Loop 2 address from the switch?

Thank you in advance for your comments and suggestions.  I've also attached a visio diagram for clarity.

[Anonymous]

-

Thank you for asking this question in the support community.  I noticed you also opened a ticket with ADTRAN Technical Support.  When you get a resolution, will you post the results on the forum so other users can benefit?

Thanks,

Noor

[Anonymous]

1. I believe you should be using the vrf classifier on your NAT statement, but I also believe you need to allow interface traffic in the policy first, then apply the NAT statement. We commonly use NAT inside VRF's on NV3448 and each one is setup in this manner.  Remember with firewalls the rule of thumb is to keep everything out, which includes keeping traffic on the LAN from reaching other networks.  You basically want to allow LAN traffic to reach anything.

ip access-list extended self

  remark Traffic to Total Access

  permit ip any  any     log

ip policy-class RED_ClientB_206

  allow list self self

  nat source list RED_ClientB_NAT address x.x.x.2 vrf RED overload

2. You should not need to put a policy on the loopback interface.  Policies should apply to the interfaces that will actually pass the traffic.

3.  You are correct to put loop 2 in VRF RED if that is the IP you are attempting to use for outbound NAT.  Your outbound default route lives in the default VRF though and herein lies the problem.  Ultimately what you want to do is get traffic to follow the default route in the default VRF, but the traffic in VRF RED is basically stuck there.   I don't think the 3448 can be configured with a "global" route table that can redistribute routes between VRF's, so this becomes problematic and things can get a little complicated.

What I have done to get around this issue is use the eth 0/2 port as another interface in the network for VRF RED but leave it in the default VRF.  Then cable a switch port in VLAN206 to eth 0/2 directly (or through a local switch) and point your VRF RED default route to the IP on eth 0/2.  You could also make this a dot1q port and trunk multiple VLANs, thus allowing you to use this for multiple VRF's.  The major caveat here is that you need to use different subnets, because the 3448 will balk at having IPs from the same subnet range assigned to VLAN106 (default VRF) and eth0/2 (default VRF).  Also, your outbound NAT statement will actually stay the same as you have it now, so you can ignore the first post (except for adding the "self" list - that is necessary regardless) because now the NAT will take place in the default VRF and the loopback doesn't need to be in the VRF either.

vlan 206

  name "vrf_red"

!

interface loop 2

  description RED_VRF_WAN

  ip address  x.x.x.2  255.255.255.255

!

interface eth 0/2

encapsulation 802.1q

!

interface eth 0/2.206

  description Use for Default route - connects from sw0/1

  ip address  10.1.1.1 255.255.0.0

  vlan-id 206

  ip access-policy RED_ClientB_206

!

interface switchport 0/1

  desc Connects to Eth0/2

  switchport mode trunk

  switchport trunk allowed vlan 206

!

ip policy-class RED_ClientB_206

allow list self self

nat source list RED_ClientB_NAT address x.x.x.2 overload

!

ip route 0.0.0.0 0.0.0.0 x.y.y.201

ip route vrf RED 0.0.0.0 0.0.0.0 10.1.1.1

Note you do not need a policy on the VLAN interface this time because it will only be forwarding its traffic to an IP within its subnet which shouldn't be a problem for the firewall.  If you do have an issue, though, just use an access policy with only the allow list self applied.

4.  Applying the "self" list to the policy should fix that issue.

Are your 3448 switchports setup as access ports or trunk ports to the separate customer networks?  If you are using the switchports as access ports, you may also have issues using the same subnets because the switchports do not recognize the VRF's and you could have issues with broadcast traffic.

I also recommend setting up each customer in their own VRF and leaving the default VRF customer network free.  This will give you a little more control over the specific customer networks without having to worry about affecting the other customers.

[Anonymous]

:

I marked this post as "assumed answered," but reply with the outcome of this post when you get a chance.

Levi

[Anonymous]

Sorry that it took a while to respond back.  Yes, the response helped in addition to clarification from Adtran Support.  My final solution was as follows:

vrf RED route-distinguisher as-2byte 2:2

!

ip firewall vrf RED

ip firewall check reflexive-traffic

!

vlan 206

  name "VRF_RED"

!

interface eth 0/1

  description WANConn

  ip address  w1.x1.y1.z1  255.255.255.252

  ip address range  w2.x2.y2.z129  w2.x2.y2.z254  255.255.255.128  secondary

  ip access-policy WAN

!

interface vlan 206

  description VRF_RED

  vrf forwarding RED

  ip address  10.1.1.45  255.255.0.0

  ip access-policy RED_VLAN_206

!

ip policy-class RED_VLAN_206

  nat source list RED_VLAN206Net_NAT address w2.x2.y2.z254 overload

!

ip policy-class WAN

  nat destination list RED_VLAN206_PrtFwd address 10.1.1.200 vrf RED

!

ip route vrf RED 0.0.0.0 0.0.0.0 10.1.255.254

!

**********************************************************************************

Things to note:

1. the IP in the VRF Route (10.1.255.254) doesn't live anywhere on the router.  For reasons I can't clearly explain, it kind of tricks the router in routing the traffic from the VRF firewall down to the default VRF firewall so the NAT can work properly.

2.  I also found that is you use the GUI then the VRF rules don't show up in the security zones however they are there.  Since the only practical way to reprioritize rules is in the GUI, I found that if you move the default VRF rules in security zones then they non-default rules flow in suite.  I realize this is probably clear as mud, basically move the rule above or below where the non-default VRF rule would be in the GUI and it will reprioritize the non-default vrf rules in sequence as if they were showing.