I have a NV3448 deployed in a small data center hosting scenario where I need to provide the same private subnet to multiple hosted customers. The NV3448 provides firewalling and NAT svcs for the hosted virtual clients while keeping all clients seperated via VLAN interfaces throughout (ie. router, switch and virtual switch). My attempts to implement Multi-VRF have been unsuccessful thus far. My NAT rule doesn't seem to be working.
The basic configuration is NV3448 (ADTRAN, Inc. OS version R10.5.3.E) with VLAN 106 (Client A) and VLAN 206 (Client B) for two internal client interfaces. Both clients are required to use the same private subnet of 10.1.0.0 /16. Utilizing different Public IPs for NAT'd traffic.
1. Created vrf RED
vrf RED route-distinguisher as-2byte 2:2
2. Turned on ip firewall for vrf RED
ip firewall
ip firewall vrf RED
3. Created multiple VLAN interfaces
vlan 106
name "ClientA"
!
vlan 206
name "ClientB_VRF RED"
!
interface vlan 106
description ClientA_NET
ip address 10.1.1.45 255.255.255.0
ip mtu 1500
ip access-policy ClientAVMNet_VLAN106
no shutdown
!
interface vlan 206
description VRF_RED_ClientB_NET
vrf forwarding RED
ip address 10.1.1.45 255.255.0.0
ip mtu 1500
ip access-policy RED_ClientB_206
no shutdown
4. Created ACLs
ip access-list extended ClientA_NAT
remark Many:1 NAT ClientA_VM_Net
permit ip 10.1.1.0 0.0.0.255 any
!
ip access-list extended RED_ClientB_NAT
remark Many:1 NAT ClientB_VM_Net
permit ip 10.1.0.0 0.0.255.255 any
5. Created ACPs
ip policy-class ClientA_106
nat source list ClientA_NAT address x.x.x.1 overload
!
ip policy-class RED_ClientB_206
nat source list RED_ClientB_NAT address x.x.x.2 overload
6. WAN Interface Config
interface eth 0/1
ip address x.y.y.202 255.255.255.252
ip address range x.x.x.1 255.255.255.128 secondary
ip access-policy WAN
no shutdown
!
interface loop 2
description RED_VRF_WAN
vrf forwarding RED
ip address x.x.x.2 255.255.255.255
ip mtu 1500
{ip access-policy WAN_RED_VRF} - not currently added, but should add correct?
no shutdown
7. Default Routes
ip route 0.0.0.0 0.0.0.0 x.y.y.201
ip route vrf RED 0.0.0.0 0.0.0.0 loop 2
*****************************************************
I have tried various options to this configuration. I've read through the Adtran "Configuring Multi-VRF in AOS" dated May 2013. On the NAT, I tried it to the interface (Loop 2) and also to the IP Address...same results. When done to the IP address, you have to add in the additional vrf RED classifier, but it still didn't make any difference. I do not have the "ip firewall check reflexive-traffic" enabled, hence why I added the loop back interace in.
Testing results:
-From switch sourced on VLAN206, I'm able to ping NV3448 VLAN206 IP.
-I'm able to ping the Loop 2 interface when sourced with 10.1.1.45 by adding vrf RED addition.
-I'm NOT able to ping Loop 2 interface from the switch when sourced on VLAN 206, nor do I see any activity in the ip policy-session vrf RED
-I'm obviously not able to ping external public IPs when sourced from 10.1.1.45 on the vrf RED.
Command Syntax Used for testing:
ping 8.8.8.8 vrf RED source 10.1.1.45
show ip policy-sessions vrf RED
I ommited several configuration sections that I didn't think were pertinent the Multi-VRF configuration. I'm seeking clarification on the following:
1. Why isn't my NAT statement working for the vrf RED network?
2. In addition to creating the interface loop 2 as shown above, is there anything else I need to do with it?
3. When do you put the loop 2 interface in the vrf and when would you not need to?
4. If I'm not seeing the ping traffic on the vrf RED policy sessions then where is it going when I ping the Loop 2 address from the switch?
Thank you in advance for your comments and suggestions. I've also attached a visio diagram for clarity.
1. I believe you should be using the vrf classifier on your NAT statement, but I also believe you need to allow interface traffic in the policy first, then apply the NAT statement. We commonly use NAT inside VRF's on NV3448 and each one is setup in this manner. Remember with firewalls the rule of thumb is to keep everything out, which includes keeping traffic on the LAN from reaching other networks. You basically want to allow LAN traffic to reach anything.
ip access-list extended self
remark Traffic to Total Access
permit ip any any log
ip policy-class RED_ClientB_206
allow list self self
nat source list RED_ClientB_NAT address x.x.x.2 vrf RED overload
2. You should not need to put a policy on the loopback interface. Policies should apply to the interfaces that will actually pass the traffic.
3. You are correct to put loop 2 in VRF RED if that is the IP you are attempting to use for outbound NAT. Your outbound default route lives in the default VRF though and herein lies the problem. Ultimately what you want to do is get traffic to follow the default route in the default VRF, but the traffic in VRF RED is basically stuck there. I don't think the 3448 can be configured with a "global" route table that can redistribute routes between VRF's, so this becomes problematic and things can get a little complicated.
What I have done to get around this issue is use the eth 0/2 port as another interface in the network for VRF RED but leave it in the default VRF. Then cable a switch port in VLAN206 to eth 0/2 directly (or through a local switch) and point your VRF RED default route to the IP on eth 0/2. You could also make this a dot1q port and trunk multiple VLANs, thus allowing you to use this for multiple VRF's. The major caveat here is that you need to use different subnets, because the 3448 will balk at having IPs from the same subnet range assigned to VLAN106 (default VRF) and eth0/2 (default VRF). Also, your outbound NAT statement will actually stay the same as you have it now, so you can ignore the first post (except for adding the "self" list - that is necessary regardless) because now the NAT will take place in the default VRF and the loopback doesn't need to be in the VRF either.
vlan 206
name "vrf_red"
!
interface loop 2
description RED_VRF_WAN
ip address x.x.x.2 255.255.255.255
!
interface eth 0/2
encapsulation 802.1q
!
interface eth 0/2.206
description Use for Default route - connects from sw0/1
ip address 10.1.1.1 255.255.0.0
vlan-id 206
ip access-policy RED_ClientB_206
!
interface switchport 0/1
desc Connects to Eth0/2
switchport mode trunk
switchport trunk allowed vlan 206
!
ip policy-class RED_ClientB_206
allow list self self
nat source list RED_ClientB_NAT address x.x.x.2 overload
!
ip route 0.0.0.0 0.0.0.0 x.y.y.201
ip route vrf RED 0.0.0.0 0.0.0.0 10.1.1.1
Note you do not need a policy on the VLAN interface this time because it will only be forwarding its traffic to an IP within its subnet which shouldn't be a problem for the firewall. If you do have an issue, though, just use an access policy with only the allow list self applied.
4. Applying the "self" list to the policy should fix that issue.
Are your 3448 switchports setup as access ports or trunk ports to the separate customer networks? If you are using the switchports as access ports, you may also have issues using the same subnets because the switchports do not recognize the VRF's and you could have issues with broadcast traffic.
I also recommend setting up each customer in their own VRF and leaving the default VRF customer network free. This will give you a little more control over the specific customer networks without having to worry about affecting the other customers.
Sorry that it took a while to respond back. Yes, the response helped in addition to clarification from Adtran Support. My final solution was as follows:
vrf RED route-distinguisher as-2byte 2:2
!
ip firewall vrf RED
ip firewall check reflexive-traffic
!
vlan 206
name "VRF_RED"
!
interface eth 0/1
description WANConn
ip address w1.x1.y1.z1 255.255.255.252
ip address range w2.x2.y2.z129 w2.x2.y2.z254 255.255.255.128 secondary
ip access-policy WAN
!
interface vlan 206
description VRF_RED
vrf forwarding RED
ip address 10.1.1.45 255.255.0.0
ip access-policy RED_VLAN_206
!
ip policy-class RED_VLAN_206
nat source list RED_VLAN206Net_NAT address w2.x2.y2.z254 overload
!
ip policy-class WAN
nat destination list RED_VLAN206_PrtFwd address 10.1.1.200 vrf RED
!
ip route vrf RED 0.0.0.0 0.0.0.0 10.1.255.254
!
**********************************************************************************
Things to note:
1. the IP in the VRF Route (10.1.255.254) doesn't live anywhere on the router. For reasons I can't clearly explain, it kind of tricks the router in routing the traffic from the VRF firewall down to the default VRF firewall so the NAT can work properly.
2. I also found that is you use the GUI then the VRF rules don't show up in the security zones however they are there. Since the only practical way to reprioritize rules is in the GUI, I found that if you move the default VRF rules in security zones then they non-default rules flow in suite. I realize this is probably clear as mud, basically move the rule above or below where the non-default VRF rule would be in the GUI and it will reprioritize the non-default vrf rules in sequence as if they were showing.