Routing Between VLANs After Upgrade

[bdsmith2]

Hello,

We just went through an equipment refresh replacing our older generation NV1238P with some a pair of new 1638Ps last week in addition to adding several VLANs.  Since we are a telecommunications dealer that sells multiple phone systems, we need to be able to quick jump from system to system with minimal impact on daily operation to troubleshoot different issues for each respective system.  As a result, we went from having 2 VLANs to having 8 VLANs on our network.  Each VLAN has it's own /24 block... VLAN 14 is 172.16.14.xxx, VLAN 16 is 172.16.16.xxx, VLAN 17 is 172.16.17.xxx and so on up through VLAN 23.  Before the refresh we had systems on VLAN 14 and VLAN 16 (main LAN) and the inter-vlan routing was working very well.  However, now that we have made these changes, we seem to be having issues.

In addition to the 1638Ps (SW01 and SW02), we have four 1531Ps (SW03-SW06) placed at key locations around the office for the users/technicians that need access to just about every system.  All ports on the endpoint 1638 (SW01) are set for trunk > allowed all VLANs with a trunk uplink between the pair of switches also set with the same parameters. On the switch that handles the servers and other telephony equipment (SW02) each port is setup to run it's appropriate VLAN. For a router we have a 4430 and it handles DHCP for each respective VLAN and has a sub-interface named giga 0/2.14, or 0/2.16 etc etc. There are no ACL's limiting traffic between VLANs.  When I first implemented the 1638s into the network I tried to use the Layer 3 functionality of the switches to route between VLANs by assigning each VLAN an IP address.  That did not work as I had understood that it would so I did away with using the 1638 for VLAN routing and handed the buck off to the 4430. 

If a few excerpts of key areas within my config are needed, I will gladly supply them.  Any ideas would be appreciated, thanks!

[jayh]

Specifically what issues are you having?

It sounds like you are doing all layer 3 routing in the 4430. If this is the case, turn off IP routing in the switches.

Determine if the problem is layer 2 or layer 3.

Are DHCP devices getting addresses, and if so are they within the correct subnet for the VLAN?

Can devices ping their own on-net gateway?

Are the MAC addresses of hosts in the ARP table of the 4430, on the correct subinterface and VLAN?

[bdsmith2]

Primarily the main VLAN (#16) could not access any servers or endpoints on the other VLANs  other than VLAN 14 (implemented back in September, however VLAN 19 was my first "test" VLAN), but each other VLAN could access VLAN 16. 

All Layer 3 was being handled by the 4430 and IP routing was disabled on the 1638s, but I'm in the process of moving that to the 1638's to alleviate some of the bottle neck on Gig 0/2 of the 4430.

DHCP is working properly and addresses are being handed out as they should for each VLAN. All devices can ping the sub-int gateway on the 4430 which in this case is 172.16.(vlan#).2...but could not get routed back to the hosts.

I played with it a little last night and ended up making an ACL with a summary that looked like this to make it work:

ip access-list extended web-acl-30 
remark Inter-VLAN Routing 
permit ip 172.16.16.0 0.0.0.255  172.16.0.0 0.0.31.255     log 
permit ip 172.16.0.0 0.0.31.255  172.16.16.0 0.0.0.255     log

What sort of bothered me is why that ACL was not needed when i first implemented 2 VLANs back in September, but was needed now. 

I would like to leverage the power of the 1638's instead of using L3 vlan routing on the 4430.  From my understanding I need to:

1) do away with my sub-interfaces on my 4430 that are in place for router-on-a-stick and revert back to primary vlan address (VLAN 16)

2) assign an IP to each VLAN interface on the 1638s and enable IP routing once again

3) create a static route on the 4430 to point each VLAN back to the 1638.  next hop address should be the interface trunking from the last 1638 to the 4430.

4) ensure the default gateway is correct in each 1638.

Is there anything else I may need to do?

[jayh]

bdsmith2 wrote:

I played with it a little last night and ended up making an ACL with a summary that looked like this to make it work:

ip access-list extended web-acl-30 
remark Inter-VLAN Routing 
permit ip 172.16.16.0 0.0.0.255 172.16.0.0 0.0.31.255 log 
permit ip 172.16.0.0 0.0.31.255 172.16.16.0 0.0.0.255 log

What sort of bothered me is why that ACL was not needed when i first implemented 2 VLANs back in September, but was needed now.

I would think that it would be needed. Is the 4430 also doing NAT to the Internet? If so you may need a bit more. I'd use a single ACL to make things easier.

ip access-list extended private-out-list
remark VLANs to each other and to Internet
permit ip 172.16.0.0 0.0.31.255 172.16.0.0 0.0.31.255
permit ip 172.16.0.0 0.0.31.255 any policy Public

I would like to leverage the power of the 1638's instead of using L3 vlan routing on the 4430. From my understanding I need to:

1) do away with my sub-interfaces on my 4430 that are in place for router-on-a-stick and revert back to primary vlan address (VLAN 16)

2) assign an IP to each VLAN interface on the 1638s and enable IP routing once again

3) create a static route on the 4430 to point each VLAN back to the 1638. next hop address should be the interface trunking from the last 1638 to the 4430.

4) ensure the default gateway is correct in each 1638.

Is there anything else I may need to do?

Unless that interface on the 4430 is really slammed, I would leave it as-is. If you really want to move the inter-VLAN stuff to one of the 1638s, basically mirror the interface and access-policy configuration from the 4430 to the 1638 then shut down the sub-interfaces on the 4430. Your hosts are going to want a single gateway so I'd do it on one 1638 and leave the other layer-2. You could, for the fun of it, go with VRRP but probably not worth the effort. The 1638 should get all of your VLANs talking with the gateway of each VLAN anchored to the VLAN interface on the 1638 acting as a router. Build your dhcp pools on the 1638 or on a server.

Then create a new VLAN between the 1638 and the 4430, it can be a /30 with an access port on the 1638. Point the 0.0.0.0/0 route on the 1638 to the 4430 on this VLAN. Create a static route on the 4430 for 172.16.0.0/19 back to the 1638 end of the /30 new VLAN and a single NAT statement on the 4430 for the whole /19 going to its public interface. Your Internet firewall and NAT now reside on the 4430 and all of the LAN routing on the 1638.

[bdsmith2]

The 4430 is doing NAT, yes... but the previous admin has it setup as "any any" in a nat list so all subnets are covered.

The 4430 is acting as a firewall, DHCP server for secondary VLANs (all but the 16 VLAN which is - for now - being handled by a Server 2003 machine but it is getting decommissioned once we move everything off to other servers) and handles all VPN traffic (we have about a dozen VPNs in place for various systems).

I made a few changes between posts and here is how I currently have it setup. Keep in mind this is not a final solution as we are still operating on VLAN 16 (primary) until the end of the business day so I haven't touched it.  All the below changes are in place on secondary VLANs.

NV1638P-SW01 (endpoints - mostly VLAN 16 with some exceptions)

     Each VLAN has an interface address of .1

     All ports trunk, allowed all VLANs, native VLAN 16

     Default gateway is 4430 address on VLAN 16

     IP routing is disabled for now...was going to enable it after 5 just in case it breaks the main office when I'm an hour away

     Port 48 (uplink to SW02) is trunk, allowed all VLANs, native VLAN 16

NV1638P-SW02 (servers/vlan routing)

     Each VLAN has an interface address of .2

     Each in-use port is on it's appropriate VLAN

     Default gateway is 4430 address on VLAN 16

     IP routing is disabled for now...was going to enable it after 5 just in case it breaks the main office when I'm an hour away

     Port 47 (uplink to SW01) is trunk, allowed all VLANs, native VLAN 16

     Port 48 (uplink to 4430) will be set to access on primary VLAN after 5, for now it is still trunk, allowed all, native 16

NV4430-RTR

     Handling firewall

     Handling DHCP for secondary VLANs

               DHCP scope for hosts is set to use .2 for a gateway on each VLAN

     Terminating VPNs

     Static routes

               Each VLAN has it's own route pointing from the 4430 to the SW02 interface for each VLAN as next hop (this was advised by Adtran)

     Sub-Interfaces for each secondary VLAN leftover from yesterday are disabled pending removal after 5pm today

     Summary ACL was removed to "break" inter-vlan routing only as it wasn't in place until late last night, but somehow it broke internet connectivity to the entire site today so I put it back 😕

We have a satellite office in the VLAN 15 connected over VPN so I don't think I can use the summary without breaking inter-site connectivity. I will change it later when I have a chance to between installs, projects and service  calls.  It's been an ongoing project since I started 2 years ago.

[jayh]

If the .1 address on NV1638P-SW01 is the gateway address for the hosts on each VLAN, then there's no need for IP routing on NV1638P-SW02. I would put the DHCP server for the VLANs on NV1638P-SW01, create a new VLAN between SW01 and the 4430, and route the whole aggregate 172.16/19 over it. Continue to terminate the VPN tunnels and the like on the 4430 but no need to backhaul each individual /24 to it.

The NV1638P-SW01 would handle all of your LAN routing, DHCP, etc. The 4430 would be your public-facing router, VPN termination, etc. The NV1638P-SW02 just becomes a layer 2 expansion switch trunked to NV1638P-SW01.

When you have your DHCP server in one place and your routing in another it makes troubleshooting more complex. This design separates the functionality.

[bdsmith2]

I think I may have got it working now.  Between your suggestions, suggestions from Adtran and trial/error I have VLAN routing working and have a separate /30 between the 1638 and the 4430.  For now, Internet and VLAN routing are working properly. Here is how it is setup now:

NV4430-RTR

IP address is now 172.16.11.1

DHCP has been offloaded

VLAN routing has been offloaded

Functioning as Firewall, Gateway to ISP and VPN termination device

Static routes for each VLAN are specified back to the /30 address of connecting 1638 until I can move our satellite office out of the middle of the /19 range.  Otherwise I get a TTL expiration when trying to access that office over VPN

NV1638P-SW01 (used to be SW02)

VLAN 16 (Data) IP is 172.16.16.2

Switch Gateway is 172.16.11.1

VLAN 2 (Internet) IP is 172.16.11.2

IP routing is ENABLED

DHCP is being served by this unit for all VLANs excluding VLAN 16 (until decommissioning of the server)

Port 47 is trunked up to SW02 serving all VLANs

port 48 is in ACCESS MODE on VLAN 2 (Internet) connected to the 4430-RTR

NV1638P-SW02 (used to be SW01)

IP address is still 172.16.16.254 -- For now...Once the existing DHCP/AD server is decommissioned, it will assume the address of .3

Gateway is configured to be 172.16.16.2 (SW01)

IP routing is DISABLED and funtioning as L2 only

Port 48 is trunked up to SW01 serving all VLANs

When I do a trace route to another VLAN from within VLAN 16, the results are (using VLAN 14 and 19 as examples):

C:\Users\Ben Smith>tracert 172.16.14.230

Tracing route to [172.16.14.230]

over a maximum of 30 hops:

  1    <1 ms     2 ms    <1 ms  172.16.16.2

  2    <1 ms    <1 ms    <1 ms  [172.16.14.230]

Trace complete.

C:\Users\Ben Smith>tracert 172.16.19.5

Tracing route to 172.16.19.5 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  172.16.16.2

  2    <1 ms    <1 ms    <1 ms  172.16.19.5

Trace complete.

It appears my information is passing from the PC on VLAN 16, through SW02 over to SW01, and finally to the appropriate host on SW01.

Going out to the internet it goes from PC on VLAN 16 > SW02 > SW01 > 4430 > Out the ISP

After my original comment, I noticed I had routing issues from the secondary VLANs to the Internet until I put in place the /30 you recommended.

Thanks for all of your help! This has been bothering me for days and I think now I can finally get a full night's rest now that it's working.  I'm still working on my journey through IT. Have only been in the field with basic certs and and Associates for 2 years now and since I don't deal in only IT but rather just about anything that has electricity running through it, I don't get as much time in the configs as I'd like to have.  The saying is true...if you don't use it, you might just lose it.

Thanks, and have a great rest of your week!

[jayh]

bdsmith2 wrote:

I think I may have got it working now. Between your suggestions, suggestions from Adtran and trial/error I have VLAN routing working and have a separate /30 between the 1638 and the 4430. For now, Internet and VLAN routing are working properly. Here is how it is setup now:

NV4430-RTR

IP address is now 172.16.11.1

DHCP has been offloaded

VLAN routing has been offloaded

Functioning as Firewall, Gateway to ISP and VPN termination device

Static routes for each VLAN are specified back to the /30 address of connecting 1638 until I can move our satellite office out of the middle of the /19 range. Otherwise I get a TTL expiration when trying to access that office over VPN

For your VPN, the most specific match wins so you should be able to make this work with your existing addressing. On the 4430, try adding a static route for the /24 of the satellite office pointed to the public IP of the remote VPN peer (or to your Internet gateway). This should send traffic to the Public interface where the crypto map will grab it and do its thing.

[bdsmith2]

I did that a couple of days ago.  Created a static route for the 172.16.15.xxx network with a route pointing to the remote peer but It was still giving me a TTL, so not too sure what was going on.  That satellite site is small enough and has only a handful of statically assigned devices, so I talked to the President of the company today and he said go ahead and change it whenever.  I would like to let this config ride for a week or 2 to make sure we don't run into any issues before moving that subnet to an entirely different address like 172.17.xxx.xxx so it would be moved out of my corporate VLAN range. That will come for another day. For now, it's working and working well so I'll monitor for a little while.