Hi all,
We're scratching our head. Customer added a new building where we've installed four Netvanta 1550P switches. We pre-configured and deployed, and all are working like a champ except for admin via LAN (http, ssh, etc).
When these four switches are disconnected from the rest of the customer's network, all admin works exactly as expected. However, when we connect to the rest of customer's network, LAN admin simply stops working. Note that all OTHER network devices in this new building remain accessible via http, as do all other non-Adtran devices in their "old" buildings. Connection to "old" building is via fiber with yet another Adtran 1550P at far end. We cannot access that switch either when rest of network is connected. Firmware on all switches is 12.3.4. Serial admin functions fine at all times. We have addresses assigned to both VLANs, no joy accessing either with rest of network connected. Customer has a third-party company managing their core network, we haven't gone too far down that road as all other devices appear to be normally accessible - only the Netvanta switches are negatively affected. We have a fairly straight-forward config - all switches use essentially the same config, with different IP addresses:
hostname "BldgC Data cabinet B"
enable password XXXX
!
clock timezone -5-Eastern-Time
!
ip subnet-zero
ip classless
ip default-gateway 172.16.0.254
ip routing
name-server 75.75.75.75 8.8.8.8
!
ip route-cache express
!
no auto-config
!
event-history on
no logging forwarding
no logging email
!
no service password-encryption
!
username "XXXX" password "YYYY"
!
!
no dot11ap access-point-control
no dos-protection
no desktop-auditing dhcp
no network-forensics ip dhcp
!
vlan 1
name "Default"
!
vlan 18
name "Public Network"
!
interface gigabit-switchport 0/1
no shutdown
switchport access vlan 18
!
interface gigabit-switchport 0/2
no shutdown
switchport access vlan 18
!
interface gigabit-switchport 0/3
no shutdown
!
interface gigabit-switchport 0/4
no shutdown
!
interface gigabit-switchport 0/5
no shutdown
!
interface gigabit-switchport 0/6
no shutdown
!
interface gigabit-switchport 0/7
no shutdown
!
interface gigabit-switchport 0/8
no shutdown
!
interface gigabit-switchport 0/9
no shutdown
!
interface gigabit-switchport 0/10
no shutdown
!
interface gigabit-switchport 0/11
no shutdown
!
interface gigabit-switchport 0/12
no shutdown
!
interface gigabit-switchport 0/13
no shutdown
!
interface gigabit-switchport 0/14
no shutdown
!
interface gigabit-switchport 0/15
no shutdown
!
interface gigabit-switchport 0/16
no shutdown
!
interface gigabit-switchport 0/17
no shutdown
!
interface gigabit-switchport 0/18
no shutdown
!
interface gigabit-switchport 0/19
no shutdown
!
interface gigabit-switchport 0/20
no shutdown
!
interface gigabit-switchport 0/21
no shutdown
!
interface gigabit-switchport 0/22
no shutdown
!
interface gigabit-switchport 0/23
no shutdown
switchport mode trunk
!
interface gigabit-switchport 0/24
no shutdown
switchport mode trunk
!
interface xgigabit-switchport 0/1
no shutdown
switchport mode trunk
speed auto
!
interface xgigabit-switchport 0/2
no shutdown
switchport mode trunk
speed auto
!
interface xgigabit-switchport 0/3
no shutdown
switchport mode trunk
speed auto
!
interface xgigabit-switchport 0/4
no shutdown
switchport mode trunk
speed auto
!
interface vlan 1
ip address 172.16.0.142 255.255.0.0
no awcp
no shutdown
!
interface vlan 18
ip address 172.18.0.62 255.255.0.0
no awcp
no shutdown
!
no tftp server
no tftp server overwrite
http server
http secure-server
no snmp agent
no ip ftp server
no ip scp server
no ip sntp server
!
snmp-server engine local 8000029803000000000000
!
line con 0
no login
!
line telnet 0 4
login
password XXXX
no shutdown
line ssh 0 4
login local-userlist
no shutdown
!
sntp server us.pool.ntp.org
!
end
I'd be thrilled if anyone can point out our boneheaded mistake. Thanks
This sounds like an issue where the CPU is handling network traffic and not responding to management interfaces ( as it is designed to do network traffic first priority).
switch# sh proc cpu will show what processes that are running. And switch# sh proc cpu hist will show the CPU load over time.
This will sometimes give you an idea of the type of network traffic. It could be as simple as a very high level of Multicast or Broadcast traffic.
Also the VLAN interfaces do not have -
ip route-cache express
This should not matter if they are not routing any traffic though.
To troubleshoot further, please provide this info to support.
This sounds like an issue where the CPU is handling network traffic and not responding to management interfaces ( as it is designed to do network traffic first priority).
switch# sh proc cpu will show what processes that are running. And switch# sh proc cpu hist will show the CPU load over time.
This will sometimes give you an idea of the type of network traffic. It could be as simple as a very high level of Multicast or Broadcast traffic.
Also the VLAN interfaces do not have -
ip route-cache express
This should not matter if they are not routing any traffic though.
To troubleshoot further, please provide this info to support.
While network traffic is possible, when we deployed there was virtually no traffic, hence the switch CPUs should have had ample time to respond. One of these switches has only three or four devices connected - so I'm thinking that there is something else at play here. I've turned off the ip route-cache express with no joy.
Cheers!
Is it your intent that this switch act as a layer 3 router between VLAN 1 and VLAN 18? If not, disable IP routing or remove the IP address from the VLAN interface you don't want to route.
From what IP address or VLAN are you trying to access the devices?
I've disabled IP routing and greatly dumbed down the config for diagnostics. Current version below.
In testing we have:
At this point, I'm fairly certain that it is something about the customer's network - but I"m at a loss to what may be wrong. I'd welcome suggestions on a valid test. We can insert a network monitor somewhere and capture packets for wireshark analysis.
Current config:
!
!
! ADTRAN, Inc. OS version R12.3.4
! Boot ROM version BVS1.0
! Platform: NetVanta 1550-24P, part number 17101524PF1
! Serial number XXXX
!
!
hostname "Switch"
enable password YYYY
!
!
!
ip subnet-zero
ip classless
no ip routing
!
!
no ip route-cache express
!
no auto-config
!
event-history on
no logging forwarding
no logging email
!
no service password-encryption
!
username "xxxx" password "yyyy"
!
!
!
!
!
!
no dot11ap access-point-control
no dos-protection
no desktop-auditing dhcp
no network-forensics ip dhcp
!
vlan 1
name "Default"
!
interface gigabit-switchport 0/1
no shutdown
!
interface gigabit-switchport 0/2
no shutdown
!
interface gigabit-switchport 0/3
no shutdown
!
interface gigabit-switchport 0/4
no shutdown
!
interface gigabit-switchport 0/5
no shutdown
!
interface gigabit-switchport 0/6
no shutdown
!
interface gigabit-switchport 0/7
no shutdown
!
interface gigabit-switchport 0/8
no shutdown
!
interface gigabit-switchport 0/9
no shutdown
!
interface gigabit-switchport 0/10
no shutdown
!
interface gigabit-switchport 0/11
no shutdown
!
interface gigabit-switchport 0/12
no shutdown
!
interface gigabit-switchport 0/13
no shutdown
!
interface gigabit-switchport 0/14
no shutdown
!
interface gigabit-switchport 0/15
no shutdown
!
interface gigabit-switchport 0/16
no shutdown
!
interface gigabit-switchport 0/17
no shutdown
!
interface gigabit-switchport 0/18
no shutdown
!
interface gigabit-switchport 0/19
no shutdown
!
interface gigabit-switchport 0/20
no shutdown
!
interface gigabit-switchport 0/21
no shutdown
!
interface gigabit-switchport 0/22
no shutdown
!
interface gigabit-switchport 0/23
no shutdown
!
interface gigabit-switchport 0/24
no shutdown
!
!
interface xgigabit-switchport 0/1
no shutdown
switchport mode access
speed auto
!
interface xgigabit-switchport 0/2
no shutdown
switchport mode access
speed auto
!
interface xgigabit-switchport 0/3
no shutdown
switchport mode access
speed auto
!
interface xgigabit-switchport 0/4
no shutdown
switchport mode access
speed auto
!
!
!
interface vlan 1
ip address 172.16.0.143 255.255.0.0
no awcp
no shutdown
!
!
!
!
!
no tftp server
no tftp server overwrite
http server
http secure-server
no snmp agent
no ip ftp server
no ip scp server
no ip sntp server
!
!
!
!
!
snmp-server engine local 8000029803000000000000
!
!
!
!
line con 0
no login
!
line telnet 0 4
login
password XXXX
no shutdown
line ssh 0 4
login local-userlist
no shutdown
!
!
!
!
end
Is this a flat /16 network? You don't have a default route in your latest configuration. Connected management host on same subnet? Connected management host getting the proper /16 mask from customer's DHCP server? Firewall or similar doing proxy-arp perhaps? When you try to ping the unresponsive Adtran device, does the correct MAC show up in the management host's ARP table?
Indeed, a flat /16 network. Latest config was one of many, in hopes of finding a silver bullet - tried without default route, as goal is truly a layer two operation with management. Host PC is on same subnet, and physically connected to the switch that we're doing most diagnostics on. PC IP address is assigned via DHCP, and getting a /16 address in same subnet. We had tried assigning static address to PC for testing, no difference. When I try to ping unit, correct MAC shows in PC's arp table.
That's a real stumper. The only thing I can think of is that something on the network, perhaps a firewall, is doing proxy-arp, has a duplicate IP to the switch, or is tearing down TCP connections with a RST, but that wouldn't explain loss of ping.
If you can work in a maintenance window, you can isolate sections of the customer network until you find the culprit.
Just to close this out, the culprit was overwhelming multicast and broadcast traffic. The good folks at support provided fabulous assistance in identifying the issues as well as some possible resolutions. We still need to deal with the culprits, but at least we can now appropriately manage the switches as we work thru the process.