Hi everyone,
Have a strange and annoying problem at a site with a Netvanta 3448. Using only a fraction of the upstream bandwidth available (less than 2Mbps on 10Mbps available), viewing a remote security camera live video, I have the following symptoms :
- extremely choppy video
- the router only 'spits' out ~60 Kbps max
- strange drops and delays
eth 0/1 (wan - cable on b2b2c) info
Hardware address is 00:00:00:00:00:00
Ip address is xx.xx.xx.xx, netmask is 255.255.255.252
IP MTU is 1500 bytes
BW is 10000 Kbit
100Mb/s, negotiated full-duplex, configured full-duplex
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: never
5 minute input rate 12256 bits/sec, 16 packets/sec
5 minute output rate 62888 bits/sec, 16 packets/sec
Queueing method
Configured Queueing Method: fifo
Effective Queueing Method: weighted fair
Output queue: 0/258/684/64/446 (size/highest/max total/threshold/drops)
Conversations 0/9/256 (active/max active/max total)
Available Bandwidth 7500 kilobits/sec
Interface Shaper: 10000/62500/62500 (rate/budget/max budget)
1250 bytes added to budget every 1 ms
packet stats: 430053/0/446/27752 (packets sent/waiting/dropped/delayed)
29772890 packets input, 31111429592 bytes
29772884 unicasts, 6 broadcasts, 0 multicasts input
0 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
25975123 packets output, 6980499359 bytes
25973498 unicasts, 2071 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 0 throttles
! ADTRAN, Inc. OS version R11.5.0
! Boot ROM version 13.03.00.SB
! Platform: NetVanta 3448, part number 1200821E1
I tried a completely different router (Mikrotik) and everything is fine, no problems, no drops, no choppy video when watching the remote security camera.
Can you guys suggest something?
Anyone? I really need to sort this out and don't know where to look anymore. If you guys need more info, just tell me what you need and I will post it.
Thanks a lot.
Can you post your running config?
What is the status of the uplink side? Are you able to view link statistics there? It is always possible that while your side sees a clean 100mb full duplex link, the other side is not auto negotiating and is stuck at half duplex. You mention you have 10mbs available for uplink bandwidth. Is this a 10mb circuit all together? In many cases, circuits that only provide 10mb overall usually have the handoff port set statically to 10mb/FD.
Does WAN traffic match what is coming in on the LAN?
packet stats: 430053/0/446/27752 (packets sent/waiting/dropped/delayed)
You have a large amount of delayed packets. If the router is delaying them, there must be a reason.
Try setting your interface counters to 30 seconds to see if you can get a better picture of bandwidth usage. Maybe something is spiking every 30-60 seconds which causes interface delays and packet drops but isn't sustained long enough to really affect the 5 minute average. From CLI, use "statistics rate-interval 30" in config mode to change the interface counters.
Thank you for your time.
> Can you post your running config?
!
!
! ADTRAN, Inc. OS version R11.5.0
! Boot ROM version 13.03.00.SB
! Platform: NetVanta 3448, part number 1200821E1
! Serial number LBADTN1117AN159
!
!
hostname "rtrgeorgebizet"
enable password xxxxx
!
clock timezone -5-Eastern-Time
!
ip subnet-zero
ip classless
ip default-gateway xxx.xxx.xxx.xxx
ip routing
ipv6 unicast-routing
!
!
domain-name "dolmen.local"
domain-proxy
name-server 4.2.2.1 4.2.2.2
!
!
no auto-config
!
event-history on
no logging forwarding
logging forwarding priority-level info
no logging email
!
no service password-encryption
!
portal-list "ALL" console ftp http-admin ssh telnet
portal-list "WEB" http-admin
!
username "xxxxx" password "xxxxx"
username "xxxxx" portal-list "WEB" password "xxxxx"
username "xxxxx" portal-list "ALL" password "xxxxx"
!
!
ip firewall
no ip firewall alg msn
no ip firewall alg mszone
no ip firewall alg h323
!
!
no dot11ap access-point-control
!
!
ip dhcp excluded-address 192.168.123.1 192.168.123.50
ip dhcp excluded-address 192.168.123.200 192.168.123.254
!
ip dhcp pool "Private"
network 10.10.10.0 255.255.255.0
dns-server 10.10.10.1
netbios-node-type h-node
default-router 10.10.10.1
!
!
qos map eth0/1QosWizard 20
! Implicit deny any
qos map eth0/1QosWizard 21
match ip list acleth0/1QosWizSignal21
set dscp 26
!
qos map Internal_to_outside 30
match ip rtp 10000 10084 all
set dscp ef
!
qos map VOIP 10
match precedence 3
match precedence 5
match precedence 6
match precedence 7
match ip rtp 10020 10053 all
priority 10000 strict-rate-limiting
set precedence 5
!
qos cos-map 1 0 1
qos cos-map 2 2 3
qos cos-map 3 4
qos cos-map 4 5 6 7
qos queue-type wrr 25 25 25 expedite
!
!
vlan 1
name "Default"
!
vlan 5
name "Lien vers St-Juste"
!
!
no ethernet cfm
!
interface eth 0/1
ip address xxx.xxx.xxx.xxx 255.255.255.252
ip access-policy Public
media-gateway ip primary
traffic-shape rate 10000000
no shutdown
no lldp send-and-receive
!
!
interface eth 0/2
no ip address
shutdown
!
!
interface switchport 0/1
no shutdown
!
interface switchport 0/2
no shutdown
!
interface switchport 0/3
no shutdown
!
interface switchport 0/4
no shutdown
!
interface switchport 0/5
no shutdown
!
interface switchport 0/6
no shutdown
!
interface switchport 0/7
no shutdown
!
interface switchport 0/8
no shutdown
switchport access vlan 5
!
!
interface vlan 1
ip address 192.168.123.254 255.255.255.0
ip access-policy Private
media-gateway ip primary
no shutdown
!
interface vlan 5
description Lien vers St-Juste
ip address 192.168.121.2 255.255.255.0
ip mtu 1500
ip access-policy "Private link"
media-gateway ip primary
no shutdown
!
!
interface t1 1/1
shutdown
!
!
ip access-list standard wizard-ics
remark Internet Connection Sharing
permit any
!
!
ip access-list extended acleth0/1QosWizSignal21
permit tcp any any eq 5080 log
permit udp any any eq 5080 log
permit udp any any eq 5060
permit tcp any any eq 5060
!
ip access-list extended self
remark Traffic to NetVanta
permit ip any any log
!
ip access-list extended web-acl-10
remark CHUB Security
permit tcp any any eq 8080 log
permit tcp any any eq 8254 log
permit tcp any any range 9010 9011 log
permit udp any any eq 8080 log
permit udp any any eq 8254 log
permit udp any any range 9010 9011 log
!
ip access-list extended web-acl-11
remark 443 vers server
permit tcp any any eq https log
!
ip access-list extended web-acl-12
remark SUPPORT DATA-SRV
permit tcp any any eq 5991 log
!
ip access-list extended web-acl-17
remark Traffic to unit
permit ip any any log
!
ip access-list extended web-acl-18
remark InterVlan
permit ip 192.168.124.0 0.0.0.255 192.168.123.0 0.0.0.255 log
!
ip access-list extended web-acl-19
remark Lien vers St-Just
permit ip 192.168.123.0 0.0.0.255 192.168.124.0 0.0.0.255 log
!
ip access-list extended web-acl-3
remark mail
permit tcp any any eq smtp log
permit tcp any any eq 5001 log
!
ip access-list extended web-acl-4
remark admin access
permit tcp any any eq 8443 log
deny udp any any eq snmp log
deny tcp any any eq ssh log
deny tcp any any eq ftp log
permit icmp any any log
!
ip access-list extended web-acl-5
remark VPN SBS-SRV
permit tcp any any eq 1723 log
!
ip access-list extended web-acl-9
remark Remote SV8100
permit tcp any any eq 8000 log
!
!
ip policy-class Private
allow list self self
allow list web-acl-19 policy "Private link" stateless
nat source list wizard-ics interface eth 0/1 overload
!
ip policy-class "Private link"
allow list web-acl-17 self stateless
allow list web-acl-18 policy Private stateless
!
ip policy-class Public
nat destination list web-acl-3 address 192.168.123.8
allow list web-acl-4 self
nat destination list web-acl-5 address 192.168.123.8
nat destination list web-acl-9 address 192.168.123.20
nat destination list web-acl-10 address 192.168.123.50
nat destination list web-acl-11 address 192.168.123.8
nat destination list web-acl-12 address 192.168.123.7
!
!
ip route 192.168.124.0 255.255.255.0 192.168.121.1
!
no tftp server
no tftp server overwrite
http server
http secure-server 8443
snmp agent
no ip ftp server
ip ftp server default-filesystem flash
no ip scp server
no ip sntp server
!
!
snmp-server community Luminet RO
!
!
sip udp 5060
sip udp 5080
sip tcp 5060
sip tcp 5080
!
!
line con 0
no login
!
line telnet 0 4
login local-userlist
password password
no shutdown
line ssh 0 4
login local-userlist
no shutdown
!
!
ntp source ethernet 0/1
ntp peer pool.ntp.org source ethernet 0/1
!
!
end
> What is the status of the uplink side? Are you able to view link statistics there?
The links are : Netvanta -> Cisco router (provided by the ISP) -> cable modem (provided by the ISP)
We have no access at all to the ISP equipment.
We also have another similar config at another branch of the office (same Netvanta, same ISP equipment and internet connection) and everything is running smooth and fine there (no drops, no delays, no problem at all).
> Try setting your interface counters to 30 seconds to see if you can get a better picture of bandwidth usage. Maybe something is spiking every 30-60 seconds which causes interface delays and packet drops but isn't sustained long enough to really affect the 5 minute average. From CLI, use "statistics rate-interval 30" in config mode to change the interface counters.
I will try to figure out how to do for the rest (I'm not familiar with Adtran routers, and even less with the CLI interface!) and report back with more info.
What kind of Cisco router is installed? Have you called the ISP to have them check their equipment for issues?
Have you tried any testing behind the router? Running a test to a speed test site? Pinging places on the web and looking for packet loss?
Generally, I don't really see an issue in the config. However, I don't see a default route in the routing table. While you have a default gateway set, that is really more for the host and not for routing. See this document - What is the difference between a default route and a default gateway?
You should put a default route in the config.
ip route 0.0.0.0 0.0.0.0 ISP.GATEWAY.IP.ADDR
You can enable 30 second counters by going into "enabled" mode in CLI, config term, and entering "statistics rate-interval 30". You describe "strange drops and delays". I would definitely look for bursting traffic.
I would then recommend running other speed tests and ping tests from behind the router. Leave some ICMP tests running and look for any packet loss patterns.
I'm not on site right now to check Cisco model and I did not call the ISP until now because I have done some testing and the results are not pointing in his direction.
1. I have duplicated the config on a test router I had on hand (not Adtran) switched it with the Netvanta and everything was working fine
2. All the speed tests (on speedtest.net), with or without the Netvanta are fine and consistent
-- last one from a couple of minutes ago : ping 19 ms, download 21 Mbps, upload 8 Mbps
Still, when I try to watch the security camera remotely, no more than 60 kbps is going out of the router (on the similar config at the other branch, doing the same thing, I can observe a sustained upstream traffic of ~ 2 Mbps). And at the problematic location, when I switch with the test router I have normal real time video stream and a traffic of ~ 1.5 -- 2 Mbps.
For the default route, I can see it present / dynamically added in the web interface.
For the delays and drops, I figured that the counters were never cleared..! I managed to clear the counters and I have the following results :
--> After 35 minutes :
------------------------------------------------------------
eth 0/1 is UP, line protocol is UP
Hardware address is xx:xx:xx:xx:xx:xx
Ip address is xxx.xxx.xxx.xxx, netmask is 255.255.255.252
IP MTU is 1500 bytes
BW is 10000 Kbit
100Mb/s, negotiated full-duplex, configured full-duplex
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: 00:34:53
5 minute input rate 925360 bits/sec, 105 packets/sec
5 minute output rate 209544 bits/sec, 90 packets/sec
Queueing method
Configured Queueing Method: fifo
Effective Queueing Method: weighted fair
Output queue: 0/1/684/64/0 (size/highest/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Available Bandwidth 7500 kilobits/sec
Interface Shaper: 10000/62500/62500 (rate/budget/max budget)
1250 bytes added to budget every 1 ms
packet stats: 79431/0/0/0 (packets sent/waiting/dropped/delayed)
84684 packets input, 76456219 bytes
84684 unicasts, 0 broadcasts, 0 multicasts input
0 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
79431 packets output, 21910783 bytes
79427 unicasts, 4 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 0 throttles
--> After 2 hours 15 minutes :
------------------------------------------------------------
eth 0/1 is UP, line protocol is UP
Hardware address is xx:xx:xx:xx:xx:xx
Ip address is xxx.xxx.xxx.xxx, netmask is 255.255.255.252
IP MTU is 1500 bytes
BW is 10000 Kbit
100Mb/s, negotiated full-duplex, configured full-duplex
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: 02:15:16
5 minute input rate 197192 bits/sec, 36 packets/sec
5 minute output rate 53760 bits/sec, 36 packets/sec
Queueing method
Configured Queueing Method: fifo
Effective Queueing Method: weighted fair
Output queue: 0/141/684/64/679 (size/highest/max total/threshold/drops)
Conversations 0/4/256 (active/max active/max total)
Available Bandwidth 7500 kilobits/sec
Interface Shaper: 10000/62500/62500 (rate/budget/max budget)
1250 bytes added to budget every 1 ms
packet stats: 308906/0/679/10980 (packets sent/waiting/dropped/delayed)
339270 packets input, 318062208 bytes
339270 unicasts, 0 broadcasts, 0 multicasts input
0 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
308906 packets output, 78807896 bytes
309571 unicasts, 14 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 0 throttles
But even in the first 35 minutes, with no apparent delay/drop, I had the same throughput problem when watching the remote camera.
What do the counters look like on your VLAN interfaces and switchports? What do their link negotiations look like? Are you connecting to a 10/100 switch into the network or are the devices directly connected into the 3448?
You still accumulated a lot of shaper induced delays. Try dropping the interface shaper and using a QOS map to prioritize the video traffic specifically and shape all other traffic behind it.
VLAN interfaces and switchports in use :
swx 0/2 is UP, line protocol is UP
Hardware address is xx:xx:xx:xx:xx:xx
100Mb/s, negotiated full-duplex, configured full-duplex
input flow control is disabled
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: 23:53:02
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
Queueing method: fifo
Output queue: 0/256/0 (size/max total/drops)
Interface Shaper: NOT ENABLED
4095797 packets input
0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 alignment errors, 0 crc errors
6522904 packets output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions
swx 0/3 is UP, line protocol is UP
Hardware address is xx:xx:xx:xx:xx:xx
100Mb/s, negotiated full-duplex, configured full-duplex
input flow control is disabled
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: 23:53:02
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
Queueing method: fifo
Output queue: 0/256/0 (size/max total/drops)
Interface Shaper: NOT ENABLED
336687 packets input
0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 alignment errors, 0 crc errors
719051 packets output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions
vlan 1 is UP
Hardware address is xx:xx:xx:xx:xx:xx
vlan 1 Ip address is 192.168.123.254, netmask is 255.255.255.0
IP MTU is 1500 bytes
BW is 100000 Kbit
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: 23:53:04
5 minute input rate 181576 bits/sec, 141 packets/sec
5 minute output rate 1059720 bits/sec, 169 packets/sec
4344309 packets input, 622218684 bytes
4104772 unicasts, 239537 broadcasts, 0 multicasts input
47350 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
6824034 packets output, 7931497187 bytes
6793627 unicasts, 30407 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 0 throttles
vlan 5 is UP
Description: Lien vers St-Juste
Hardware address is xx:xx:xx:xx:xx:xx
vlan 5 Ip address is 192.168.121.2, netmask is 255.255.255.0
IP MTU is 1500 bytes
BW is 100000 Kbit
ARP type: ARPA; ARP timeout is 20 minutes
Last clearing of "show interface" counters: 23:53:04
5 minute input rate 90704 bits/sec, 50 packets/sec
5 minute output rate 92736 bits/sec, 50 packets/sec
3634597 packets input, 4711280427 bytes
3634597 unicasts, 0 broadcasts, 0 multicasts input
0 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
1580004 packets output, 242778140 bytes
1579861 unicasts, 143 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 0 throttles
The LAN is connected to a Netgear JGS524E 10/100/1000 switch and to the Netvanta on the port swx 0/2. A phone system is connected directly to swx 0/3.
I activated the shaper only to let know the Netvanta that no matter the negotiated port speed, the available upload bandwidth is max 10 Mbps on Internet port eth 0/1.
QOS maps are defined, but inactive for testing purposes only.
I have, indeed caught the following message in the CLI occasionally :
2015.11.03 14:22:50 LLDP LLDP: Speed Duplex mis-match on swx 0/2. Speed/Duplex: "100M/full-duplex" Neighbor Speed/Duplex: "? duplex"
I will go on site to investigate, maybe change the network cable / switch port just in case.
Despite what common sense says, I've seen plenty of issues with auto-negotiation between 1000mb and 100 mb ports. That is definitely something to investigate, since there is a good chance that a.) auto-neg just isn't behaving or b.) the switch trunk port is statically set to be 1000/F.
Good luck, hopefully resolving that error will fix the problem you are having.
I thought of a couple of suggestions if you'd like to focus on the camera interface rather than the ISP.
Do the settings on the Netgear complement the settings on the Adtran? I looked over the Netgear manual and there are settings for QoS and 802.1q, so that's encouraging. Can you bypass the Netgear and connect the camera directly to the Adtran? That will give you much more control, and at least one less hop, to the camera.
If this camera operates on CS4 / AF41 / DSCP 34, then consider adding something like this to your global configuration:
qos dscp-cos 0 8 16 24 34 46 48 56 to 0 1 2 3 4 5 6 7
Also, consider adding these commands to the camera interface on the Adtran.
spanning-tree edgeport
no lldp send-and-receive
qos default-cos 4
- take this with a grain of salt
- this is my best guess since I don't know what QoS setting your camera prefers
- make sure this maps correctly to the dscp-cos map
- at a minimum, this will help you troubleshoot
HTH
Thank you for your effort.
I did check the connections yesterday and changed the ethernet cable between the Netvanta and the LAN switch.
I also confirmed with the ISP that everything is fine on their side and there is nothing unusual to report from their hardware and link quality.
They also confirmed that there is no targeted bandwidth limitation or QOS of any kind in the Cisco router our anywhere else that could possibly cause our problem.
I probably wont be able to connect directly the camera system to the Netvanta because of the distance between the equipment.
Do you know what would be the easiest way to real-time monitor the activity, traffic, etc on the Netvanta?
Were you able to log into the switch and confirm the port configuration? Are you still getting duplex mismatch notifications? That is a big issue to resolve, regardless if it actually fixes this problem or not.
To monitor traffic real-time, you can use a "sh interface eth 0/1 realtime" to view interface counters in near real time. You will be benefited by changing the counters to 30 seconds prior to monitoring this way.
You could also run a debug ip packet. Perhaps something is happening in the packet stream.