Netvanta 3448 slow throughput / too many drops and delays

[mmmtl]

Hi everyone,

Have a strange and annoying problem at a site with a Netvanta 3448. Using only a fraction of the upstream bandwidth available (less than 2Mbps on 10Mbps available), viewing a remote security camera live video, I have the following symptoms :

- extremely choppy video

- the router only 'spits' out ~60 Kbps max

- strange drops and delays

eth 0/1 (wan - cable on b2b2c) info

Hardware address is 00:00:00:00:00:00

  Ip address is xx.xx.xx.xx, netmask is 255.255.255.252

  IP MTU is 1500 bytes

  BW is 10000 Kbit

  100Mb/s, negotiated full-duplex, configured full-duplex

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: never

  5 minute input rate 12256 bits/sec, 16 packets/sec

  5 minute output rate 62888 bits/sec, 16 packets/sec

    Queueing method

        Configured Queueing Method: fifo

        Effective  Queueing Method: weighted fair

    Output queue: 0/258/684/64/446 (size/highest/max total/threshold/drops)

      Conversations  0/9/256 (active/max active/max total)

      Available Bandwidth 7500 kilobits/sec

    Interface Shaper: 10000/62500/62500 (rate/budget/max budget)

      1250 bytes added to budget every 1 ms

      packet stats: 430053/0/446/27752 (packets sent/waiting/dropped/delayed)

    29772890 packets input, 31111429592 bytes

    29772884 unicasts, 6 broadcasts, 0 multicasts input

    0 unknown protocol, 0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 no buffer, 0 overruns, 0 internal receive errors

    0 alignment errors, 0 crc errors

    25975123 packets output, 6980499359 bytes

    25973498 unicasts, 2071 broadcasts, 0 multicasts output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions, 0 underruns

    0 internal transmit errors, 0 carrier sense errors

    0 resets, 0 throttles

! ADTRAN, Inc. OS version R11.5.0

! Boot ROM version 13.03.00.SB

! Platform: NetVanta 3448, part number 1200821E1

I tried a completely different router (Mikrotik) and everything is fine, no problems, no drops, no choppy video when watching the remote security camera.

Can you guys suggest something?

[mmmtl]

Anyone? I really need to sort this out and don't know where to look anymore. If you guys need more info, just tell me what you need and I will post it.

Thanks a lot.

[Anonymous]

Can you post your running config?

What is the status of the uplink side?  Are you able to view link statistics there?  It is always possible that while your side sees a clean 100mb full duplex link, the other side is not auto negotiating and is stuck at half duplex.  You mention you have 10mbs available for uplink bandwidth.  Is this a 10mb circuit all together?  In many cases, circuits that only provide 10mb overall usually have the handoff port set statically to 10mb/FD. 

Does WAN traffic match what is coming in on the LAN? 

packet stats: 430053/0/446/27752 (packets sent/waiting/dropped/delayed)

You have a large amount of delayed packets.  If the router is delaying them, there must be a reason. 

Try setting your interface counters to 30 seconds to see if you can get a better picture of bandwidth usage.  Maybe something is spiking every 30-60 seconds which causes interface delays and packet drops but isn't sustained long enough to really affect the 5 minute average.  From CLI, use "statistics rate-interval 30" in config mode to change the interface counters.

[mmmtl]

Thank you for your time.

> Can you post your running config?

!

!

! ADTRAN, Inc. OS version R11.5.0

! Boot ROM version 13.03.00.SB

! Platform: NetVanta 3448, part number 1200821E1

! Serial number LBADTN1117AN159

!

!

hostname "rtrgeorgebizet"

enable password xxxxx

!

clock timezone -5-Eastern-Time

!

ip subnet-zero

ip classless

ip default-gateway xxx.xxx.xxx.xxx

ip routing

ipv6 unicast-routing

!

!

domain-name "dolmen.local"

domain-proxy

name-server 4.2.2.1 4.2.2.2

!

!

no auto-config

!

event-history on

no logging forwarding

logging forwarding priority-level info

no logging email

!

no service password-encryption

!

portal-list "ALL" console ftp http-admin ssh telnet

portal-list "WEB" http-admin

!

username "xxxxx" password "xxxxx"

username "xxxxx" portal-list "WEB" password "xxxxx"

username "xxxxx" portal-list "ALL" password "xxxxx"

!

!

ip firewall

no ip firewall alg msn

no ip firewall alg mszone

no ip firewall alg h323

!

!

no dot11ap access-point-control

!

!

ip dhcp excluded-address 192.168.123.1 192.168.123.50

ip dhcp excluded-address 192.168.123.200 192.168.123.254

!

ip dhcp pool "Private"

  network 10.10.10.0 255.255.255.0

  dns-server 10.10.10.1

  netbios-node-type h-node

  default-router 10.10.10.1

!

!

qos map eth0/1QosWizard 20

  ! Implicit deny any

qos map eth0/1QosWizard 21

  match ip list acleth0/1QosWizSignal21

  set dscp 26

!

qos map Internal_to_outside 30

  match ip rtp 10000 10084 all

  set dscp ef

!

qos map VOIP 10

  match precedence 3

  match precedence 5

  match precedence 6

  match precedence 7

  match ip rtp 10020 10053 all

  priority 10000 strict-rate-limiting

  set precedence 5

!

qos cos-map 1 0 1

qos cos-map 2 2 3

qos cos-map 3 4

qos cos-map 4 5 6 7

qos queue-type wrr 25 25 25 expedite

!

!

vlan 1

  name "Default"

!

vlan 5

  name "Lien vers St-Juste"

!

!

no ethernet cfm

!

interface eth 0/1

  ip address  xxx.xxx.xxx.xxx  255.255.255.252

  ip access-policy Public

  media-gateway ip primary

  traffic-shape rate 10000000

  no shutdown

  no lldp send-and-receive

!

!

interface eth 0/2

  no ip address

  shutdown

!

!

interface switchport 0/1

  no shutdown

!

interface switchport 0/2

  no shutdown

!

interface switchport 0/3

  no shutdown

!

interface switchport 0/4

  no shutdown

!

interface switchport 0/5

  no shutdown

!

interface switchport 0/6

  no shutdown

!

interface switchport 0/7

  no shutdown

!

interface switchport 0/8

  no shutdown

  switchport access vlan 5

!

!

interface vlan 1

  ip address  192.168.123.254  255.255.255.0

  ip access-policy Private

  media-gateway ip primary

  no shutdown

!

interface vlan 5

  description Lien vers St-Juste

  ip address  192.168.121.2  255.255.255.0

  ip mtu 1500

  ip access-policy "Private link"

  media-gateway ip primary

  no shutdown

!

!

interface t1 1/1

  shutdown

!

!

ip access-list standard wizard-ics

  remark Internet Connection Sharing

  permit any

!

!

ip access-list extended acleth0/1QosWizSignal21

  permit tcp any  any eq 5080   log

  permit udp any  any eq 5080    log

  permit udp any  any eq 5060 

  permit tcp any  any eq 5060

!

ip access-list extended self

  remark Traffic to NetVanta

  permit ip any  any     log

!

ip access-list extended web-acl-10

  remark CHUB Security

  permit tcp any  any eq 8080   log

  permit tcp any  any eq 8254   log

  permit tcp any  any range 9010 9011   log

  permit udp any  any eq 8080    log

  permit udp any  any eq 8254    log

  permit udp any  any range 9010 9011    log

!

ip access-list extended web-acl-11

  remark 443 vers server

  permit tcp any  any eq https   log

!

ip access-list extended web-acl-12

  remark SUPPORT DATA-SRV

  permit tcp any  any eq 5991   log

!

ip access-list extended web-acl-17

  remark Traffic to unit

  permit ip any  any     log

!

ip access-list extended web-acl-18

  remark InterVlan

  permit ip 192.168.124.0 0.0.0.255  192.168.123.0 0.0.0.255     log

!

ip access-list extended web-acl-19

  remark Lien vers St-Just

  permit ip 192.168.123.0 0.0.0.255  192.168.124.0 0.0.0.255     log

!

ip access-list extended web-acl-3

  remark mail

  permit tcp any  any eq smtp   log

  permit tcp any  any eq 5001   log

!

ip access-list extended web-acl-4

  remark admin access

  permit tcp any  any eq 8443   log

  deny   udp any  any eq snmp    log

  deny   tcp any  any eq ssh   log

  deny   tcp any  any eq ftp   log

  permit icmp any  any     log

!

ip access-list extended web-acl-5

  remark VPN SBS-SRV

  permit tcp any  any eq 1723   log

!

ip access-list extended web-acl-9

  remark Remote SV8100

  permit tcp any  any eq 8000   log

!

!

ip policy-class Private

  allow list self self

  allow list web-acl-19 policy "Private link" stateless

  nat source list wizard-ics interface eth 0/1 overload

!

ip policy-class "Private link"

  allow list web-acl-17 self stateless

  allow list web-acl-18 policy Private stateless

!

ip policy-class Public

  nat destination list web-acl-3 address 192.168.123.8

  allow list web-acl-4 self

  nat destination list web-acl-5 address 192.168.123.8

  nat destination list web-acl-9 address 192.168.123.20

  nat destination list web-acl-10 address 192.168.123.50

  nat destination list web-acl-11 address 192.168.123.8

  nat destination list web-acl-12 address 192.168.123.7

!

!

ip route 192.168.124.0 255.255.255.0 192.168.121.1

!

no tftp server

no tftp server overwrite

http server

http secure-server 8443

snmp agent

no ip ftp server

ip ftp server default-filesystem flash

no ip scp server

no ip sntp server

!

!

snmp-server community Luminet RO

!

!

sip udp 5060

sip udp 5080

sip tcp 5060

sip tcp 5080

!

!

line con 0

  no login

!

line telnet 0 4

  login local-userlist

  password password

  no shutdown

line ssh 0 4

  login local-userlist

  no shutdown

!

!

ntp source ethernet 0/1

ntp peer pool.ntp.org source ethernet 0/1

!

!

end

> What is the status of the uplink side?  Are you able to view link statistics there?

The links are : Netvanta -> Cisco router (provided by the ISP) -> cable modem (provided by the ISP)

We have no access at all to the ISP equipment.

We also have another similar config at another branch of the office (same Netvanta, same ISP equipment and internet connection) and everything is running smooth and fine there (no drops, no delays, no problem at all).

> Try setting your interface counters to 30 seconds to see if you can get a better picture of bandwidth usage.  Maybe something is spiking every 30-60 seconds which causes  interface delays and packet drops but isn't sustained long enough to really affect the 5 minute average.  From CLI, use "statistics rate-interval 30" in config mode to change the interface counters.

I will try to figure out how to do for the rest (I'm not familiar with Adtran routers, and even less with the CLI interface!) and report back with more info.

[Anonymous]

What kind of Cisco router is installed?  Have you called the ISP to have them check their equipment for issues?

Have you tried any testing behind the router?  Running a test to a speed test site?  Pinging places on the web and looking for packet loss?

Generally, I don't really see an issue in the config.  However, I don't see a default route in the routing table.  While you have a default gateway set, that is really more for the host and not for routing.  See this document - What is the difference between a default route and a default gateway?

You should put a default route in the config.

ip route 0.0.0.0 0.0.0.0 ISP.GATEWAY.IP.ADDR

You can enable 30 second counters by going into "enabled" mode in CLI, config term, and entering "statistics rate-interval 30".  You describe "strange drops and delays".  I would definitely look for bursting traffic.

I would then recommend running other speed tests and ping tests from behind the router.  Leave some ICMP tests running and look for any packet loss patterns.

[mmmtl]

I'm not on site right now to check Cisco model and I did not call the ISP until now because I have done some testing and the results are not pointing in his direction.

1. I have duplicated the config on a test router I had on hand (not Adtran) switched it with the Netvanta and everything was working fine

2. All the speed tests (on speedtest.net), with or without the Netvanta are fine and consistent

-- last one from a couple of minutes ago : ping 19 ms, download 21 Mbps, upload 8 Mbps

Still, when I try to watch the security camera remotely, no more than 60 kbps is going out of the router (on the similar config at the other branch, doing the same thing, I can observe a sustained upstream traffic of ~ 2 Mbps). And at the problematic location, when I switch with the test router I have normal real time video stream and a traffic of ~ 1.5 -- 2 Mbps.

For the default route, I can see it present / dynamically added in the web interface.

For the delays and drops, I figured that the counters were never cleared..! I managed to clear the counters and I have the following results :

--> After 35 minutes :

------------------------------------------------------------

eth 0/1 is UP, line protocol is UP

  Hardware address is xx:xx:xx:xx:xx:xx

  Ip address is xxx.xxx.xxx.xxx, netmask is 255.255.255.252

  IP MTU is 1500 bytes

  BW is 10000 Kbit

  100Mb/s, negotiated full-duplex, configured full-duplex

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: 00:34:53

  5 minute input rate 925360 bits/sec, 105 packets/sec

  5 minute output rate 209544 bits/sec, 90 packets/sec

    Queueing method

        Configured Queueing Method: fifo

        Effective  Queueing Method: weighted fair

    Output queue: 0/1/684/64/0 (size/highest/max total/threshold/drops)

      Conversations  0/1/256 (active/max active/max total)

      Available Bandwidth 7500 kilobits/sec

    Interface Shaper: 10000/62500/62500 (rate/budget/max budget)

      1250 bytes added to budget every 1 ms

      packet stats: 79431/0/0/0 (packets sent/waiting/dropped/delayed)

    84684 packets input, 76456219 bytes

    84684 unicasts, 0 broadcasts, 0 multicasts input

    0 unknown protocol, 0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 no buffer, 0 overruns, 0 internal receive errors

    0 alignment errors, 0 crc errors

    79431 packets output, 21910783 bytes

    79427 unicasts, 4 broadcasts, 0 multicasts output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions, 0 underruns

    0 internal transmit errors, 0 carrier sense errors

    0 resets, 0 throttles

--> After 2 hours 15 minutes :

------------------------------------------------------------

eth 0/1 is UP, line protocol is UP

  Hardware address is xx:xx:xx:xx:xx:xx

  Ip address is xxx.xxx.xxx.xxx, netmask is 255.255.255.252

  IP MTU is 1500 bytes

  BW is 10000 Kbit

  100Mb/s, negotiated full-duplex, configured full-duplex

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: 02:15:16

  5 minute input rate 197192 bits/sec, 36 packets/sec

  5 minute output rate 53760 bits/sec, 36 packets/sec

    Queueing method

        Configured Queueing Method: fifo

        Effective  Queueing Method: weighted fair

    Output queue: 0/141/684/64/679 (size/highest/max total/threshold/drops)

      Conversations  0/4/256 (active/max active/max total)

      Available Bandwidth 7500 kilobits/sec

    Interface Shaper: 10000/62500/62500 (rate/budget/max budget)

      1250 bytes added to budget every 1 ms

      packet stats: 308906/0/679/10980 (packets sent/waiting/dropped/delayed)

    339270 packets input, 318062208 bytes

    339270 unicasts, 0 broadcasts, 0 multicasts input

    0 unknown protocol, 0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 no buffer, 0 overruns, 0 internal receive errors

    0 alignment errors, 0 crc errors

    308906 packets output, 78807896 bytes

    309571 unicasts, 14 broadcasts, 0 multicasts output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions, 0 underruns

    0 internal transmit errors, 0 carrier sense errors

    0 resets, 0 throttles

But even in the first 35 minutes, with no apparent delay/drop, I had the same throughput problem when watching the remote camera.

[Anonymous]

What do the counters look like on your VLAN interfaces and switchports?  What do their link negotiations look like?  Are you connecting to a 10/100 switch into the network or are the devices directly connected into the 3448?

You still accumulated a lot of shaper induced delays.  Try dropping the interface shaper and using a QOS map to prioritize the video traffic specifically and shape all other traffic behind it.

[mmmtl]

VLAN interfaces and switchports in use :

swx 0/2 is UP, line protocol is UP

  Hardware address is xx:xx:xx:xx:xx:xx

  100Mb/s, negotiated full-duplex, configured full-duplex

  input flow control is disabled

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: 23:53:02

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

    Queueing method: fifo

    Output queue: 0/256/0 (size/max total/drops)

    Interface Shaper: NOT ENABLED

    4095797 packets input

    0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 alignment errors, 0 crc errors

    6522904 packets output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions

swx 0/3 is UP, line protocol is UP

  Hardware address is xx:xx:xx:xx:xx:xx

  100Mb/s, negotiated full-duplex, configured full-duplex

  input flow control is disabled

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: 23:53:02

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

    Queueing method: fifo

    Output queue: 0/256/0 (size/max total/drops)

    Interface Shaper: NOT ENABLED

    336687 packets input

    0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 alignment errors, 0 crc errors

    719051 packets output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions

vlan 1 is UP

  Hardware address is xx:xx:xx:xx:xx:xx

  vlan 1 Ip address is 192.168.123.254, netmask is 255.255.255.0

  IP MTU is 1500 bytes

  BW is 100000 Kbit

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: 23:53:04

  5 minute input rate 181576 bits/sec, 141 packets/sec

  5 minute output rate 1059720 bits/sec, 169 packets/sec

    4344309 packets input, 622218684 bytes

    4104772 unicasts, 239537 broadcasts, 0 multicasts input

    47350 unknown protocol, 0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 no buffer, 0 overruns, 0 internal receive errors

    0 alignment errors, 0 crc errors

    6824034 packets output, 7931497187 bytes

    6793627 unicasts, 30407 broadcasts, 0 multicasts output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions, 0 underruns

    0 internal transmit errors, 0 carrier sense errors

    0 resets, 0 throttles

vlan 5 is UP

  Description: Lien vers St-Juste

  Hardware address is xx:xx:xx:xx:xx:xx

  vlan 5 Ip address is 192.168.121.2, netmask is 255.255.255.0

  IP MTU is 1500 bytes

  BW is 100000 Kbit

  ARP type: ARPA; ARP timeout is 20 minutes

  Last clearing of "show interface" counters: 23:53:04

  5 minute input rate 90704 bits/sec, 50 packets/sec

  5 minute output rate 92736 bits/sec, 50 packets/sec

    3634597 packets input, 4711280427 bytes

    3634597 unicasts, 0 broadcasts, 0 multicasts input

    0 unknown protocol, 0 symbol errors, 0 discards

    0 input errors, 0 runts, 0 giants

    0 no buffer, 0 overruns, 0 internal receive errors

    0 alignment errors, 0 crc errors

    1580004 packets output, 242778140 bytes

    1579861 unicasts, 143 broadcasts, 0 multicasts output

    0 output errors, 0 deferred, 0 discards

    0 single, 0 multiple, 0 late collisions

    0 excessive collisions, 0 underruns

    0 internal transmit errors, 0 carrier sense errors

    0 resets, 0 throttles

The LAN is connected to a Netgear JGS524E 10/100/1000 switch and to the Netvanta on the port swx 0/2. A phone system is connected directly to swx 0/3.

I activated the shaper only to let know the Netvanta that no matter the negotiated port speed, the available upload bandwidth is max 10 Mbps on Internet port eth 0/1.

QOS maps are defined, but inactive for testing purposes only.

I have, indeed caught the following message in the CLI occasionally :

2015.11.03 14:22:50 LLDP LLDP:  Speed Duplex mis-match on swx 0/2. Speed/Duplex: "100M/full-duplex"   Neighbor Speed/Duplex: "? duplex"

I will go on site to investigate, maybe change the network cable / switch port just in case.

[Anonymous]

Despite what common sense says, I've seen plenty of issues with auto-negotiation between 1000mb and 100 mb ports.  That is definitely something to investigate, since there is a good chance that a.) auto-neg just isn't behaving or b.) the switch trunk port is statically set to be 1000/F. 

Good luck, hopefully resolving that error will fix the problem you are having.

[petetransitguy]

I thought of a couple of suggestions if you'd like to focus on the camera interface rather than the ISP.

Do the settings on the Netgear complement the settings on the Adtran? I looked over the Netgear manual and there are settings for QoS and 802.1q, so that's encouraging. Can you bypass the Netgear and connect the camera directly to the Adtran? That will give you much more control, and at least one less hop, to the camera.

If this camera operates on CS4 / AF41 / DSCP 34, then consider adding something like this to your global configuration:
qos dscp-cos 0 8 16 24 34 46 48 56 to 0 1 2 3 4 5 6 7

Also, consider adding these commands to the camera interface on the Adtran.
spanning-tree edgeport
no lldp send-and-receive
qos default-cos 4
- take this with a grain of salt

- this is my best guess since I don't know what QoS setting your camera prefers
- make sure this maps correctly to the dscp-cos map

- at a minimum, this will help you troubleshoot

HTH

[mmmtl]

Thank you for your effort.

I did check the connections yesterday and changed the ethernet cable between the Netvanta and the LAN switch.

I also confirmed with the ISP that everything is fine on their side and there is nothing unusual to report from their hardware and link quality.

They also confirmed that there is no targeted bandwidth limitation or QOS of any kind in the Cisco router our anywhere else that could possibly cause our problem.

I probably wont be able to connect directly the camera system to the Netvanta because of the distance between the equipment.

Do you know what would be the easiest way to real-time monitor the activity, traffic, etc on the Netvanta?

[Anonymous]

Were you able to log into the switch and confirm the port configuration?  Are you still getting duplex mismatch notifications?  That is a big issue to resolve, regardless if it actually fixes this problem or not.

To monitor traffic real-time, you can use a "sh interface eth 0/1 realtime" to view interface counters in near real time.  You will be benefited by changing the counters to 30 seconds prior to monitoring this way.

You could also run a debug ip packet.  Perhaps something is happening in the packet stream.