I have a Netvanta 3200 that is ~7 hours away from me, and the site is reporting problems.
I'm able to login to the 3200 via telnet, but "show process cpu" is giving me 100% usage.
System load: 1sec:99.66% 1min:100.00% 5min:100.00% Min: 0.00% Max: 100.00%
Context switch load: 0.34%
Full output:
Invoked Exec Time Runtime Load %%
Task Id Task Name PRI STA (count) (usec) (usec) (1sec)
1 Idle 0 W 795667 1001 0 0.00
3 PC Config 7 S 463882 1345 56222 5.62
4 PacketRouting 38 W 575872 22 651806 65.18
5 Timer 39 W 1053662 42 8735 0.87
6 Timer-00 10 W 1132233 11 787 0.08
7 Nm01 5 W 0 14019541 0 0.00
8 Clock 9 W 12904 74 113 0.01
9 FrontPanel 37 W 60211 366 3956 0.40
10 con0 39 W 352 28 0 0.00
11 ICP Session 8 W 531 34 0 0.00
12 Thread Pool 4 R 1503 162 0 0.00
13 RouteTableTick 6 R 1345 237 0 0.00
14 OSPF 6 R 1036 425 0 0.00
15 IGMPTick 6 R 857 155 0 0.00
16 IGMP-Receiver 6 W 0 13553631 0 0.00
17 IP Events 24 W 5033 108 108 0.01
18 tcptimer 22 W 3915 51 351 0.04
19 tcpinp 22 W 2636 587 4196 0.42
20 tcpout 22 W 6251 217 4104 0.41
21 WWW 19 W 1209 180 0 0.00
22 DnsProxy 16 W 1282 116 0 0.00
23 DnsTable 16 W 880 15 0 0.00
24 SnmpThread 6 R 38017 37 0 0.00
25 Port Manager 9 W 50171 128 1664 0.17
26 eth 0/1 39 W 151171 4161 220758 22.08
27 FramerBaseThre~ 21 W 129714 127 1766 0.18
28 FramerBaseThre~ 21 W 128210 62 807 0.08
42 FTPServer List~ 5 W 1 306 0 0.00
43 SMTP Client 16 W 0 415 0 0.00
44 SNTP Client 19 W 214 69 0 0.00
47 RipOut 6 R 819 21 0 0.00
48 RipIn 6 W 3 214 0 0.00
Looks like Eth 0/1 is showing high usage, no?
eth 0/1 is UP, line protocol is UP
Hardware address is 00:A0:C8:4A:6E:6A
Running 802.1Q Encapsulation
100Mb/s, negotiated full-duplex, configured full-duplex
Last clearing of counters never
5 minute input rate 65184920 bits/sec, 82003 packets/sec
5 minute output rate 691960 bits/sec, 1557 packets/sec
Queueing method: fifo
Output queue: 0/256/0 (size/max total/drops)
Interface Shaper: NOT ENABLED
31308420 packets input, 3201993017 bytes
73025 unicasts, 31235395 broadcasts, 0 multicasts input
0 unknown protocol, 0 symbol errors, 0 discards
0 input errors, 0 runts, 0 giants
0 no buffer, 0 overruns, 0 internal receive errors
0 alignment errors, 0 crc errors
1416816 packets output, 87654194 bytes
0 unicasts, 0 broadcasts, 0 multicasts output
0 output errors, 0 deferred, 0 discards
0 single, 0 multiple, 0 late collisions
0 excessive collisions, 0 underruns
0 internal transmit errors, 0 carrier sense errors
0 resets, 142246 throttles
So yeah, something is abusing the Ethernet interface. (Note that this router was rebooted maybe 10-15 minutes ago.) At this point, I suspect there may be a bridge loop at the site - I have a managed switch onsite, but I can't access it because the abuse of the interface is causing me to lose my access to the switch management!
Is there anything I can do to try to mitigate this with the Adtran, so I can perhaps get to the switch management and then find the problem port and shut it off?
Thank you for asking this question in the support community. As you mentioned, the unit is being over-utilized and can't process traffic fast enough. You can see this in the output you included. First, in the show process cpu command the one second system load is "99.66%" used. Furthermore, PacketRouting is "65.18%" and eth 0/1 is "22.08%."
Then when you look at the interface Ethernet 0/1, the main thing to notice is the number of throttles. Throttles indicate the number of times an AOS device is unable to handle the incoming data rate and, as a result, proactively drops packets to get it to a rate that it can handle. (Please, see this post for further information on throttles and possible causes: https://supportforums.adtran.com/message/2058#2058)
Unfortunately, you are limited on how to troubleshoot this remotely, but one thing I would recommend is try to determine if a single host on the network is causing the problem. If you issue the show ip policy-sessions and show ip policy-stats commands you can see all the sessions going through the firewall. Often, if a single machine has a virus or is transmitting an abnormal amount of traffic, you will see a lot of policy-sessions in the firewall. Then you can create an access-control list (ACL) to block this IP address, which might provide you a window to get to the rest of the LAN.
I hope this makes sense, but please don't hesitate to reply to this post with any additional information or questions. I will be happy to help in any way I can.
Levi
Thank you for asking this question in the support community. As you mentioned, the unit is being over-utilized and can't process traffic fast enough. You can see this in the output you included. First, in the show process cpu command the one second system load is "99.66%" used. Furthermore, PacketRouting is "65.18%" and eth 0/1 is "22.08%."
Then when you look at the interface Ethernet 0/1, the main thing to notice is the number of throttles. Throttles indicate the number of times an AOS device is unable to handle the incoming data rate and, as a result, proactively drops packets to get it to a rate that it can handle. (Please, see this post for further information on throttles and possible causes: https://supportforums.adtran.com/message/2058#2058)
Unfortunately, you are limited on how to troubleshoot this remotely, but one thing I would recommend is try to determine if a single host on the network is causing the problem. If you issue the show ip policy-sessions and show ip policy-stats commands you can see all the sessions going through the firewall. Often, if a single machine has a virus or is transmitting an abnormal amount of traffic, you will see a lot of policy-sessions in the firewall. Then you can create an access-control list (ACL) to block this IP address, which might provide you a window to get to the rest of the LAN.
I hope this makes sense, but please don't hesitate to reply to this post with any additional information or questions. I will be happy to help in any way I can.
Levi
I went ahead and flagged this post as “Assumed Answered.” If any of the responses on this thread assisted you, please mark them as either Correct or Helpful answers with the applicable buttons. This will make them visible and help other members of the community find solutions more easily. If you still need assistance, I would be more than happy to continue working with you on this - just let me know in a reply.
Levi
I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and select another in its place with the applicable buttons. If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.
Thanks,
Noor