NV3140 being used for a SIP Trunk
Issues are intermittent fast busy when calling out, and possibly some incoming calls failing. Secondary issue is the customer is trying to use a fax service across this, which we have recommended against, and faxes are all partial failures.
I added "ip ffe max-entries 500000" to the interfaces and it has greatly helped with cpu usage lowering from hitting 100% consistently to only rarely hitting as high as 92%.
The sip trunk is limited to 8 call paths and doing a sip debug it looks normal. I am not seeing SIP messages from any outside IP's or reasons for failed calls. This is leading me to believe the high cpu usage was causing issues and while ffe has helped it does not solve the problem with the config.
#show running-config
Building configuration...
! ADTRAN, Inc. OS version R13.5.1
! Boot ROM version R11.5.0
! Platform: NetVanta 3140, part number 1700341F1
! Serial number xxxxxxxxx
hostname xxxxxxx
enable password md5 encrypted xxxxxxx
clock timezone -5-Eastern-Time
ip subnet-zero
ip classless
ip routing
ipv6 unicast-routing
name-server xxxxxxx xxxxxxx
no auto-config
auto-config authname xxxxxx encrypted password xxxxxx
event-history on
no logging forwarding
no logging email
service password-encryption
username "admin" password encrypted "xxxxxx"
;
ip firewall
ip firewall stealth
no ip firewall alg msn
no ip firewall alg mszone
no ip firewall alg h323
no dot11ap access-point-control
qos map VoIP 0
match ip list SIP_port
set dscp 24
qos map VoIP 1
match ip list RTP_port
match dscp 46 ef
set dscp 46
interface gigabit-eth 0/1
description UPLINK
ip address xxx.xxx.218.135 255.255.255.192
ip ffe max-entries 500000
ip access-policy Private
no shutdown
media-gateway ip primary
interface gigabit-eth 0/2
no ip address
shutdown
interface gigabit-eth 0/3
description PBX
ip address 192.168.150.1 255.255.255.252
ip ffe max-entries 500000
ip access-group SIP_CPE_ACL in
qos-policy out VoIP
no shutdown
media-gateway ip primary
ip access-list standard SIP_NET_ACL
remark SIP Trunk to Voice Access Net
permit host xxx.xxx.172.244
permit host xxx.xxx.172.245
permit host 192.168.150.2
ip access-list standard VTY_SVC
remark ACL for VTY Service on Router
permit host xxx.xxx.247.133 log
permit host xxx.xxx.247.18 log
permit host xxx.xxx.247.4 log
ip access-list extended RTP_port
permit udp any range 1634 65535 any
ip access-list extended SIP_CPE_ACL
remark SIP Trunk to CPE
permit udp host 192.168.150.2 range 5060 5061 host 192.168.150.1
permit udp host 192.168.150.2 range 1634 32767 host 192.168.150.1
ip access-list extended SIP_port
permit udp any any eq 5060
ip policy-class Private
allow list self self
nat source list wizard-ics interface gigabit-ethernet 0/2 overload
nat source list wizard-ics interface gigabit-ethernet 0/3 overload
ip route 0.0.0.0 0.0.0.0 xxx.xxx.218.129
no tftp server
no tftp server overwrite
no http server
http secure-server
no snmp agent
no ip ftp server
no ip scp server
no ip sntp server
auto-link
auto-link server primary xxxxxx
auto-link server secondary xxxxxx
sip
sip udp 5060
no sip tcp
voice feature-mode network
voice forward-mode network
voice spre 1 *[12345689]X&
voice spre 2 *78&
voice spre 3 *77
voice spre 4 *75XX&
voice spre 5 *73
voice spre 6 *72&
voice spre 7 *70
voice spre 8 *67&
voice spre 9 *55
voice dial-plan 1 international 011-N$
voice dial-plan 2 local M11
voice dial-plan 3 local NXX-NXX-XXXX
voice dial-plan 4 local [01]911
voice dial-plan 5 local [01][235678]11
voice dial-plan 6 long-distance 1-NXX-NXX-XXXX
voice dial-plan 7 operator-assisted 00
voice dial-plan 8 operator-assisted 0[23456789]1[023456789]XXXXXXX
voice dial-plan 9 operator-assisted 0[23456789][023456789]XXXXXXXX
voice dial-plan 10 operator-assisted [01]411
voice dial-plan 11 specify-carrier 101XXXX$
voice codec-list Default_List
codec g711ulaw
codec g729
voice trunk-list CPE_TRK
trunk T30
voice trunk-list NET_TRK
trunk T10
voice trunk T10 type sip
description "Trunk to Voice Access Network SIP_245"
sip-server primary 10.195.1.4
outbound-proxy primary xxx.xxx.172.245
dial-string source to
hmr SIP_META out
register xxxxxx auth-name "xxxxxx" password encrypted "xxxxxx"
codec-list Default_List both
authentication username "xxxxxx" password encrypted "xxxxxx"
voice trunk T30 type sip
description "Trunk to CPE"
sip-server primary 192.168.150.2
grammar from host local
transfer-mode network
sip-server monitor
no shutdown
voice grouped-trunk NET
description "Route inside to outside"
trunk T10
accept $ cost 0
permit list CPE_TRK
!deny all other trunks
!deny all other ani
voice grouped-trunk CPE
description "Route outside to inside"
trunk T30
accept $ cost 0
permit list NET_TRK
!deny all other trunks
!deny all other ani
sip access-class ip "SIP_NET_ACL" in
sip qos dscp 24
ip rtp media-anchoring
hmr policy SIP_META
rule-set META_AUTH 10
hmr rule-set META_AUTH
message-rule CONTACT_HEADER message-type any 10
modify header contact position first match-value /sip:.*@/ new-value /sip:xxxxxx@/ 10
message-rule INVITE message-type request 20
set private-variable INVITE_PAI header sip-req-uri position first match-value "/^INVITE /" new-value true 10
message-rule ADD_PAI message-type request 30
match private-variable INVITE_PAI match-value true
add header p-asserted-identity position first new-value /<sip:xxxxxx@10.195.1.4>/ 10
line con 0
no login
line telnet 0 4
login local-userlist
shutdown
ip access-class VTY_SVC in
line ssh 0 4
login local-userlist
no shutdown
ip access-class VTY_SVC in
ntp server xxxxxx
ntp server xxxxxx
0 | Idle | 0 | R | 2,593,785,152 | 0 | 867 | 86.78 |
1 | init | 0 | W | 1,600,000 | 0 | 0 | 0 |
2 | kthreadd | 0 | W | 0 | 0 | 0 | 0 |
3 | ksoftirqd/0 | 0 | W | 190,000 | 0 | 0 | 0 |
4 | kworker/0:0 | 0 | W | 0 | 0 | 0 | 0 |
5 | kworker/u:0 | 0 | W | 10,000 | 0 | 0 | 0 |
6 | rcu_kthread | 99 | W | 16,930,000 | 0 | 0 | 0 |
7 | khelper | 0 | W | 0 | 0 | 0 | 0 |
8 | kworker/u:1 | 0 | W | 0 | 0 | 0 | 0 |
85 | sync_supers | 0 | W | 200,000 | 0 | 0 | 0 |
87 | bdi-default | 0 | W | 0 | 0 | 0 | 0 |
89 | kblockd | 0 | W | 0 | 0 | 0 | 0 |
184 | rpciod | 0 | W | 0 | 0 | 0 | 0 |
185 | kworker/0:1 | 0 | W | 170,000 | 0 | 0 | 0 |
209 | khungtaskd | 0 | W | 10,000 | 0 | 0 | 0 |
214 | kswapd0 | 0 | W | 0 | 0 | 0 | 0 |
278 | fsnotify_mark | 0 | W | 0 | 0 | 0 | 0 |
284 | nfsiod | 0 | W | 0 | 0 | 0 | 0 |
292 | crypto | 0 | W | 0 | 0 | 0 | 0 |
375 | mtdblock0 | 0 | W | 0 | 0 | 0 | 0 |
380 | mtdblock1 | 0 | W | 0 | 0 | 0 | 0 |
385 | mtdblock2 | 0 | W | 0 | 0 | 0 | 0 |
390 | mtdblock3 | 0 | W | 0 | 0 | 0 | 0 |
395 | mtdblock4 | 0 | W | 0 | 0 | 0 | 0 |
400 | mtdblock5 | 0 | W | 0 | 0 | 0 | 0 |
405 | mtdblock6 | 0 | W | 0 | 0 | 0 | 0 |
410 | mtdblock7 | 0 | W | 0 | 0 | 0 | 0 |
432 | edac-poller | 0 | W | 0 | 0 | 0 | 0 |
439 | rcS | 0 | W | 10,000 | 0 | 0 | 0 |
470 | ubi_bgt0d | 0 | W | 0 | 0 | 0 | 0 |
479 | ubi_bgt1d | 0 | W | 20,000 | 0 | 0 | 0 |
486 | ubifs_bgt0_0 | 0 | W | 0 | 0 | 0 | 0 |
488 | ubifs_bgt1_0 | 0 | W | 0 | 0 | 0 | 0 |
605 | runstarter | 0 | W | 0 | 0 | 0 | 0 |
611 | syslogd | 0 | W | 0 | 0 | 0 | 0 |
617 | amom | 0 | W | 2,020,000 | 0 | 0 | 0 |
618 | AIPC Session Th | 0 | W | 0 | 0 | 0 | 0 |
619 | AMOM network th | 0 | W | 3,450,000 | 0 | 0 | 0 |
632 | khubd | 0 | W | 0 | 0 | 0 | 0 |
751 | starter | 0 | W | 0 | 0 | 0 | 0 |
753 | Init | 1 | W | 420,000 | 0 | 0 | 0 |
755 | InterruptThread | 52 | W | 0 | 0 | 0 | 0 |
756 | KCall | 49 | W | 2,874,060,000 | 0 | 41 | 4.37 |
757 | PC Config | 8 | R | 694,680,000 | 0 | 15 | 1.14 |
758 | PacketRouting | 45 | W | 35,940,000 | 0 | 0 | 0 |
759 | Timers | 47 | W | 211,220,000 | 0 | 0 | 0 |
760 | I2C | 47 | W | 0 | 0 | 0 | 0 |
761 | CallControlQue~ | 38 | W | 5,120,000 | 0 | 0 | 0 |
762 | Thread Pool | 5 | W | 50,000 | 0 | 0 | 0 |
763 | FrontPanel | 44 | W | 44,530,000 | 0 | 5 | 0.51 |
764 | RSTP | 44 | W | 0 | 0 | 0 | 0 |
765 | sec | 47 | W | 1,125,030,000 | 0 | 15 | 1.14 |
766 | con0 | 33 | W | 0 | 0 | 0 | 0 |
767 | gigTSEC | 47 | W | 1,037,060,000 | 0 | 0 | 0 |
768 | ICP Session | 9 | W | 110,000 | 0 | 0 | 0 |
769 | RSTP | 44 | W | 16,430,000 | 0 | 0 | 0 |
770 | RSTP-BG | 43 | W | 0 | 0 | 0 | 0 |
771 | MLD Thread | 7 | W | 0 | 0 | 0 | 0 |
772 | RouteTableTick | 7 | W | 2,900,000 | 0 | 0 | 0 |
773 | RouteTableTick | 7 | W | 2,880,000 | 0 | 0 | 0 |
774 | IGMPTick | 7 | W | 2,400,000 | 0 | 0 | 0 |
775 | IGMP-Receiver | 7 | W | 0 | 0 | 0 | 0 |
776 | IP Events | 28 | W | 3,710,000 | 0 | 0 | 0 |
777 | tcptimer | 26 | W | 530,000 | 0 | 0 | 0 |
778 | tcpinp | 26 | W | 6,030,000 | 0 | 0 | 0 |
779 | tcpout | 26 | W | 5,310,000 | 0 | 0 | 0 |
780 | DnsClient | 20 | W | 2,020,000 | 0 | 0 | 0 |
781 | DnsProxy | 20 | W | 300,000 | 0 | 0 | 0 |
782 | DnsTable | 20 | W | 140,000 | 0 | 0 | 0 |
783 | PhoneManagerQu~ | 42 | W | 6,090,000 | 0 | 0 | 0 |
784 | SnmpThread | 7 | R | 47,460,000 | 0 | 0 | 0 |
785 | WWW | 23 | W | 58,980,000 | 0 | 0 | 0 |
786 | MediaConnectio~ | 40 | W | 10,870,000 | 0 | 0 | 0 |
787 | FTPServer List~ | 6 | W | 0 | 0 | 0 | 0 |
788 | SMTP Client | 20 | W | 0 | 0 | 0 | 0 |
789 | SNTP Client | 23 | W | 0 | 0 | 0 | 0 |
790 | CPU Usage | 9 | R | 2,514,200,000 | 0 | 35 | 3.33 |
791 | CLIInjectQ | 7 | W | 0 | 0 | 0 | 0 |
793 | OSPF | 7 | W | 0 | 0 | 0 | 0 |
795 | RipOut | 7 | W | 1,070,000 | 0 | 0 | 0 |
796 | RipIn | 7 | W | 0 | 0 | 0 | 0 |
797 | AUTOLINKQ | 5 | W | 280,000 | 0 | 0 | 0 |
798 | HttpClientQ | 7 | W | 100,000 | 0 | 0 | 0 |
799 | SIP_Stack | 39 | W | 8,960,000 | 0 | 0 | 0 |
800 | SIP Registrati~ | 37 | W | 17,630,000 | 0 | 0 | 0 |
801 | ntpd | 23 | W | 10,420,000 | 0 | 0 | 0 |
802 | ActiveQueueDea~ | 4 | W | 60,000 | 0 | 0 | 0 |
803 | DHCPv6 | 35 | W | 0 | 0 | 0 | 0 |
804 | RvSipProc0 | 40 | W | 23,810,000 | 0 | 0 | 0 |
805 | UDP Relay | 23 | W | 0 | 0 | 0 | 0 |
806 | PacketCapture | 5 | W | 3,730,000 | 0 | 0 | 0 |
807 | DHCP Server | 35 | W | 0 | 0 | 0 | 0 |
808 | Flow Meter Log~ | 21 | W | 2,100,000 | 0 | 0 | 0 |
809 | OSPFv3 | 7 | W | 0 | 0 | 0 | 0 |
810 | TWAMP-Control | 7 | W | 0 | 0 | 0 | 0 |
811 | TWAMP-Test | 20 | W | 0 | 0 | 0 | 0 |
812 | UDP In | 43 | W | 2,850,000 | 0 | 0 | 0 |
The CPU is over 86% idle in your example, this doesn't seem like excessive CPU usage. Note that "idle" in the listing is not usage, it's actually the percentage of non-usage.
Fax can be made quite reliable on most networks with the proper tweaks, typically T.38 will help but it needs to be supported by the other endpoint as well.
I am seeing CPU spikes to 100%, and the times match to when the user reports dropped calls or one way talk path on the call. Some of these I have tracked back to a occurring during fax attempts.
I was able to reproduce 100% cpu just by sending 2-3 ssh session attempts at once. Even with mgmt heavily restricted it looked like general port scanning and attempts were causing the high cpu.
We replaced the 3140 with another 3140 using the exact same config. Even trying with dozens of login attempts the highest I can get it is 86%. While that still seems high it should not be an issue for the customer. Right now I am hoping it was just a bad CPU/board.
Another 3140 that I am using for SIP to PRI with almost the same config barely reacts to ssh attempts, cpu never going over 50%. The difference in the two are the NAT rules since the SIP to PRI does not allow for any nat or data traffic from the lan. We may have room for improvement in NAT and firewall rules.