NV1238 High CPU utilization

[Anonymous]

When I show processes cpu on a NV1238 PoE, we're constantly seeing 48-60%+ load for all three counters.  The switch is a couple months old and seems to have exhibited this behavior the whole time.  It's running ver 18.01.04 firmware.

It has only six ports open and almost all ports use PoE.  Twelve ports are IP security cameras and they constantly stream video to a recording server on giga-swx 0/1.  The remaining 100Mbps devices are phones and phone server equipment with fairly low usage.  There is an LACP trunk using giga-swx 0/3 and 0/4 to our NV1638 core switch.  Giga-swx 0/2 is a VLAN trunk port going to a VMware server (primarily used for voice applications, which is why it's on the 1238 with the phones).

No one is complaining about any symptoms, I just want to see what's up with this.

I tried shutting down one of the LACP ports, then the other, just to see what happens.  But there's no change.  No reason to suspect LACP; just trying stuff.  Also, I tried shutting down the camera ports and the camera server port to see if eliminating the bulk of this switch's traffic would have an effect.  But that made no difference either.

When I show processes cpu, the big offender is bcmLINK.0 and it's always around 30-45% load.  I'm not sure what this task is, but I think it's the culprit.  See the output example (TXT file attached).  CFG attached too.

Thanks in advance!

--

CJ

[Anonymous]

CJ:

Thank you for your questions.  I will review your output and configuration and follow up with you by close of business tomorrow.  Do not hesitate to reply to this post with additional information or questions before then.

Levi

[Anonymous]

CJ:

The "bcmLINK.0" is what the switch uses for a linkscan thread.  It is responsible for polling the physical interfaces and updating status.  One reason for this to be high could be if there are a lot of changes constantly happening on the network.  I would recommend removing the commands "switchport access vlan 7" from each of the "phone" ports.  I do not recommend having "voice vlan" and "access vlan" on the same interface.

Levi

[Anonymous]

I removed "access vlan" from all phone ports.  I wrote the config and reloaded (just for good measure).  Phones are now tagging VLAN 7; everything is functioning okay.  But the system load is the same, with task bcmLINK.0 remaining the culprit.  What should we try next?

[Anonymous]

FWIW, I tried unplugging all patch cables. Utilization and the offending task remained, even with all patch cords disconnected.

[Anonymous]

CJ:

Thank you for the updated information.  I will see if I can test and reproduce these results in our lab with your configuration to see why the bcmLINK is high with nothing plugged in.  I'll get back to you by close of business tomorrow with the results.

Levi

[Anonymous · Accepted solution]

CJ:

Please open a ticket with Technical Support so we can troubleshoot these symptoms further. You can create a ticket in several ways:

- Over the phone by calling 888-423-8726

- Emailing support@adtran.com

- Opening a webticket on the Adtran website

Levi

[Anonymous]

Will do. Thanks for your efforts, sir.

[mj_data]

did you all figure out what the issue was on this?  im seeing this same behavior, with the jacked up processes being "bcmARL.0" and "bcmLINK.0".  whats odd is that this switch is doing nothing except basic layer 2 switching, am confused by it ??? am about to start updating the software, please post what ended up being the problem.  thanks

[Anonymous]

We upgraded software and there was no change. We're not really seeing any symptoms though, so we just put it out of mind for the time being.

[mj_data]

Levi,

I have a netvanta 1234 switch that is having similiar issues, with the exception being that my "bcmLINK.0" is only around 6%, while my bcmARL.0 is running around 25%.  Can you elaborate on what the bcmARL process handles?  I saw your summary of bcmLINK being a linkscan thread.  This switch is acting as a media converter basically, and doing only L2 switching.  MM fiber in, gig copper out, and thats it?  Also, the sfp that im using is from cisco, could that be an issue?  When the 1234 recognized, initialized, and ran it free of errors i assumed it was ok.  Thank you for the help,

Mike

[mj_data]

thanks for that tidbit, will save me from upgrading only to still see high cpu

[Anonymous]

:

The bcmARL.0 process monitors changes to the MAC table and maintains a copy of what the hardware table is.  On the Netvanta 123x version 1 products it is common for this process to be high as it takes a lot of CPU resources on those products.

Levi

[johnyarbi]

Did you ever get this resolved?  I have a client seeing the same issue with bcmLINK.0 but they are also seeing network performance issues.  We are getting ready to replace the switches with newer gigabit models (for other reasons) but I was curious if there was an answer for this or if it is just a fact of life with these 1238's.  Thanks!

[Anonymous]

No, not really. In our case, we're not really doing anything with high throughput on our 1200 switches. Everything pretty much stays in the 1600 series for that. So we just kinda forgot it.

Chris