ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback

[davshark]

Received this error log and am not sure if this indicates a hardware failure or reset occurred.  Shortly after the entire network went down and restored by itself after two hours of down time.

The switch had the correct time implemented and NTP configured.  Not sure how it would reset, Maybe battery fail on the device reset the time?  No configuration loss was found.  The system went down about 1:30PM and restored 3pm.

The Gigabit links are connected to other switches.  there are 11 switches on the network and this is a core switch were half of them combine.

I did a hard reset of the switch about 2pm (power plug)

Any ideas and help would be useful.   I am interested in the failed to process Link Notify Call Back.  I am curious if a Spanning Tree Election was triggered, but I don't see that in the logs (or if it would be).  The syslog is configured log everything at the debug level.  And the firmware was updated on this switch about a months ago.

This is an Adtran NetVanta 1238 PoE Switch

Firmware Version	18.01.06.00
Part Number	1700599G1

Memory	Total Heap: 47,496,176 Bytes Free Heap: 40,115,184 Bytes
CPU Utilization	System Load: 35.32% 1 Min Avg Load: 37.34%     5 Min Avg Load: 38.34% Min Load: 0%     Max Load: 100% Context Switch Load: 0.19%
File System	Total: 6,697,320 Bytes Used: 4,686,696 Bytes Free: 2,010,624 Bytes

00 duplex:1

1970.01.01 00:00:05 ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback: unit:0 port:52 status:1 enable:1 speed:1000 duplex:1

1970.01.01 00:00:05 ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback: unit:0 port:41 status:1 enable:1 speed:100 duplex:1

1970.01.01 00:00:06 ETHERNET_INTERFACE.Switch Failed to process Link Notify Callback: unit:0 port:52 status:0 enable:1 speed:1000 duplex:1

1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/1 link up

1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/4 link up

1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/8 link up

1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/42 link down

1970.01.01 00:00:06 ETHERNET_INTERFACE.swx 0/47 link down

1970.01.01 00:00:06 ETHERNET_INTERFACE.giga-swx 0/1 link down

1970.01.01 00:00:06 ETHERNET_INTERFACE.giga-swx 0/2 link down

2014.05.07 01:01:25 OPERATING_SYSTEM.USER Created user 'admin'

2014.05.07 01:01:25 ETHERNET_INTERFACE.swx 0/42 link up

2014.05.07 01:01:26 ETHERNET_INTERFACE.giga-swx 0/1 link up

2014.05.07 01:01:26 ETHERNET_INTERFACE.swx 0/47 link up

2014.05.07 01:01:27 ETHERNET_INTERFACE.giga-swx 0/2 link up

2014.05.07 01:01:27 ETHERNET_INTERFACE.giga-swx 0/4 link up

2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/1 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/4 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/8 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/42 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.swx 0/47 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.giga-swx 0/1 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.giga-swx 0/2 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.giga-swx 0/4 changed state to up

2014.05.07 01:01:41 INTERFACE_STATUS.vlan 1 changed state to up

2014.05.07 01:01:41 OPERATING_SYSTEM Cold Start

2014.05.07 01:01:47 POWER_OVER_ETHERNET.swx 0/10 changed state from SEARCHING to DELIVERING

2014.05.07 01:01:47 POWER_OVER_ETHERNET.swx 0/10 classified to IEEE Class 0

2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/16 changed state from SEARCHING to DELIVERING

2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/16 classified to IEEE Class 0

2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/17 changed state from SEARCHING to DELIVERING

2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/17 classified to IEEE Class 0

2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/19 changed state from SEARCHING to DELIVERING

2014.05.07 01:01:48 POWER_OVER_ETHERNET.swx 0/19 classified to IEEE Class 0

2014.05.07 01:01:49 POWER_OVER_ETHERNET.swx 0/30 changed state from SEARCHING to DELIVERING

2014.05.07 01:01:49 POWER_OVER_ETHERNET.swx 0/30 classified to IEEE Class 0

2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/10 link up

2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/16 link up

2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/30 link up

2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/17 link up

2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/10 changed state to up

2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/16 changed state to up

2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/17 changed state to up

2014.05.07 01:01:49 INTERFACE_STATUS.swx 0/30 changed state to up

2014.05.07 01:01:49 ETHERNET_INTERFACE.swx 0/19 link up

2014.05.07 01:01:50 INTERFACE_STATUS.swx 0/19 changed state to up

2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/10 link down

2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/16 link down

2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/30 link down

2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/17 link down

2014.05.07 01:01:56 ETHERNET_INTERFACE.swx 0/19 link down

2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/10 changed state to down

2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/16 changed state to down

2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/17 changed state to down

2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/19 changed state to down

2014.05.07 01:01:56 INTERFACE_STATUS.swx 0/30 changed state to down

2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/16 link up

2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/10 link up

2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/17 link up

2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/30 link up

2014.05.07 01:01:58 ETHERNET_INTERFACE.swx 0/19 link up

2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/10 changed state to up

2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/16 changed state to up

2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/17 changed state to up

2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/19 changed state to up

2014.05.07 01:01:58 INTERFACE_STATUS.swx 0/30 changed state to up

2014.05.07 14:31:30 ETHERNET_INTERFACE.giga-swx 0/4 link down

2014.05.07 14:31:31 INTERFACE_STATUS.giga-swx 0/4 changed state to down

2014.05.07 14:55:08 ETHERNET_INTERFACE.giga-swx 0/4 link up

2014.05.07 14:55:09 INTERFACE_STATUS.giga-swx 0/4 changed state to up

2014.05.07 15:00:22 ETHERNET_INTERFACE.giga-swx 0/3 link up

2014.05.07 15:00:22 INTERFACE_STATUS.giga-swx 0/3 changed state to up

2014.05.07 15:15:57 ETHERNET_INTERFACE.swx 0/47 link down

2014.05.07 15:15:58 INTERFACE_STATUS.swx 0/47 changed state to down

2014.05.07 15:15:59 ETHERNET_INTERFACE.swx 0/47 link up

2014.05.07 15:15:59 INTERFACE_STATUS.swx 0/47 changed state to up

2014.05.07 15:46:38 ETHERNET_INTERFACE.swx 0/47 link down

2014.05.07 15:46:39 INTERFACE_STATUS.swx 0/47 changed state to down

2014.05.07 15:46:40 ETHERNET_INTERFACE.swx 0/47 link up

2014.05.07 15:46:40 INTERFACE_STATUS.swx 0/47 changed state to up

2014.05.07 15:50:35 ETHERNET_INTERFACE.swx 0/47 link down

2014.05.07 15:50:35 INTERFACE_STATUS.swx 0/47 changed state to down

2014.05.07 15:50:37 ETHERNET_INTERFACE.swx 0/47 link up

2014.05.07 15:50:37 INTERFACE_STATUS.swx 0/47 changed state to up

2014.05.07 15:51:42 ETHERNET_INTERFACE.swx 0/47 link down

2014.05.07 15:51:43 INTERFACE_STATUS.swx 0/47 changed state to down

2014.05.07 15:51:44 ETHERNET_INTERFACE.swx 0/47 link up

2014.05.07 15:51:44 INTERFACE_STATUS.swx 0/47 changed state to up

[Anonymous]

:

Thank you for asking this question in the support community.  How long was the switch up?  The "link notify callback" event messages, appeared to have happened immediately after the switch was booting up.  Is this the only time you have had a similar network outage like this?  Do you know what brought the network back?  It appears that a hard reboot on the ADTRAN did not make any difference.  If you get a chance, to reply with a current copy of the configuration (please, remember to remove any information that may be sensitive to the organization), I will be happy to review it for you.

Levi

[davshark]

Thank you very much for the reply.  We basically have 11 switches a primary VLAN for all PCs and server and a VLAN for VOIP phones.  THe PBX, Router, and 4 other switches connect to this switch below.   I had a "sniffer" port enabled on Trunk Port 47.  Router is on 1, PBX (VLAN) is on G1.

We have had the switches go crazy 4 times in about 4 months.  This is first time i have seen this error message on any of the switches.  The first two times logging was not enabled so it may have occurred. 

The switch had been on for 4 weeks since it's last power cycle.  This network outage occurred 5 weeks ago, then occurred one week before that.  Strangly they seem to be similar parts of the day (12 to 2pm).

I don't know what brought the network.  Previously we had restarted some or all of the switches and normally after 30 minutes it came back up.  But I couldn't tell if was the result of anything we did.  This most recent time we were changing wires and it took two hours to come back, it normally was 45min to 1 hour.  So maybe our attempts to repair and reset delays to system from coming back up.

I though it was spanning tree flipping out and closing all traffic for a while, but i don't see any indication in the logs that it was even doing a spanning  tree election.  In some cases PCs were not able to ping the switch they were directly connected to.  And some times some people on a switch were fine and other completely dropped... (this was on multiple switches, I'm focusing on this one because it has the unique error message and it is a core switch.)

!

!

! ADTRAN, Inc. OS version 18.01.06.00

! Boot ROM version 17.03.02.SB

! Platform: NetVanta 1238 PoE, part number 1700599G1

! Serial number #################

!

!

hostname "MDFSwitch01"

enable password ##############

!

clock timezone -5-Eastern-Time

ip default-gateway ##.##.1.1

!

!

no auto-config

!

event-history on

logging forwarding on

logging forwarding priority-level debug

logging forwarding receiver-ip ###.50.1.207

no logging email

!

no service password-encryption

!

username "########" password "##########"

!

banner motd #

                 ****** Important ******

Enable and Telnet passwords are configured to "#######".

Please change them immediately.

The vlan 1 interface is enabled with an address of ###.10.10.1

Telnet access is also enabled.

#

!

banner exec <

MDF Switch 1 >

<

!

!

!

!

!

!

no dot11ap access-point-control

!

!

!

!

!

!

!

!

vlan 1

  name "Default"

!

vlan 200

  name "AllworxPhones"

!

interface switchport 0/1

  description Router / PBX

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1,200

!

interface switchport 0/2

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/3

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/4

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/5

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/6

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/7

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/8

  description ASA5505 Firewall

  speed 100

  spanning-tree edgeport

  no shutdown

!

interface switchport 0/9

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/10

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/11

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/12

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/13

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/14

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/15

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/16

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/17

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/18

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/19

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/20

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/21

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/22

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/23

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/24

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/25

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/26

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/27

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/28

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/29

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/30

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/31

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/32

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/33

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/34

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/35

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/36

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/37

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/38

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/39

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/40

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/41

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/42

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/43

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/44

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/45

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/46

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/47

  spanning-tree edgeport

  no shutdown

  switchport voice vlan 200

  switchport voice-signaling vlan 200

!

interface switchport 0/48

  description Allworx Phone System

  spanning-tree edgeport

  no shutdown

!

!

interface gigabit-switchport 0/1

  description Allworx Phone System

  spanning-tree edgeport

  no shutdown

  switchport access vlan 200

!

interface gigabit-switchport 0/2

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1,200

!

interface gigabit-switchport 0/3

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1,200

!

interface gigabit-switchport 0/4

  spanning-tree edgeport

  no shutdown

  switchport mode trunk

  switchport trunk allowed vlan 1,200

!

!

!

interface vlan 1

  ip address  ###.###.1.253  255.255.0.0

  no shutdown

!

interface vlan 200

  no ip address

  shutdown

!

!

!

no tftp server

no tftp server overwrite

ip http server

ip http secure-server

no ip snmp agent

no ip ftp server

ip ftp server default-filesystem flash

no ip scp server

no ip sntp server

!

!

!

!

!

!

!

!

!

line con 0

  no login

!

line telnet 0 4

  login

  password ########

  no shutdown

line ssh 0 4

  login local-userlist

  no shutdown

!

sntp server #############

!

!

!

end

[Anonymous]

:

Thank you for replying with this information.  I think ADTRAN may be able to assist you better with troubleshooting if you open a Technical Support ticket about this. 

- Open a webticket by clicking on this link: Create a Service Request

- Open a ticket by emailing support@adtran.com

- Open a ticket by phone by calling 1-888-423-8726

Levi

[Anonymous]

-

I went ahead and flagged this post as "Assumed Answered". If any of the responses on this thread assisted you, please mark them as Correct or Helpful as the case may be with the applicable buttons. This will make them visible and help other members of the community find solutions more easily. If you have any additional information on this that others may benefit from, please come back to this post to provide an update. If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.

Thanks,

Noor