Netvanta 4430 WAN Encapsulation Link Status Down

[Anonymous]

Hi all,

Our bandwidth monitoring tool shows we are only using up to 4.5mb at a certain site while the total bandwidth should be 6.0mb; initially I figured that perhaps 1.5mb was dedicated for VOIP.  While checking the Netvanta 4430 router at the location I noticed that T1 3/1 Link is Down under WAN Encapsulation Interfaces.  I don't know what that means as there are no alarms at the SmartJack and our monitoring tool shows the link to be up.

So I did a little bit more digging and compared configurations and statistics for each of the T1 interfaces.  While configurations for all 4 interfaces are the same, statistics are different for T1 3/1 when compared to the other 3.  The highlighted area shows a big difference in the Input/output rate of T1 3/1.  Is someone able to explain the difference and advise if this might be a problem?  I rebooted the router yesterday to see if that would "fix" it but nothing changed.

Appreciate the help.

[Anonymous]

:

Thank you for asking this question in the Support Community.  From the screenshots you supplied, it appears the Frame Relay interface is down for T1 3/1.  When you get a change will you reply with a copy of the current configuration? (Please, remember to remove any information that may be sensitive to the organization).

Also, would you reply with the output from the following commands:

show interfaces

sho frame-relay lmi

show frame-relay pvc

Please, do not hesitate to reply to this email with any additional questions or information.  I will be happy to help in any way I can.

Levi

[Anonymous]

Thank you Levi.  Attached is the information requested.  In the showinterfaces.docx  I highlighted an item relating to T1 3/1 that kind of stuck out as it shows "link state is ADD_RX" while the other 3 T1s all show "link state is up".

[jayh]

It looks like you're up on the physical layer but the other end my not have your T1 3/1 in the bundle or be misconfigured.  If you have access to the other end of the circuit, take a look there.  If not, ask the carrier to have a look. 

Some other CLI tools you can use are:

show frame-relay multilink

(You'll probably see that T1 3/1 is up with link-state down)

debug frame-relay multilink t1 3/1

   and compare with

debug frame-relay multilink t1 3/2

You should see hellos sent outbound (O) every ten seconds followed by a hello_ack inbound (I) in both directions every ten seconds.

My suspicion is that the other end hasn't put the first T1 into the multilink bundle.  Did you originally have a single frame-relay T1 on 3/1 and then grow it to a 4-loop bundle? 

[Anonymous]

I got the following from running the commands you suggested:

show frame-relay multilink

Bundle: frame-relay 1 is UP; class A bundle

        Near-end BID:
MFR1; Far-end BID: Multilink0/4/0/0/6

  Bundle links:

    Link thru t1 3/1
is UP, link state is ADD_RX

    Link thru t1 3/2
is UP, link state is UP

    Link thru t1 3/3
is UP, link state is UP

    Link thru t1 3/4
is UP, link state is UP

t1 3/1 Debug

1. 2014.03.06 15:47:08 FRAME_RELAY.MULTILINK_INT t1 3/1 (I):
   msg=ADD_LINK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX
2. 2014.03.06 15:47:08 FRAME_RELAY.MULTILINK_INT t1 3/1 (O):
   msg=ADD_LINK_ACK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX
3. 2014.03.06 15:47:15 FRAME_RELAY.MULTILINK_INT t1 3/1 (O):
   msg=ADD_LINK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX
4. 2014.03.06 15:47:22 FRAME_RELAY.MULTILINK_INT t1 3/1 (I):
   msg=ADD_LINK, Link=t1 3/1 1, Bundle=MFR1, BL state=ADD_RX      

t1 3/2 Debug

1. 2014.03.06 15:56:07 FRAME_RELAY.MULTILINK_INT t1 3/2 (I):
   msg=HELLO, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
2. 2014.03.06 15:56:07 FRAME_RELAY.MULTILINK_INT t1 3/2 (O):
   msg=HELLO_ACK, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
3. 2014.03.06 15:56:10 FRAME_RELAY.MULTILINK_INT t1 3/2 (O):
   msg=HELLO, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
4. 2014.03.06 15:56:10 FRAME_RELAY.MULTILINK_INT t1 3/2 (I):
   msg=HELLO_ACK, Link=t1 3/2 1, Bundle=MFR1, BL state=UP
5. 2014.03.06 15:56:16 FRAME_RELAY.MULTILINK_INT t1 3/2 (I):
   msg=HELLO, Link=t1 3/2 1, Bundle=MFR1, BL state=UP

No Hellos for interface T1 3/1; I'm wondering if it is a problem on our device or the ISP.

[jayh]

juad wrote:

No Hellos for interface T1 3/1; I'm wondering if it is a problem on our device or the ISP.

Most likely the ISP.  Could be the transport carrier but I would expect to see errors if this were the case.

To test, during a period of low traffic swap the cables between T1 3/1 and T1 3/2 and see if the problem follows the circuit or stays on T1 3/1. Note that can take up to a minute for frame-relay circuits to handshake and train.

Another thought, have you or the ISP changed router hostnames or multilink identifier names since the equipment was first installed? 

We had a really strange similar scenario a while back with multilink PPP.  Cisco uses the router hostname as an "endpoint identifier" to ensure that all of the links in a bundle go to the same place.  We had changed the hostname on a customer router which caused no immediate problems.  Some weeks later one of the T1s in the bundle went down.  When it came back up it refused to join the multilink bundle because the "endpoint identifier" was different.  Very subtle problem which we struggled with for a while until debugs clued us in.  We wound up having to shut/no-shut all of the links and let the bundle sort out that the other end now had a different identifier.

You may need to do something similar.  If your swap test results in BOTH 3/1 and 3/2 now being unhappy, then you'll need to tear down all of the links and let the bundle negotiation start over.  Obviously best done after hours.

[Anonymous]

I contacted the ISP and provided them the same information.  They said the circuit tested good to the Smartjack and therefore was a problem on our end since they could not loop to the CSU.  I wonder if a shut/ no shut on interface t1 3/1 might resolve the issue...or would I have to do that for the other 3?  Would doing a shut/ no shut on just that interface cause any problems for the rest?

[jayh · Accepted solution]

juad wrote:

I contacted the ISP and provided them the same information.  They said the circuit tested good to the Smartjack and therefore was a problem on our end since they could not loop to the CSU.  I wonder if a shut/ no shut on interface t1 3/1 might resolve the issue...or would I have to do that for the other 3?  Would doing a shut/ no shut on just that interface cause any problems for the rest?

The physical link looks good, so I would expect their test to the smartjack would be fine.  Their loopback testing would have had to take that T1 out of service so it has already been "bounced" and I suspect that a shut/no-shut isn't likely to help.

You can indeed do a shut/no-shut on that link without affecting traffic.  Before doing so verify that this T1 isn't the only external timing source.  When you shut it down you'll want to take clock from  a different T1.  "Show system" should list a primary and secondary clock source.  Both should be configured to different T1s in the bundle so that when you shut down 3/1 there is another T1 clock source available.

A shut/no-shut may fix it simply by forcing both sides to renegotiate, but likely not.

If this does not fix it, the next step as I suggested earlier would be to temporarily swap the affected T1 with one of the others.  Do this during a period of low traffic.  If the problem remains on T1 3/1 then the problem would seem to be within the Adtran (but it isn't obvious where).  If the problem moves to the other physical interface on the Adtran then contact the carrier again.  Let them know that it isn't a physical issue but something with the frame-relay configuration on their end.  Note that it may take up to 60 seconds for frame-relay LMI to settle down when testing.

Swap the links back after testing so that your documentation is correct with regard to circuit IDs for future repair tickets.

[Anonymous]

Our equipment vendor did the shut/ no shut on T1 3/1 interface and that didn't fix it.  He found no problems with the device so I swapped the cable with T1 3/2.  T1 3/1 came back online and the problem rolled over to T1 3/2.  I contacted the ISP again.  Hopefully they can fix it soon.

[Anonymous]

Just a quick follow-up.  Our ISP found the issue on their network and was able to correct the problem.  Thanks for all the help.