Seeing Errored Seconds and Controlled Slip Seconds on TA-904

[Anonymous]

We recently starting using software release A5-03-00 on newly installed 900 IADs.  We are seeing Errored seconds and Controlled Slip Seconds on the network T1.  We are connecting to a 5000 shelf to a DS1 PPP 32port card.  All IADs using this software release on a number of different 5000 shelves is experiencing these errors.  Customers are also complaining of difficulty sending and receiving faxes.  We connect faxes on the FXS ports using mgcp.  We were not experiencing this on A02.07 versions of the software.  Are others seeing this problem and is there a solution short of going back to an earlier version of the software?

Thanks,

Bill Stoner

Daystar Communications

941-206-7845

[Anonymous]

Please send the output of debug isdn l2-formatted and check the event log to see if there are any errors.

You can try downgrading to A4.11.00.

[Anonymous]

Unified,

Not sure of the value of the isdn debug.  This problem is on the network T1 which is a PPP interface.  The T1 connected to the customer's PBX which isisdn/pri is not experiencing any error seconds or controlled slip seconds. Nothing in the event logs.

I think you're right on the downgrading, but was hoping that Adtran had addressed the problem if it is a software issue.

Thanks.

[Anonymous · Accepted solution]

Sorry I misread your post.

I thought the PRI had the slips.

[Anonymous]

No problem.  Thanks for taking the time to respond.

[Anonymous]

Mr. Stoner,

Thanks for posting!  I believe I have found another reported issue in A5.03.00 firmware that indicates CSS on the web GUI when doing continuous monitoring of the T-1 interface.  At this time, this appears to be a cosmetic/display issue and not indicative of actual clock slips on the T-1 interface.  Can you verify how and where exactly you are seeing the slips?  Also, do you know if A4.11.00 or A5.02.00 firmware exhibit the problem?

Thanks,

David

[Anonymous]

David,

I am seeing the slips on both the GUI and in the cli, however, I wondered if it was just a display error too. I don’t see this on version A4.11 and I haven’t tried A5.02. We were first alerted to this when a customer started complaining about fax calls not working on a B1 line on one of the FXS ports configured for MGCP. Again, not sure if the slips are real or just cosmetic or are a factor in the faxing issues too.

The other symptom is on the Call Quality Stats where we are seeing hundreds of thousands of “Out of Order” packets as noted in the screenshot below many calls. I’m not sure when this field was introduced though as I went from A2.07 to A5.03, but this is way not right if it is real.

Thanks for any insight you can provide.

Bill

Bill

Network Engineer

Daystar Communications

941-206-7845

[Anonymous]

Dave,

Since my last post I had a customer that was seeing the slips in his PBX.  I rolled the firmware back to A2.07.  The slips stopped on both the Network and DSX-1 interfaces on the IAD and the slips stopped on his PBX.  They were complaining of faxing issues too, but have not determined if the rollback has fixed this issue too.

This is offered as information in case Atran wishes to investigate the slipping issues.

Thanks,

Bill

[Anonymous]

Bill,

At this point, we believe the CSS counter increments because of web GUI monitoring of T-1 interfaces in A5.03.00 firmware.  This would show up in the CLI after the fact.  To test this, we could set one of your units to A5.03.00 firmware and disable HTTP and HTTPS on the unit, then issue a reboot.  If you still see CSS, we will likely need to troubleshoot directly on one of your units.  In this case, it would be better to handle this with a regular Technical Support ticket by calling 888-423-8726.  We could also look into the voice quality statistics and any fax issues.

Thanks!

David

[Anonymous]

I'm using a not yet released code by Adtran for the TA908 (R10-2-1) to fix another issue they admitted to, starting in A4.08. and seeing the same Errored Seconds & Controlled Slip Seconds in this version. My T-1 is clean. The issue seems to be software related. I 'm sure Adtran is aware of the matter but unsure what they are doing to address it.

[Anonymous]

David,

I believe that there is a problem in the system clock. The T1 clock parameter changed somewhere between A4.11 and A5-03 replacing the system clock with a choice of Line, Internal, or Through. This tells me something changed in that portion of the firmware enough to suspect there may be a problem with it.

When the PBX vendor said he was getting slips on his pri and after downgrading the IAD to A2.07, his slips stopped, I can only assume the problem is not just cosmetic. Adtran can set this up in their lab to test it if they want to pursue a fix. I will continue to run A2.07, which seems to be error free enough for my use.

Thanks for your interest and suggestions… I hope Adtran will take my observations and look into it.

Bill

Network Engineer

Daystar Communications

941-206-7845

[Anonymous]

Bill,

The only issue we have been able to reproduce with clock slips occurs only when the web GUI is used to monitor or change settings on the T-1 interface.  This issue is currently being investigated and will be fixed in a future firmware release. 

If you disable the web GUI, save, and then reboot the unit, you should not see this known problem.  The save and reboot is important because the unit will continue to report clock slips if the T-1 interface is accessed within the GUI just one time.  If you are still seeing clock slips, please let us know because we may have another issue to troubleshoot.

Thanks,

David

[Anonymous]

David,

I see what you are saying. Slips don’t occur until you go the T1 interface on the GUI. So although this is what triggers it, is it correct to say that once triggered the slips are real and will affect packet transmission on the wire?

Thanks,

Bill

Network Engineer

Daystar Communications

941-206-7845

[Anonymous]

Bill,

It appears that the CSS counts increasing is just a cosmetic issue.  It is not that actual slips are occurring on the T-1, they are just being falsely reported in the interface statistics.

Thanks,

David

[Anonymous]

David,

Just a follow up to our conversation. I put a T-Berd T1 tester on the circuit and observed that the transmit frequency on the Network T1 0/1 drops from 1544000 bps to 1543986 bps. Not a significant change, but is it enough cause slips? I don’t know. I get the same results when I place the system clock source on internal. To me this is indicative of the clock source for the Network T1 being changed from “Line” to “Internal” when you look at the T1 0/1 interface in the GUI. Note that this can be reversed or corrected by cycling the system clock source Primary and Secondary to internal and then back to Primary T1 0/1 and Secondary T1 0/2. Note also that since the T1 0/2 (DSX-1 ) interface is clock source is set to “through” its transmit frequency follows the Network clock source and I observed this on the T-Berd.

Again, just food for thought for the engineers looking into the problem.

Bill

Network Engineer

Daystar Communications

941-206-7845

[Anonymous]

Mr. Stoner,

Thank you for these additional details.  I've forwarded this information to the Design Engineer investigating the problem.  We will update this thread once we have more concrete information regarding a software fix for this issue.  Until then, preventing web GUI access to the unit should prevent the problem from occurring.

Thanks!

David

[Anonymous]

This is also a known issue with A5.03 - which is why it was suddenly removed for download.   We also ran into the pesky bug, so I asked Adtran about it, and an ETA:

"AOS R10.3.1 (the next release) is scheduled for Monday, 9/24.  The issue with slips only occurs if you choose the "continuous monitoring" function in the GUI on a T1 interface.  Simply logging into the GUI and doing anything else does not result in slips."

[Anonymous · Accepted solution]

Bill et al.,

I wanted to provide an update on this issue.  During our testing, we were able to verify that if the web GUI was accessed to view or modify settings on a T-1 interface, the clock source setting would change to internal and potentially cause clock slips (CSS).   This issue should be resolved with the R10.3.2 release which is now posted on our main website, www.adtran.com.  This issue will also be fixed in R10.5.0 and R10.4.1, when it is released.

Best regards,

David

[Anonymous]

Bill,

I went ahead and flagged this post as "Assumed Answered".  If the response on this thread assisted you, please mark it as Correct or Helpful as the case may be with the applicable buttons.  This will make it visible and help other members of the community find solutions more easily.  If you still need assistance, I would be more than happy to continue working with you on this - just let me know in a reply.

Thanks!

David

[Anonymous]

Sthenge,

I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and select another in its place with the applicable buttons.  If you still need assistance, we would be more than happy to continue working with you on this - just let us know in a reply.

Thanks,

David