TA 908e "DM/CSS/ES 15 min threshold exceeded"

[jonathanblack]

I have a couple of 908e units configured with data in on ETH 0/1, and PRI connections on NET 0/3 and 0/4 to legacy IVR equipment.  We have configured SIP service with a couple of carriers over the data connection.  This is working. 

However, we are getting a lot of these "threshold exceeded" errors.  Here's a sampling:

2013.09.25 12:45:03 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 12:46:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 12:46:25 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 12:46:30 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 12:46:43 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:00:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:01:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 13:01:41 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:01:54 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 13:02:07 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:16:09 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:16:13 T1.t1 0/4 DM 15 min threshold exceeded, ES 15 min threshold exceeded

2013.09.25 13:16:26 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:30:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:31:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 13:31:28 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:31:37 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 13:31:51 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 13:46:03 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 13:46:09 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 13:46:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 13:47:02 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:00:03 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:01:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:01:21 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:01:32 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:01:34 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 14:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:16:10 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:16:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:16:45 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:16:58 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 14:30:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:31:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:31:17 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 14:31:40 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:32:10 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 14:46:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 14:46:18 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 14:46:28 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 14:46:42 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:00:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:00:56 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 15:01:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:01:53 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 15:02:06 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:15:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:16:11 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 15:16:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:16:25 T1.t1 0/3 CSS 15 min threshold exceeded, DM 15 min threshold exceeded

2013.09.25 15:30:03 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:31:05 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 15:31:13 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:31:37 T1.t1 0/4 DM 15 min threshold exceeded

2013.09.25 15:31:50 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:45:02 T1.t1 0/4 CSS 15 min threshold exceeded

2013.09.25 15:46:09 T1.t1 0/3 DM 15 min threshold exceeded

2013.09.25 15:46:12 T1.t1 0/4 ES 15 min threshold exceeded

2013.09.25 15:46:35 T1.t1 0/3 CSS 15 min threshold exceeded

2013.09.25 15:47:02 T1.t1 0/4 DM 15 min threshold exceeded

From my research, it seems that these could be caused by a clock source problem.  Our clock source is set to internal, since there is no T1 from a carrier involved.  Is this a correct assumption?

We are also experiencing lags in the audio between caller and recipient.  It can reach as much as a second or more, which results in the parties talking over each other.  I'm trying to determine if these two could be related.  I also have the carrier exploring this issue from their end.

[jayh]

You definitely have a clocking issue.  CSS is controlled slip seconds which is likely the cause of the errored and degraded alerts.

The IVR is probably also set to internal, or to another T1 elsewhere.

Each T1 should have exactly one source of clock.  If the legacy IVR equipment is connected to a carrier or another source of T1 clock, then you probably want to have the IVR clock from that carrier and the TA908e clock from the IVR.

If the TA908e is the only TDM connection to the IVR, then you can really go either way, either set the IVR to clock from the TA908e or have the TA908e clock from the IVR, whichever is easier.

You don't want them both internal nor do you want each clocking from the other.

As far as the latency, fix the clock slips and see if it goes away.  There will be a small amount of latency inherent in an RTP-to-TDM conversion but rarely to that extent.  What do the MOS scores look like in VQM on the TA908e?

[jonathanblack]

Thanks, jayh.

We do not have carrier T1s plugged in to the IVR equipment, which is why I was a bit confused about where to get clock source.  However, your explanation does make sense.  I am analyzing my entire setup to ensure that the clock sources are consistent throughout.

I'm obviously fairly new to this, so I wasn't aware of the RTP monitoring built into AOS.  Thanks for the heads up on that.  It's not turned on, so I'll do some testing with it turned on.  Do you know if there is a performance penalty to turning on RTP monitoring?

[jayh]

jonathanblack wrote:

We do not have carrier T1s plugged in to the IVR equipment, which is why I was a bit confused about where to get clock source.  However, your explanation does make sense.  I am analyzing my entire setup to ensure that the clock sources are consistent throughout.

You're on the right track.  The rule is that T-1 circuits are point-to-point with two ends.  Exactly one of those ends must source clock for the span.

I'm obviously fairly new to this, so I wasn't aware of the RTP monitoring built into AOS.  Thanks for the heads up on that.  It's not turned on, so I'll do some testing with it turned on.  Do you know if there is a performance penalty to turning on RTP monitoring?

Nothing significant in our experience and we push the TA900 series pretty hard.  Probably more impact rendering the flash on the page to view it than collecting the data but it's very well-behaved.

[jonathanblack]

I turned on RTP monitoring (and the firewall) over the weekend.  Unfortunately, on two occasions one of my carriers started getting 504 errors (server timeout) from our end.  This wasn't immediate, but after running for several hours, with a couple of additional hours between the incidents.  After the second time, I turned it back off, and since then no further 504 errors.  I'm not 100% sure the problem was this, but I'm now reluctant to turn it back on.

The two Adtran 908e units are 1st generation.  Is it possible that RTP monitoring on the 1st gen units could cause an overload and this was fixed in 2nd gen?

[jayh]

It could be a resource issue.  There was a software version that resulted in SIP resources not being released but that was fixed a while back.

The latest firmware for Gen. 1 is A4.11 .

Were you able to fix the timing slips and errors?

[jonathanblack]

We're on A4.11.00.E for both.

Still working on changing the clock sources throughout.  Since these are live systems, I have to do it during maintenance windows, which aren't always the same for each system.  I actually have a total of three 908e's and 4 IVR servers all interconnected at some level.

[Anonymous]

Jonathan,

I just thought I would check in with you and see if you still needed assistance.  I would make clearing the CSS the first priority.  This can affect voice quality and, in some cases, call control.  Once that is resolved we can work on the 504 responses.  We may need a long term debug capture in order to understand the reason for those failures.  Below are the common debug commands we use to determine the point of failure.

debug sip stack message

debug sip cldu

debug voice verbose

debug isdn L2-formatted

Our document Enabling Persistent Debug Logging can help you setup a debug capture that can run without closing down before the event occurs.

Thanks!

David

[jonathanblack]

David,

Thanks for your response.  I'm still working on the clock source change.  We have a particular Dialogic board on which I'm having trouble setting the clock source.  (It's a D/480JCT-2T1, and I can't figure out how make it get clock source from one T1 versus the other.)  However, that's not an Adtran issue.

Regards,

Jonathan

[jonathanblack]

Ok, I've changed the clock source to what I believe is correct (only one source per span).  This seems to have cleared the Degraded Minutes, but I'm still getting Controlled Slip Seconds and Errored Seconds (1411 of each in the last 24 hours on one PRI).

[jayh]

Controlled slip seconds are almost certainly a timing issue, although there's a very rare possibility that it's bad hardware or wiring.

On the one PRI still showing slips, is that the only T1 connected to the PBX?  What are the settings on that T1 on the TA900 and on the PBX? 

[jonathanblack]

We have four PBX/servers, each of which can have 1-3 PRI/T1s.  We have three 908e units.  The only external T1 (from a carrier) is a data T1 plugged into one of the 908e's.  (The other two get their data from a fiber connection which comes in as eth 0/1.)  We are using the data T1 as the clock source passing through to all the other units, since they are all interconnected. 

The settings are as follows:

Adtran1: Primary clock - t1 0/1

Connects to PBX1 (t1 0/4) and PBX3 (t1 0/3)

PBX1: clock source is line (only one T1)

PBX3: clock source is t1 0/3 from Adtran1.  Also connects to Adtran2 on t1 0/4.

Adtran2: Primary clock t1 0/4

Connects to PBX3 (t1 0/4) and PBX5 (t1 0/3)

PBX5: clock source is t1 0/3 from Adtran2.  Also connects to Adtran3 on t1 0/4.

Adtran3: Primary clock t1 0/4

Connects to PBX2 (t1 0/3) and PBX5 (t1 0/4)

PBX2: clock source is t1 0/3 from Adtran3

The only T1 that is currently experiencing slips is Adtran 2 t1 0/3.  Of course, that's also the one with the most traffic.

Here's a quick sketch that may make it easier to follow (arrow directions indicate the flow of the clock source):

[jayh · Accepted solution]

Double-check PBX5 and ensure that its primary clock is that coming from Adtran 2. If its T1s are swapped this would cause the problem you're seeing. Secondary clock on all devices should probably be internal.

Traffic isn't going to be a factor with regard to slips.

[jayh]

For what it's worth, this scenario is screaming for an Atlas 550 in the middle.

[jonathanblack]

Thanks, we'll look into that.  We've wondered if there wasn't a larger device that would handle more than two PRIs out.  Of course, our other option is to upgrade the "PBX/server" equipment to handle SIP natively and not do the conversion.  Unfortunately that investment would be fairly large in re-programming, equipment, licensing, etc.

[Anonymous]

jonathanblack

Take a look at the NV 644 (4xT1/E1 PRI SIP Gateway):  http://www.adtran.com/web/url/NV644

-Burgermeister

[jonathanblack]

Ok, thanks, I'll take a look at it. 

By the way, that link results in a 404.  I went digging and this link: https://www.adtran.com/web/page/portal/Adtran/product/1700144G1 gets to the page, at which point the URL above is displayed.  Kind of strange...

[Anonymous]

Hello Jonathanblack,

I went ahead and flagged the "Correct Answer" on this post to make it more visible and help other members of the community find solutions more easily. If you don't feel like the answer I marked was correct, feel free to come back to this post and unmark it and follow up with additional questions. 

Thanks,

Geoff

[jonathanblack]

Unfortunately, I am still experiencing these errors, so, no, the issue is not yet resolved. 

[Anonymous]

I agree with "Double-check PBX5 and ensure that its primary clock is that coming from Adtran 2."

also, clocks degrade the more hops they make.

I would suggest using Adtran 2 as the clock source, thus reducing the max number of clock hops from 5 to 3. Probably not THE issue, but it would be a better practice.

and yes the 644 is a good idea with it having more T1 PRIs, and it also supports two independent timing domains.