Hello Community,
This is my first time posting here and I'm looking for some ideas/feedback on how to address a problem we are having. The current configuration existed before I arrived but the problems have never corrected themselves.
We currently operate 32 locations all running over an MPLS network with a cloud style managed firewall. We have our own PBX system at the corporate office that operates our phone system across this network. Each of our 32 locations has their own T1 connection/s that carry both phone and data. We are using 3100 and 3400 series adtran routers at each location. Our ISP who manages the firewall has a second Adtran router that they manage that provides our QOS and traffic control for the most part.
We are having issues with phone quality on a regular basis. This quality is accompanied by a high level of dropped packets during those times of poor quality. We have been trying to work with out provider to have the QOS modified but it seems as though they are using QOS profiles and we can't seem to eliminate the loss of packets no matter how the modify their end.
With 32 locations and each being a smaller office we would need to be able to setup something up that works at each site. A hard limit on traffic by IP or port would be ideal. Our internet traffic speed is never business critical since we aren't a point of sale type business. I've looked into some of the options on limiting by port but didn't see that as an option on the 3100 series...am I correct?(Mostly 3120s) The 3400 series though seemed like it was possible?(3448 currently in use)
I'm open to suggestions and or configuration ideas that don't involve dumping the whole system...buying 32 new routers or trying to change our provider. Thanks for sharing your ideas.
There are a couple of threads on here about this. I'll post them shortly. If you are seeing packet loss it's usually either the phones not tagging traffic correctly or your carrier not having something setup right somewhere along the line. With that big of a network there are many places for things to break down. The equipment you have should be fine to accomplish the QOS you need. What type of PBX are you using and have you checked it's config to make sure it's marking packets correctly?
Good threads:
https://supportforums.adtran.com/message/11573#11573
https://supportforums.adtran.com/message/9760#9760
Good documents to read and verify your configs are correct:
https://supportforums.adtran.com/docs/DOC-1649
https://supportforums.adtran.com/docs/DOC-3587
There are a couple of threads on here about this. I'll post them shortly. If you are seeing packet loss it's usually either the phones not tagging traffic correctly or your carrier not having something setup right somewhere along the line. With that big of a network there are many places for things to break down. The equipment you have should be fine to accomplish the QOS you need. What type of PBX are you using and have you checked it's config to make sure it's marking packets correctly?
Good threads:
https://supportforums.adtran.com/message/11573#11573
https://supportforums.adtran.com/message/9760#9760
Good documents to read and verify your configs are correct:
https://supportforums.adtran.com/docs/DOC-1649
https://supportforums.adtran.com/docs/DOC-3587
Thanks for the quick reply. I'll read up on those links after I post.
We are using the Panasonic TDE200. We actually have 2 of them operating at 2 separate locations to divide the traffic. The packet loss is on and off which has raised my suspicion that it could be related to times of increase network traffic at that location but I haven't pinpointed that. Thanks
Hmm a quick read on that system and it does not look like it's got alot of option for QOS. The manual I found for it actually talks about using ToS which I've not had a carrier tell me they support. Most use either the DSCP value or UDP port ranges. If you've got a PBX vendor that can help maybe they can clarify the settings on your PBX and on the phones. The guide I saw also gave examples for KX-NT300 and 200 phones. If those are what you have I'd check the settings on them. I'd recommend doing a packet capture with wireshark or having your vendor do it and see if your VOIP traffic is getting marked at all and how it's tagged. You can then go back to your provider and see if/how they are matching traffic.
Here's the guide I was reading through. Wish I could help more but I've never worked on the Panasonics
My suggestion is that you schedule a call or meeting with your MPLS provider, your firewall provider, your phone system vendor, and anyone other than you who is responsible for network configuration.
Broad overview:
VoIP has two distinctly different flows that need to be treated differently.
QoS mechanisms can be based on a number of things and this is why the MPLS provider, phone system vendor, firewall vendor and you (or if not you the person responsible for router and switch configuration) need to be on the same page. Some things used for decision making:
Once you gain knowledge of the mechanism used by the phone system, the MPLS network, and the firewall(s), then it's a matter of ensuring that the QoS is classified correctly, marked appropriately for each device and/or network, and that the forwarding decisions honor the marking. Also necessary is reclassifying and/or stripping QoS tags applied to traffic that your management policy decides are not to be given priority (such as online video games).
Then it's a matter of configuration and testing. You shouldn't need to replace your Adtran gear, it's a matter of setting things up correctly and keeping them that way in the event of moves/adds/changes. I'm assuming that the corporate office has higher bandwidth than a T1 which is only good for about 20 simultaneous G.711 calls assuming no other traffic.
Thanks for looking. We have our traffic tagged and the QoS appears to be picking up the calls and placing them in the appropriate level. When we've investigated they've said that our traffic is overwhelming the QoS and that the voice calls are getting bumped. We could setup a limit on our router at each location that limits bandwidth by port or IP but would that accomplish what we were looking for if our provider has a QoS set up that differs? I've opened dialogue with them today to see what else they can do.
jterrence wrote:
Thanks for looking. We have our traffic tagged and the QoS appears to be picking up the calls and placing them in the appropriate level. When we've investigated they've said that our traffic is overwhelming the QoS and that the voice calls are getting bumped. We could setup a limit on our router at each location that limits bandwidth by port or IP but would that accomplish what we were looking for if our provider has a QoS set up that differs? I've opened dialogue with them today to see what else they can do.
Useful information. Investigate the following:
Or a combination of two or more of the above....
I couldn't see whether the QoS/DSCP values had been confirmed in this thread. Jayh's suggestions for typical SIP treatment across MPLS are fantastic. If your voice equipment isn't tagging with DSCP values like that (or if the MPLS carrier wants other values), then you can leverage your ADTRAN routers to add or modify DSCP tags on voice traffic using a feature of Policy Based Routing (PBR). Basically, you can match existing DSCP-tagged packets and/or match against an ACL in order to identify SIP and RTP packets. Then you can apply new DSCP values prior to forwarding across MPLS links.
Something like this would match SIP/RTP at ingress of Eth 0/1 based on UDP ports in an ACL, then add (or modify) the DSCP value before forwarding:
!
ip local policy route-map VOIP
!
!
interface eth 0/1
description LAN
ip address 192.168.100.1 255.255.255.0
ip policy route-map VOIP
no shutdown
!
!
route-map VOIP permit 10
match ip address tag-voip
set ip dscp 46
!
!
ip access-list extended tag-voip
remark tag SIP and RTP
permit udp any any eq 5060
permit udp any any range 10020 11000
!
Hopefully you don't need to mess with it, but it's good to have a tool for the job if you can't otherwise match up with the MPLS provider. If you have to settle on one tag or class for SIP and RTP, I think 46/EF (expedited forwarding) is usually best. Most SIP agents will re-transmit if no response is received within a short amount of time, but audio is unforgiving.
Also, is the degraded audio happening in one direction or another (toward endpoint or from endpoint), or both ways? The reason I'm asking is because a PBX or server may tag DSCP or ToS however you configure it, but endpoints may need to be handled differently. For example, automatically configuring DSCP values via LLDP-MED in your switches or manually assigning in each phone (hopefully not). You'll need to make sure that traffic in each direction can be identified by the carrier properly so it can be prioritized.
Best,
CJ