Port Mirroring (monitor sessions) mutliple ports to an uplink switch (w/ Link Aggregation (LACP) enabled)

[blue_waves]

Hi Support,

I would like to implement a similar solution as depicted in the below diagram (Adtran's sample network).   This would be my final configuration setup.  However, before I reach this advanced phase, there are some issues we are experiencing on a very simple setup that almost closely matches the diagram below.

Objective:-

1. Port Mirroring all switch ports.
   1. We are implementing (installing) an IDS / IPS appliance in the network that needs to capture all of the traffic from all of the switch ports.  So we would like to Port Mirror (RSPAN, SPAN or Monitor Session) on all of the switch ports and direct this to a port on the NetVanta 1550-24 switch where the IDS/IPS appliance would be connected.
2. LACP / Link Aggregation.
   1. Once port mirroring and capturing works, we would like to then add the LACP configuration to the switches.
3. In essence, the final configuration should support capturing all the traffic from all the switch ports (using Adtran's Port Mirroring feature Monitor Session) to the UpStream / Core switch, where an appliance will be placed that is capturing and analyzing of this activity.  Once this main feature is up and running properly, then add Link Aggregation (LACP) in the mix. With the final network almost looking and working as depicted in the above diagram.

Current Configuration:-

Our Adtran NetVanta switches are arranged physically and logically as shown above.  However, there is NO LACP / Link Aggregation currently running on any of the switches.  Additionally, our upstream switch is a NetVanta 1550-24 and the closet switches are two (2) NetVanta 1534 switches.  There is NO NetVanta 1534P POE switch in our setup.  Just three (3) Adtran NetVanta 1500 series switches altogether.  Only a single VLAN is in use i.e., the default VLAN.

So the tasks / objectives are basically two (2) things:-

1. Port mirror all switch ports to the NetVanta 1550-24 port where an IDS / IPS will be capturing all the traffic.  We have a similar setup at another location, but using Cisco's RSPAN SPAN technology.
2. Once #1 is successful, then try to get LACP working.

Initial Testing in basic setup configuration for Port Mirroring / Monitor Session:-

1. Our first testing of using the Port Mirroring feature to an Uplink switch / UpStream switch (core switch) was not successful.  It worked at the start (first couple of minutes, about 10min), then all traffic just stopped.  That is, we lost connectivity to the servers and none of the devices were reachable via ping or the client apps didn't see the server and network drives were no longer accessible.
2. Here is what we did to get started with the setup/testing:-
   1. Activated Port Mirroring on only one of the NetVanta 1534 switches (source) and on the NetVanta 1550 switch (destination).
   2. So we used the CLI command 'monitor session', to port mirror ports 1-23 (Source ports) and made port 24 the destination port.
   3. On the NetVanta 1550 we port mirrored switch port 24 (uplink port servicing / connected to the NetVanta 1534 switch) as the source.  And port mirror switch port #3 as the destination port.
   4. This testing was done only with a single-uplink.  No LACP.
   5. Only Single VLAN, VLAN 1
   6. Using the packet sniffing tool, Wireshark, we saw and confirmed that all traffic on the network was reaching the Upstream / Core switch, i.e. the NetVanta 1550-24 switch.
   7. However, all traffic / network connectivity was lost by the servers and clients / workstations.  Nothing could be reached.
   8. Once the Adtran switches were rebooted on which we made the configuration changes, all network traffic services was restored and things went back to normal.  Pings got thru etc.
      1. We didn't save the changes made.  So we could easliy revert if it didn't work.

Our configuration looked like this:-

• On the NetVanta 1534 switch

          !

           monitor session 1 source interface gigabit-switchport 0/1 both

           monitor session 1 source interface gigabit-switchport 0/2 both

          monitor session 1 source interface gigabit-switchport 0/3 both

          monitor session 1 source interface gigabit-switchport 0/4 both

          monitor session 1 source interface gigabit-switchport 0/5 both

          monitor session 1 source interface gigabit-switchport 0/6 both

           monitor session 1 source interface gigabit-switchport 0/7 both

           monitor session 1 source interface gigabit-switchport 0/8 both

           monitor session 1 source interface gigabit-switchport 0/9 both

           monitor session 1 source interface gigabit-switchport 0/10 both

           monitor session 1 source interface gigabit-switchport 0/11 both

          ..... continues on until gigabit-switchport 0/23

           monitor session 1 destination interface gigabit-switchport 0/24

• On the NetVanta 1550-24 switch

          monitor session 1 source interface gigabit-switchport 0/24 both

          monitor session 1 destination interface gigabit-switchport 0/3

Problem(s) / Results:-

Traffic just stops after about 10mins or so.

Question/s:-

1. Why would the Adtran switches just stop passing traffic?
2. Is this configuration acceptable or good to do port mirroring and send all traffic to an uplink port?
3. In doing some research, someone on another forum says that the Adtran NetVanta can't handle the traffic to an uplink and eventually causes an unintentionally network loop. Seems like a reasonable explanation.  A workaround was recommended.  But I wanted to check here first as the Adtran support is great and the forum members very helpful and insightful.
4. Once you can confirm that this basic minimum configuration can work, how can we then add LACP / Link Aggregation to the Adtran switches and still make the Port Mirroring work to an uplink port (and then to the UpStream (/Core) switch?

Regards,

[Anonymous]

Hi,

To help the forum address this question -

1.  Monitoring all ports with a port mirror to an uplink switch has many potential issues that would have to designed around.  Some examples would be ;

• Multicast traffic - could consume the network
• Broadcast traffic - same issue
• Spanning-tree network design - would need to be designed so that the root would not block traffic it saw as in a loop.
• The monitored traffic would have to be less than the destination port speed, or traffic will not get to the monitor
• Switch traffic will only stop if the traffic never gets set up by the CPU or if Spanning-tree blocks a port.  Otherwise, to determine a hardware failure,  packet captures of both the input and output traffic would be required to show a hardware issue.

2.  All traffic can be sent to one port, but the previously listed issues would have to be managed or designed around.  I would suggest that aggregating the monitor ports separate from the data uplink ports would make this design less difficult.

3.  The hardware can handle speeds up to the uplink port speeds without issue, but the spanning-tree design is expecting BPDU's to only be sent and received to the next device on a port and does not know about a port mirror situation that could duplicate this packet on another port.  

4.  I would not recommend adding Link Aggregation to the uplink ports if you will be monitoring all traffic to one port, since the bandwidths would cause congestion and drop traffic on the monitor port.  It would still be possible if the aggregation is not used for additional bandwidth and only as a failover connection.

5.  Take the following example, and determine how the switch should work in the configuration provided with all traffic monitored on a 1544 port in the diagram.

• Lets say we have a meeting using skype with multicast traffic.  The video being shown to the group is a normal broadcast quality of 6 Mbps down and 3 Mbps up.
• How would spanning-tree be set up to determine a loop but still not use port mirrored BPDU's that are intended by the port mirror design?
• 12 ports on one switch are involved in the conference.  12 x (6+3)Mbps = 108Mbps
• One Destination port would not be able to monitor just 12 ports on the switch.
• But if this was audio only or the number of ports used at once was controlled this can still work with network engineering.

Hope this provides the insight required to engineer a supportable solution.

Regards,

Product support  

[Anonymous]

The NV1638 /1534/1531 that will provide one direction of traffic with a VLAN tag and the other untagged.  On these devices, the hardware chipset is the cause and it cannot be changed, so this will not be fixed.

Port mirror with 1638/1534/1531 - RX only work around

1.  Put all the devices you are interested to monitor on a new VLAN, will call VLAN M.

2.  Create a loop of two ports A and B

a.   Set switch port A
        -   switch port access VLAN M
        -   spanning-tree bpdufilter enable

b.   Set switch port B
        -   switch port access VLAN old    <----- this is the VLAN that was originally uplinked to the next switch/router
        -   spanning-tree bpdufilter enable

3.  Set up port mirror

  !
  monitor session 1 destination interface gigabit-switchport 0/D     D is the destination port of the monitor session
  monitor session 1 source interface gigabit-switchport 0/A rx
  monitor session 1 source interface gigabit-switchport 0/B rx

Note - The rx only setting is to allow this configuration to keep functioning even if the firmware was updated and it restored the TX .

Hopes this helps those that run into this issue.

[blue_waves]

Hi JRoad / Adtran;

Thanks very much for the follow-up and detailed response.  Also for providing a workaround until Engineering can resolve.   Can you let me know the status of engineering and if they are close to releasing something.

Regards.

[Anonymous]

(Update) NV1550 is working as expected, and the Wireshark PC was determined to be causing the issue. 

The other switches are using ASICs that cannot be changed to correct the issue, so will not be fixed.

[blue_waves]

Hi JRoad / Support;

Thank you very much for following up and the update.  A few more questions for further clarification:-

1. Are you saying that the NV1550 can support port mirroring all ports to a single port similar to RSPAN used in Cisco switches (without any weird stuff going on)?  If it were just one switch being used (in standalone mode)?  That is, a single switch with nothing else connected, thus eliminating the complexities of having multiple switches in the topology.
2. The NV1550 will not have the Rx only issue as you'll discovered with the latest firmware and we can port mirror to a single port and capture all traffic?
3. We should be able to accomplish our objectives if we decide to upgrade/replace all of the current Adtran NetVanta 1534 switches with the newer Adtran NetVanta 1550 switches.
4. So, for example, if we replace all (both) of the NetVanta 1534-24 switches in our current configuration with all NetVanta 1550 switches, which will then bring our entire LAN topology to using either 3 x NetVanta 1550-24  (or, 1 x NetVanta 1550-48 & 1 x NetVanta 1550-24), that we should be able to achieve our objectives of port mirroring (MONITOR SESSION) all traffic to a single designated port on each switch and then carry, where necessary that monitored traffic to an uplink port so that traffic can be monitored by security appliance?

Regards.

[blue_waves]

Hi JRoad;

Thanks for this tip and how to.   To confirm the setup.  Would it be like this image (attached)?   Also, when the NetVanta 1550 issue is looked into and corrected, do I just remove the parameter "rx" in the CLI "

monitor session 1 source interface gigabit-switchport 0/B rx"?

Thanks.

Regards.

[Anonymous]

1.  NV1550 can support port mirroring all ports ( except the destination port ).  Not familiar with RSPAN enough to say.

2.  The RX issue reported earlier turned out to be a PC issue and not the switch. So the answer is Yes.

3.  Objectives not specified.

4.  The switch can port mirror all 48-1 or 24-1 port to the destination port.  This is not supported remotely as in the SDX series switches.

[Anonymous]

To get all traffic on a port -

monitor session <number> source interface <interface> both

See - Configuring Port Mirroring in AOS

I do not believe this will work since the MAC tables of the switch with the Server will get scrambled, and we do not have a way to not learn MAC addresses and still pass them to the server.

But if you had two NICs on the server connected to the Destinations Ports, that would work.

Doing remote mirroring is not supported on any AOS switch currently.  See Sales to get information about the ADTRAN SDX series switch that supports remote mirroring.