XDE may not receive multicast frames from physical underlay

[FelixMcFelix]

Reading up on mc_multicst when reviewing #905 led me to look into t4_mc_multicst and toy around receipt of multicast frames on T6 hardware¹. Packets with arbitrary multicast MACs will not be handed from the NIC to the OS (and thus will not appear on any ring) unless the NIC is programmed with a corresponding multicast MAC filter by illumos. I ended up testing this out by having one half of sn9/sn14 transmit exclusively to a multicast destination MAC, and then turning snoop -d cxgbe1 on/off on the receiving side to shunt the NIC into/out of promiscuous mode. Simnets will typically ignore these limitations.

We need to do a little more work to ensure that packets are received at XDE, by one of the following:

1. Have XDE call mac_multicast_add/mac_multicast_remove on the underlay devices according to each group's derived multicast addresses. Each multicast address on the client has a flow_entry_t created. However, it appears the cxgbe driver is unable to handle more than 336 entries on a given interface today.
2. Have sled-agent open a socket for each underlay multicast group address of interest, via UdpSocket::join_multicast_v6. This ultimately results in the same calls being made to plumb multicast filters, and we expect that the socket should receive no packets due to XDE's siphon/flow hook structure. This will run into the same limitation as 1.
3. Put the device into promiscuous mode. Since we control the upstream network we're not at risk of receiving additional unicast traffic (and paying a cost for doing so), but the main killer would be that this must be done by registering a callback on a client, for which illumos will duplicate all traffic.
4. Configure the T6 to receive all multicast traffic via the t4_set_rxmode method (the allmultien bitfield of the FW_VI_RXMODE_CMD T6 firmware command). This avoids the filter table size limit, but there's not really a sane interface for configuring this today, but then we lose hardware filtering by multicast address.

@zeeshanlakhani I think 1. is probably the most reasonable approach, since we can pull the MAC client handle from either the DlsStream or MacSiphon. That may not be doable if we change things to send and receive on the MAC flow directly, but we can probably expose a client function to pull out the mac_client_handle_t. What do you think?

Footnotes

1. My main motivation was in making sure that OPTE-bound packets would still transit from the ring to one OPTE's flows if it were to arrive on a multicast destination MAC in an IPD45 setting. This seems to be the case, but only if multicast packets arrive on the ring at all (i.e., via promisc mode). ↩