I'm setting up a 2gbps connection for the aforementioned Thumper, and wading through all the various ways of doing things, many of which do not work. Here's what I've found.

IPMP is IP Multipathing. This automatically shifts an IP address between multiple interfaces to ensure that even if the primary interface goes down, the IP is still available on another interface. IPMP is for high availability, and doesn't inherently provide more bandwidth than a single connection.

Sun Trunking supports combining up to 4 or 16 interfaces (depending on the underlying hardware) into a single higher speed link. Sun apparently used to charge for Trunking, but bundled v1.3 into the first release of Solaris 10 for free, and then removed it from later releases of Solaris 10 (quite surprising behavior, actually).

dladm is apparently the new and improved replacement for both of these, but it only works with certain Ethernet drivers (fortunately the Thumper's e1000g driver is supported). dladm was not in the initial release of Solaris 10. Link aggregation via dladm is handled at the Ethernet layer, which means it requires switch support. Additionally, link aggregates function as a single link between two hosts (typically one is a server, and the other a switch). Aggregates are not suitable for switch failover, although that could be handled via IPMP on top of aggregates.

Apparently more large changes are planned under Project Clearview, but it's been underway at least since 2005, so I'm not sure how soon Clearview will be complete.


dladm itself is in a state of flux, too. I found a couple of blog posts which describe how to create aggregates, using a numeric key as identifier. But Sun's manual page says this is deprecated in favor of aggregate names. Unfortunately, the manual page's recommended syntax doesn't work -- it describes something newer than the current release of Solaris 10 (10/08, or Solaris 10 U6). That's ridiculous. Perhaps Sun's (commercial) Solaris documentation actually describes OpenSolaris...

More missing features from the documentation -- dladm show-phys & dladm show-ether, described in Sun's System Administration Guide: Network Interfaces and Network Virtualization, are not actually available on our fully patched Solaris 10 10/08 system. Feh!


Link aggregation (or teaming, bonding, or channeling) doesn't change the physical properties of the physical links. There are still multiple independent circuits at their rated speed (1 gbps Ethernet in our case), rather than a real higher-speed link (there is no such thing as a 1gbps circuit) -- the host & switch just balance traffic between the multiple available channels.

There are a few different algorithms for determining which packets go across which link to spread the traffic (hopefully evenly). Basically all three consist of taking a hash of some packet header data and matching that against one of the available channels. With L2 policy, the determination is based on MAC addresses. With L3 policy, it's determined by IP addresses. With L4 policy, more of the TCP/UDP headers are used (including ports). On Solaris, this is set with "dladm -P L2", "dladm -P L3", or "dladm -P L4".

We're using L4 because most of our traffic is from an instrument to one server, and then from that server to a backup server. The backup server mostly communicates with a single partner, so hashing on MAC or IP would leave all the traffic over a single link, and gain no benefit from the second 1gbps link. For a general purpose backup server (NetBackup, NetWorker, etc.), it shouldn't matter, unless one backup client is much faster/busier than the others and tends to dominate network bandwidth, in which case L4 might be useful.

dladm is noteworthy because it's one of very few Solaris commands that makes a persistent change directly -- things like configuring IP addresses and changing running processes generally need to be codified in configuration files or init scripts to persist past reboot, but dladm takes care of recording and re-applying configuration changes behind the scenes -- apparently a welcome attempt to make administration simpler and easier, also seen in ZFS.