The Pretties

Inspired by When data center cabling becomes art from Andrew T Laurence & Chuck Goolsbee's pics of Digital Forest, I took some photos of Rockefeller's new data center. We've been planning out various scenarios for 5 years at this point, but we finally moved most of our systems in this month. Note that the network guys (mostly Eric) took care to run cables connecting to ports on the left half of each device in from the left, and come in from the right for ports on the right. This makes more work for them in preparation, since one cannot simply plug a cable into a free port, but makes things look prettier, and also reduces cable snarling. 3 KVMs & baby + LCD

More Connectivity, Please

Since we first started discussing data center plans, I've been saying we need more connectivity. The new DC has 48 patches per 42U rack, and some of the new racks are indeed running out of ports before they run out of vertical space. In our racks 2U is used for patch panels and 2 cables control APC managed power strips, so we have 40U and 46 patch ports for servers. Our Linux servers have Ethernet, serial console, & KVM; Suns have Ethernet & console; Windows have Ethernet & KVM. In the worst case, 40 1U Linux servers need 120 connections, but we only have 46 available. If the rack is full of 2U Suns & Windows servers, we're okay with 6 'extra', available for dual-connected servers or whatever. As we get more dense, we begin to run out of ports. Cat6 flowing down

Blades

Blades are no better -- their chassis tend to blow out the power budget because they're even more dense than 1Us (although they do get more servers per rack), and with all the redundancy they still require a lot of cabling. For a reasonable IBM BladeCenter, we need 4 x 2 for GE switches (FC cables don't go in these patch panels). Then 2 x 2 for (Ethernet & KVM) for management modules per chassis = 12 ports for 7U. For our new HP c7000 chassis with basic networking, we have 16 GE ports, 2 GE console ports, 2 OA Ethernet ports, and 2 2 OA serial ports (again, ignoring the fiber-optic GE ports): 22 ports in 10U. I'm sure somewhere HP has demo chassis, filled them with fully-connected GE switch modules: (9 x 8 + 4 = 74 patches) & (4 x 8 = 32 fiber-optic ports) = 106 cables total (not counting power connections -- 6 in our case). In 10U -- 1/4 of a rack -- insane! c7000: 30 ports

Update 2008/2/5: Eric pointed out I was wrong about the ports -- the Cisco switches have 8 uplink ports, 4 of which are either fiber-optic or copper (you can see they're 17-20 in the photo); the other 4 copper ports seem intended for cross-linking to the other switch. So the max copper patch count remains, but the the fiber connections would be instead, rather than in addition, and we may fully connect our 2 switches with only 8 GE uplinks rather than 16 going out of the chassis.