Extra Pepperoni

To content | To menu | To search

Tuesday, January 22 2013

Dell DRAC & Macs

We have been vexed by Dell's lack of Mac compatibility for some time, but things have gotten better lately. Our problems are specifically with DRAC's virtual console feature.

The virtual console on our R910 (iDRAC 6) was incompatible with the Mac keyboard layout. We could bring up a virtual console but the characters on-screen did not match keys pressed. Our workarounds were a) to connect to a Linux system via VNC and run Firefox there, or b) to connect from a Linux or Windows VM running on the Mac. Imagine my surprise when I tried last week and the keyboard worked properly! We haven't updated the iDRAC firmware on that system lately, so presumably this was fixed in Java, but it's a major improvement.

Second, when I tried to bring up the the virtual console on an R820 I got an uninformative timeout. Some helpful Dell folks pointed me to v1.30.30 of the iDRAC 7 firmware, which adds Mac Safari compatibility.

To get v1.30.30, search for your service tag on http://support.dell.com/, select ESM (Embedded Systems Management), and choose the 'Application' .exe download. Only choose one item to download, or Dell requires you to use Akamai's download manager, which doesn't work for me. The .exe file appears to be a self-extracting .zip archive, so just unzip it and feed firmimg.d7 to the iDRAC 7 Firmware Update page.

Tuesday, March 30 2010

Dell DRAC: Technical and Support Limitations

Update 2010/04/23: Our keyboard deafness problems (both via USB and over DRAC) were apparently due to problems with the DRAC card itself. They remained after a downgrade to v1.4.5, but went away with replacement of the DRAC card.

I have spent a considerable amount of time trying to fix a Dell PowerEdge R900 server since it started reporting vague but serious memory errors last week. Since it's quite awkward to physically visit, I have been making extensive use of DRAC 5 (Dell Remote Access Card). Unfortunately, DRAC has a slew of poorly documented and poorly understood restrictions, which vary between hardware (DRAC 5) and software (v1.4.5/v1.5.1) versions.

Dell recommends Windows and supports certain flavors of Linux, but the farther you get away from the ActiveX control for IE under Windows, the more difficult things become. Since we use CentOS (unsupported) and don't have any Windows hosts on the private network with the DRAC interfaces, we have to jump through a few hoops to connect.

The compatibilities are complicated enough that I put together a table of what the different DRAC versions and components require (these are the issues I've encountered -- there are probably more in the release notes, and undocumented), with workarounds where available:

DRAC Compatibility Firefox Java keyboard stability security
DRAC v1.4.5
  • Incompatible with 64-bit Firefox.1
  • Serves ActiveX plugins to Firefox/Linux & Firefox/Mac.2
Input is garbled with Mac keyboard (even running Firefox on Linux via X11 from Mac).3
DRAC v1.5.1 32-bit Sun JRE 1.6.0-11 or earlier (-18 is current). Remote & USB keyboards are both unreliable (may be a local problem, rather than DRAC).4
  • Login errors.
  • Plugins crash more often than run successfully.
vmcli v1.5.1 Requires password on command line.5
Note .jnlp files do not launch javaws by default (not Dell's fault).6
  1. Workaround: rpm -e --allmatches firefox; yum install firefox.i386
  2. By default, DRAC attempts to serve up the ActiveX controls to Firefox/Linux. The workaround is to manually specify 'Java' rather than 'Native' for VKVM & VM.
  3. Direct login from a Mac browser shows the screen properly but typing produces the wrong characters. X11 forwarding from a Mac through a Linux system is the same. Workarounds include running Firefox in a Linux VM and tunneling X11 through ssh, or connecting to Firefox running on the Linux host via VNC.
  4. This may be an unrelated problem, which coincidentally appeared after upgrading to DRAC 5 v1.5.1. I hope to determine this soon. This was apparently the fault of a bad DRAC card.
  5. Partial workaround: precede vmcli with a space, to avoid recording the DRAC password in bash history. This does not help with ps sniffing, though.
  6. Workaround: specify a suitable javaws (such as /usr/java/default/javaws/javaws) from a compatible 32-bit Sun JRE. Additionally, saving .jnlp files for later use without redownloading in Firefox does not work. We do this on Solaris ILOM, as it avoids having to launch Firefox, login, and redownload a fresh .jnlp file for every connection.

Unfortunately, this all is poorly supported and understood by Dell. Phone techs can do 3 things: tell you to update (BIOS, DRAC, etc.), provide update URLs, and help work through the release notes for incompatibilities. Anything else is hit-and-miss. Specifically, mention of "CentOS Linux" tends to end support; X11 forwarding causes confusion and generally exceeds their knowledge and ability to assist. There are back-end folks they can escalate to, but I never managed to escalate successfully -- I had to give up after a half-hour on hold. But I did get (via a painfully long and circuitous route through our sales team) support from a helpful gentleman on the OpenManage team, who didn't have answers but was helpful in verifying problems.

To me, the moral is that Dell doesn't do software well (I haven't used their other software, so I don't know if this really generalizes, but I'm sticking to it unless I see a counterexample). It's such a small part of their business that it's not really surprising, but still a problem when you need a piece of Dell software to work, or to figure out what's wrong.


See also http://www.dell.com/downloads/global/solutions/DellRemoteAccessController5Security.Pdf.

Wednesday, June 24 2009

DRAC Notes

We have several systems with DRAC (Dell Remote Administration Card) v5. It's inferior to HP's for many reasons, among them the simple fact that dell.com hits aren't available in Google and www.dell.com is horrible for finding useful information. www.hp.com, in contrast, is navigable and their forum answers are well indexed by Google (useful for general Linux info, not just HP-specific stuff).

For flavor, check out my recent DRAC rant, and my older DRAC rant.

More bad things about DRAC 5

  1. Finding things on www.dell.com is amazingly difficult (HP does better on all these sub-issues).
    1. Older versions show up alongside (above) newer versions.
    2. Documentation references are stale (referenced documents don't exist).
    3. Refining searches, and searching for only relevant hits don't work (both HP and IBM can show downloads relevant to a certain Linux distro/version on a particular hardware platform).
    4. Dell's site makes it very difficult to find technical information. For example, compare Dell's R900 specs page to Sun's X4540 specs page.
  2. Google doesn't return results from www.dell.com. This makes finding authoritative info on Dell products more difficult.
  3. Compatibility
    1. Remote KVM doesn't work on Mac (serves ActiveX; uses wrong keymapping -- workarounds include VNC & Parallels).
    2. Remote media doesn't work on Mac (HP's does).
    3. Remote KVM & media don't work on Linux with 64-bit Firefox. They haven't updated these plugins in years, and probably never will -- newer servers use DRAC 6 instead.
  4. Dell sells a lot of cluster systems, but doesn't support cluster toplogies (Dell Gold Server Support assumes a Windows administrative workstation on the cluster segment, and cannot cope with X11 tunneling to access DRAC on nodes).
  5. Dell only supports the operating system purchased from Dell. The word 'CentOS' is verboten, even though Dell's diagnostic tools are CentOS-based! I don't know how many people pay for dozens or hundreds of Red Hat licenses for cluster nodes, but I'm certain it's less than the number of people using free/free distros like CentOS, Rocks, Scientific Linux, or even Fedora. This is also a problem for dual-boot laptops...
  6. Dell's installer (dup) doesn't work. It generally tells me it's already running, even after rebooting. Booting from (virtual) CD for diagnostics wastes my time.
  7. Some of Dell's firmware updaters still use Windows boot floppies. I recently decided that I didn't really need to update that firmware, rather than dig up floppies and find a Windows workstation to make a boot floppy.
  8. Dell doesn't identify parts. Sun is very good about putting meaningful 7-digit numbers on everything. This is very useful. In contrast, I had to grovel through Wikipedia to find out that a Dell card I inherited was in fact a SAS card.
  9. Dell doesn't indicate MAC addresses anywhere. Sun puts them on the packing slip. We had to rack and boot up an R900 in a temporary location to get its MAC addresses -- are required before we can put it on the network, after we re-box and move it.
  10. DRAC's serial console only works on COM2/ttyS1 and at 57600bps

DRAC Trivia & Tips

  1. DRAC cards have what looks like a PCI edge connector, but it's not used in our 1950s or R900 -- instead they connect purely through a couple ribbon cables.
  2. Not all DRAC 5 cards are the same. Specifically, the R900 card has a larger ribbon connector, which goes with a longer cable. In contrast, the 1950 cable is not long enough to reach the R900 connector. I of course brought the wrong card to our machine room, but in that case the longer cable fit the 1950. When I got back, I discovered the other card wouldn't reach in the R900, so had to go back to swap cards. I never did find any identification of which DRAC was for the 1950 vs. the R900.
  3. Supposedly, Alt-F Alt-E should reset the syste BIOS, although it didn't work for me. DRAC has a 'reset to defaults' option, but not the main (Phoenix) BIOS.

Tuesday, May 19 2009

OMFG! Dell requires procmail!

I've been fighting with Dell DRAC for months at this point. Some of their updaters insist that a copy of the updater is already running (even across reboots), and can only be installed by booting from a diagnostic CD. I've seen this with multiple updaters using DUP (Dell Update Packages).

The DRAC vKVM control doesn't work in 64-bit Firefox, and DRAC serves the ActiveX control to Firefox/Mac. When I point out this is a stupid bug, and the non-ActiveX plugin which works on Firefox/Linux should actually work on Macs, I'm told Macs are unsupported and this won't be fixed.

But today takes the cake. I wanted to install the new DRAC firmware on a PowerEdge 1950, so I downloaded Dell's DRAC5 Update Package for Red Hat Linux, v1.45, A00. I run RAC_FRMW_LX_R209365.BIN (well, of course that's v1.45 -- isn't it obvious??), but it requires an X11 DISPLAY. Why? It's not a graphical application. Dunno, just do it.

Okay, I log out, log back in with X11 forwarding, and run it. Dozens of prelink errors (about the same files, repeating) scroll off the screen, and then the xterm closes before I see what it did. Run it again, and it's failing because Dell expects a lockfile binary in my PATH. I don't have one. What does Google think? Workaround: yum install procmail, install, then rpm -e procmail to clean up. What possible excuse could Dell have for requiring (a piece of) procmail on my system to upgrade DRAC?

Then it failed again, but at least that time it spat out a useful error message:

Error while loading shared libraries: libstdc++.so.5:
cannot open shared object file: No such file or directory.
You must install the Linux compatibility libraries.
To install the compatibility libraries, use the following command:
"rpm -ih compat-libstdc++-33-3.2.3-47.3.i386.rpm"

Fortunately I was saved from banging my head against a wall any more -- I found Dell's "Hard-Drive" updater. That too is a Windows executable, but unzip f_drac5v145_A00.exe strips out the useful bit: firmimg.d5. The upload failed from Safari on my Mac, but I got it to work in Firefox/Linux (next time I'll try Firefox/Mac, which should work).

Parting Peeves

Why does Update sit under the "Remote Access" subhead, rather than the "System" heading? It's firmware for the whole DRAC system, not just the Remote Access part. I shouldn't have to hunt for the upgrade page. Adding insult to injury, the (remote) Console & Media tabs are under System, not Remote Access. Does anybody think that makes sense?

DRAC 5 Update page

In a final bit of brilliance, as the updater reached 100%, I got a warning that my session was about to be closed. Next time, guys, make sure you don't break your own firmware updates by timing the user out in the middle, okay?

Tuesday, January 27 2009

Dell's DRAC: Garbage

I've spent the morning fighting with Dell's DRAC 5 on a PowerEdge 1950 server (part of a cluster). People are asking me about it, so here's a brief summary of its failings:

  1. DRAC has a built-in graphical console (rac5vkvm). In Firefox, it's a Java applet, but the keyboard doesn't work if I start the process from a Mac -- I get the wrong letters when I type. Workaround: Start from a Linux VM. Since this is on a cluster, I launch a Parallels Linux VM, ssh into a cluster head, run mozilla on the head, navigate to the DRAC, log in, click a link, and get a serial console. My co-worker has the same issue, and a PMG5, so she cannot run Parallels to get a Linux VM.
  2. rac5vkvm fails to install. I got it to work as root on one of our head nodes, but it fails to install as pepper or even as root on other nodes, (presumably permissions, but the error points me to an apparently bogus log for more information). I'm told it also doesn't work with current versions of firefox.
  3. DRAC requires me to type racadm to prefix all the internal commands. What a nuisance! The exception is connect, which is the other main command. I have to find these both through Google, because help doesn't provide the list of (only 2!) top-level commands.
  4. racadm help config tells me the funky syntax to change settings, but doesn't list the available settings to configure!
  5. connect only works with com2. Why not with com1? What do the -b & -h options do? I don't know and apparently neither does Google.
$ connect help
Usage: connect [-bh] com2
  1. connect com2 doesn't actually work at 9600bps (the default speed for Linux and GRUB); I had to configure 57,600bps before I got a Linux tty.
  2. Dell's online documentation is lacking, and for some reason Google sends me to blogs & wikis instead. I eventually found a Dell page on DRAC 5 serial consoles, but it doesn't explain why whould I enable vs. disable "Redirection after Boot". What does "External Serial Connector": "Remote Access Device" mean? Perhaps it means admins will connect via RAC ("Remote Access Device"), and should be able to connect to a virtual serial port (for Linux console), but I'm just guessing.
  3. It's good that Dell provides a variety of options, but they conflict with each other, and Dell's explanations are inadequate.

There are apparently 3 different ways to get a Linux serial console on our systems:

  1. Connect to the physical serial port -- this works, but console switches cost money and consumes rack power & space.
  2. ssh into DRAC and connect com2.
  3. Use ipmish to make an IPMI connection to the BMC (a lower level management controller). I haven't tried this, as our head nodes lack prerequisites for OpenIPMI.