Extra Pepperoni

To content | To menu | To search

Friday, February 8 2008

HP c-Class c7000 Chassis & Onboard Administrator Notes

The Onboard Administrators (we got a pair for redundancy) each ship with a unique password. When you connect them, it appears the active OA resets the standby password to match the active. This was a bit confusing, as OA #2 came up active, and the passwords were not as expected; SSL certificates are created and reloaded in terms of "Active" & "Standby", so I initially loaded new certs onto the wrong OAs.

ssh Implementation Flawed

The OAs support ssh access and ssh keys, but apparently only for the single Administrator account. This is documented incorrectly -- the docs say the last word on the key line is the username the key is for, but actually they're all linked to Administrator. HP Support doesn't know much about it. It's bad when security features don't work as documented -- in this case, it would be easy to follow instructions and upload a key for an unprivileged Operator or User account, unintentionally granting full Administrator access -- we had this for a while, until I figured out what was really going on.

The web interface doesn't allow copy & paste of keys -- they must be downloaded by the OA from a web server. Afterwards, though, the public keys (which had to be accessible on through a web server, remember) are not visible to other authorized users of the OAs -- only Administrator can see or modify keys. Feh.

Additionally, the web interface shows line breaks as '^', so the keys look corrupt. Despite this they work, and display correctly in the command-line interface.

OA doesn't automatically configure its accounts onto blade iLO. Instead, it creates an account for OA itself on each blade's iLO. This is a bit odd, as it means authorized users cannot connect directly to iLO -- instead they must connect through an OA, and have the OA login, before using iLO. We will presumably use the Compaq iLO configuration language to deploy our accounts to iLO, but this shouldn't be necessary.

Good News

On the bright side, the chassis is easier to mount than our (smaller) IBM BladeCenter chassis; it's also better labeled. The Onboard Administrator interface is better laid out, although it doesn't work in Safari (seems fine in Firefox/Mac). The command line is a bit less bizarre than IBM's.

HP makes it easy to dump the configuration to a text file, tweak it, and load it into another chassis, although we haven't tested yet; they call this "Configuration Scripts".

Saturday, January 5 2008

Cyrus IMAPd: only about as complex as a USENET news server

For several years, I've been saying Apple made a bad choice when they picked Cyrus IMAPd as the POP/IMAP server for Mac OS X Server. It's a huge and complicated system, encompassing IMAP, POP, SSL, Sieve filtering, LMTP delivery, USENET news, clustering/proxy (Murder), pluggable authentication (SASL), etc. I cannot think of a single company outside Cupertino where it would make sense to run an enterprise mail system on Mac OS X Server, but Apple continues to add these inexplicable high-end features to its mail server, most recently XSan-based email clustering in Leopard Server.

The statement that convinced me (shortly after I had migrated to Cyrus IMAPd on Mac OS X Server 10.4 "Tiger") that I would never choose to run Cyrus for my personal use, was the following -- which I came across again today:

Installation Overview

This system should be expected to have the same order-of-magnitude installation complexity as a netnews system. Maintenance should have similar complexity, except administrators will have to deal with creation and deletion of users and will have the option of managing quotas and access control lists.

USENET news is infamously demanding and bandwidth intensive. It would be wonderful if Apple had taken Cyrus IMAPd, repackaged it (without too many changes!), and put a powerful and simple interface on top. The did this quite successfully with Apache httpd (although Server Admin breaks down on complicated configurations and has obscure bugs). Lots of people use Mac OS X Server to run websites and think it's easy & simple. Considering the typical reactions of those same people to the httpd .conf files "under the hood", this is a noteworthy triumph. Similarly, Time Machine provides a reasonable approximation of scheduled snapshots on a high-end NAS for do-it-yourself file recovery, with a simple interface that insulates users from the nitty-gritty of copy-on-write and hard links.

Cyrus did not get as much attention, though. Basically, Apple makes it pretty easy to create email accounts, provides a Repair button for the overall Cyrus database, and provides a Reconstruct button for individual accounts. That's about it. Unfortunately, Apple doesn't really document maintenance beyond "press the button and it will fix your problem". I've had several serious database problems which Apple's Repair button did not help with. Those were bad times.

Similarly, I have had problems where users could not log in, but Workgroup Manager claimed their accounts were usable. I eventually discovered that resetting passwords with passwd works sometimes, and re-setting passwords in Workgroup Manager works consistently, but when I asked Apple about it, the eventual response was basically, "Yes, that's bad; you should restore your accounts from your recent Open Directory export." Not a good answer.

It doesn't help that Apple's SpamAssassin and ClamAV installations are broken, as these result in more spam and slower deliveries.

So why am I planning to migrate to Cyrus IMAPd on CentOS 5.1? Well, I'd really like to just copy my 5gb mail directory to the new system and have my clients not notice the difference. Eudora doesn't handle (IMAP) change well -- renaming a single IMAP directory can force it to download all messages again, and various other things can cause Eudora to lose date stamps on sent mail, or message state information (when it gets disassociated from the actual message on the IMAP server). If I can make Cyrus work, I'll be very happy, and if I can't I'll try Dovecot (Red Hat's default) or Courier (which I hear is also good).

Also, I know it can work, and I have a rough model to work from on my Tiger Server, but if I wasn't using Cyrus already I would stay away from it, as I wish Apple had done.

Thursday, November 1 2007

As a system admin, excitement is generally bad: HVAC Oops!

machine room pictures

So today they cut the wires to our main machine room's A/C. This occurred as part of the general campus work, which is why we were expecting to be out of our old machine room by now. Alas, the new machine room is not quite ready yet, so our primary systems were in a very warm room. It was a bit uncomfortable working there, although not too bad.

So around 3:30, my bosses (2) came over to ask me what could be shut down; in a perfect world, this would be just stringing a bunch of hostnames together, between "dsh -w" and "shutdown -h now" (for Linux) and "shutdown -y -g0 -i5" (for Solaris), from my desk. Instead I tromped over and started reading labels on servers (many of which were out of date -- now updated!), and deciding what we could do without, calling users to ask them which machines could be turned off for a while. We had my boss, boss^2, and boss^3, as well as a bunch of the Plant Ops guys and their boss.

After I'd shut down a dozen or so, they told us the A/C might be back within 15 minutes (hooray!). The first repair didn't hold (fuse immediately blew), but within 25 minutes we had (partial but insufficient) A/C, and I turned most things back on.

For a while we opened the door to the FDR drive, which cooled the room a bit. I got a few pictures of the drive and of blinkenlights.

Friday, July 27 2007

Verizon USB720: Useful but Disappointing

After a week at the beach with a Verizon USB720, I've found it useful (neither of our home laptops has a modem, and there's no Internet access in our beach house), but very frustrating. Apparently the Novatel hardware includes a GPS receiver, but Verizon doesn't make it accessible (neither does Sprint, who offers the same gadget). The connection always comes up at 144,000bps, which is about what my ISDN modem used to provide -- somewhat better than double the speed of a 56kbps analog modem, but about 10% of my home DSL speed; I don't think they're allowed to call this "broadband" in most markets.

Downloading my email (spam) takes tens of minutes; Safari keeps timing out and telling me I'm not on the Internet because it's getting no response on requests. I find myself alternating between: a) connecting, starting a Eudora mail check and loading a bunch of Safari tabs plus a Plucker run, and coming back (much) later; and b) connecting, trying to use the Mac, and wanting to howl in frustration because I can't read my mail or web pages; I have to wait a long long time before the Mac has loaded the content I want. I keep finding myself reading a novel, while supposedly using the computer. At least Pattern Recognition is very good.

I've wasted several hours of this vacation waiting -- for mail or pages to download or transfer, for connections that were actually down, for pages that refused to load (again). Pfeh!

I keep getting disconnected -- perhaps half the time this includes a scary message about disconnecting a device and losing data, although that's not such a big concern as reconnecting lets the application retry -- this tends to sort it out, except when the connection doesn't have enough bandwidth to satisfy the pending requests. I get a new IP on each reconnection, though, which gets AIM in a twist.

On the other hand, I have to respect any operating system that considers it an error condition if Internet access is completely unavailable.

PS-Happy SysAdmin Day, folks!

Sunday, June 24 2007

32-bit vs. 64-bit

The difference between 32-bit and 64-bit has come up again recently, this time around Mac OS X 10.5 "Leopard", which is going to be more 64-bit than Mac OS X 10.4 "Tiger", but not quite as 64-bit as originally claimed. The difference between 32-bit and 64-bit for processors and operating systems is not simple, but it's also not too complicated. So here's my attempt at an explanation.

A 32-bit processor has full support for working with 32-bit values. In the simplest terms, this might be a 32-bit unsigned integer -- a number between 0 and 4,294,967,296 (2^32 - 1). They can add, subtract, multiply, divide, and perform various other computations which form the building blocks of modern "computation", including things that don't seem to have any connection to mathematics, such as word processing and working with images (or audio, or video).

The predecessors of 16-bit processors worked with 16-bit quantities. Their successors (which include the majority of processors currently sold) are 64-bit capable. They are backwards compatible, meaning they can also perform the same 32-bit operations as their predecessors.

The first advantage of 64-bit processors is that certain operations are faster, because they can be handled directly instead of as multiple smaller operations. Obviously both 32-bit and 64-bit processors are adequate for adding 1 + 1, and there's no 64-bit advantage there, but what about adding 100,000,000,000 + 400,000,000,000? These are both larger than 4,294,967,296, so this addition would typically require more than one operation (clock cycles) for a 32-bit processor, but a 64-bit processor could do it in one operation.

This would be a substantial win for speed, but the mainstream 32-bit processors (Intel's Pentium & early Core models, the PowerPC G4 & G5, and AMD's Athlon & Opteron) all offer a range of 128-bit Single Instruction Multiple Data operations. These "SIMD" instructions cater to the most common "multimedia" operations, such as Photoshop filters, audio encoding/decoding, and video encoding/decoding. For things which are be handled by the SIMD units (MMX/SSE on Intel, AltiVec/Velocity Engine on PowerPC) 64-bit operations aren't any faster. For things which are not well suited to SIMD but are larger than 32 bits, there is still a benefit from 64-bit processing.


The other major advantage to 64-bit processors is memory addressing. A 32-bit processor can directly address 4 gigabytes of memory. 4gb is pretty good, but some of the Mac/Windows/Linux 32-bit memory "address space" is reserved for special purposes, such as I/O devices (disks, network cards, etc.). This is why Apple claimed early Intel Core MacBooks and MacBook Pros only supported up to 3gb of RAM. They could hold more, but some of it wasn't usable, and Apple didn't want to deal with the complaints from people who bought 4gb RAM and were only able to use 3.5gb. Current MacBooks, MacBook Pros, and Mac Pros are all 64-bit clean, and can fully take advantage of 4gb+ of RAM. There are workarounds for large memory addressing on 32-bit processors, but Mac OS X doesn't support them.

Large memory addressing is particularly important for high-end servers and heavy-duty media workstations, where 4gb+ is quite useful.

One reason there is misinformation about the benefits of 64-bitness is simply that it's complicated -- there are mitigating factors all around. Some benefits are available with a 64-bit processor and OS, but others are unlocked only with supporting chipset (the reason some Core 2 MacBooks and MacBook Pros still didn't fully support 4gb), and others require application upgrades as well (this is why Carbon APIs not being upgraded to 64-bit is a problem for some people).


Saturday, April 21 2007

A Plethora of Wireless Networks

26 WLANs

Here's a neat indicator of urban density. Julia & Amy came outside to play this afternoon, and then wandered off, but not before I brought my aluminum PBG4 out to join them. I briefly checked signal strength of our network with iStumbler, and was quite impressed to find 26 visible WLANs from our front door, significantly more than the 16 I picked up inside the apartment shortly before we moved in.

Saturday, February 24 2007

Props to the Network Guys

We have a bunch of 48-port terminal servers (they're Linux/ssh based, and quite good). Unfortunately, one of ours has a bad Ethernet port (intermittent connections -- no good for lights out management!)

Today (Friday), I spent from 4:15 to 5:30 labelling 48 Cat5 cables, replacing the old terminal server (a tight fit!), reconnecting the cables, and testing. It increased my respect for our Network group, as they do this type of thing all the time (although usually with less ports), and scheduling network downtime is much tougher than scheduling console downtime. Lots more people notice. Fortunately, the terminal servers are for our group, and used almost entirely by 4 particular people, so notification and scheduling was easy.

Still, it wasn't fun. At the end I had a label maker with dead batteries, a whole bunch of garbage from the labels, and grimy fingers, but we regained remote access for the weekend, which was my goal.

Next time I'll ask a hardware guy to do the cable swapping!

Sunday, December 10 2006

Going to the beach

So Friday I spent several hours in a machine room, dealing with a mulish array. Having failed to anticipate how much time this would take, I also failed to bring a jacket. We spent most of the time waiting for various people and things.

Back in the day, I used to read Wired to see what new words and phrases they'd come up with to describe geek life.

My own contribution, inspired by the hours spent in our machine room, waiting for a callback/reboot/answer/whatever (both yesterday and on other days):

"Going to the beach": Moving from the "cold aisle" (in front of the equipment, where the keyboards and displays are, and where the air conditioning system dumps chilled air) to the "hot aisle" (where the hot air is exhausted from the backs of the servers, before being sucked back into the A/C system for another cycle).

Example: "I'm going to the beach for a few minutes, while we wait for a callback from someone with a clue why the tool failed."

Tuesday, September 19 2006

Crowded Airspace!

I spent the day waiting for Verizon to install a new phone line, but they never showed up (again!). Supposedly tomorrow.

But I did find 16 networks from the new place, and had Internet connectivity most of the day, which made the wait somewhat less unpleasant.


Update: After 5-7 visits; including two simply skipped; one where the installer decided to leave instead of doing the install; one where the voice installer disconnected the DSL circuit to provide voice; another where the DSL installer (despite specific instructions to the contrary) disconnected the voice circuit to provide DSL; and a DSL installer who couldn't get the circuit up, but did manage to break the inside wiring; we finally got DSL working (thanks, Alex!). What a fiasco. I even tried to switch from Speakeasy service (Covad equipment over Verizon's wires) to Verizon-branded business DSL service, but couldn't wait over 2 weeks for the appointment. So we're back with Speakeasy until FIOS becomes available.

Sunday, June 4 2006

USENIX 2006, Boston

I just got back from USENIX in Boston, which was interesting. You know you're at a good technical conference when all the rooms have open wireless, and each table has a power strip...

But why wasn't there a session on advanced SSH???

This year's social event was (watching the penguins at) the Aquarium. The poor BSDers didn't get to visit with Beastie, though. I got a bunch of pics and a little video (snake

Thursday, May 4 2006

Network Monitoring with ping

For long-term low-overhead network monitoring, I came up with the following (note that the ping arguments are for Mac OS X ping -- Linux syntax is a bit different).

Put these into cron (with crontab -e):

0  0   *   *   *   ping -c 144 -Q -i60 www.speakeasy.net >> ~/public_html/dsl.log
*/10    *   *   *   *   date >> ~/public_html/dsl.log

And watch the status something like this: tail -f ~/public_html/dsl.log

page 2 of 2 -