DEBRIS.COMgood for a laugh, or possibly an aneurysm

The subject of hearing protection came up at rehearsal last night. One of the guys has tinnitus, which can be pronounced either of the ways you’re wondering might be correct. The band’s ex-bassist had it too.

Meanwhile I’m sitting there whacking away at my drum kit, seemingly oblivious to the fact that 30 years from now my ears will be ringing like a cheap steel snare. Maybe by then the Dream Theater fan club will be offering group discounts on hearing aids.

I suspect my drums are louder 10 feet away than they are for me, in the middle of the kit — ironically, my seat might be the quietest one in the house. Even so, it’s dumb for me to risk my hearing, especially given that I own the world’s best earplugs. I’ll remedy that shortly.

You should know that one over-loud concert can nuke your hearing for life. An co-worker of mine from years back, an audiophile and music-lover, has tinnitus in one ear because he was seated in front of a speaker tower at a Journey show about 30 years ago. His right ear has been ringing ever since. He said it was a great show. For what that’s worth.

A company I’ve mentioned in this space before, Etymotic, makes a product designed to allow users to enjoy loud music without damaging their hearing: Musician’s Earplugs. The earplugs are custom-fit, and accept three strengths of filter: -9dB, -15dB, -25dB.

Dime-store foam earplugs attenuate at a greater level — by which I mean, foam plugs are quieter than these — but the problem is not strictly volume. Foam plugs cut high frequencies much more than low frequencies. As a result, music sounds dull, muffled and fake. The Etymotic plugs achieve a much flatter frequency response, so music sounds more natural, but quieter.

Because these are custom made for each user, the earplugs can’t be bought off the shelf (or on Ebay). You have to go through an audiologist, because you’ll need extra-deep impressions made of your ears. More on this in a minute.

Etymotic offers a list of earmold labs, but if you call them you’ll likely find that they refer you out to someone local. You might be better off with your own local Yellow Pages.

The plugs cost about $150/pair, plus whatever the audiologist charges to take the impression. The earplugs come with one set of filters. I recommend the 9dB model for spectators, or the 15dB model for people on stage. See this attenuation recommendation chart.

A friend of mine was initially turned away by the audiologist because of earwax buildup. Running an occasional swab through your ears is not an appropriate, nor sufficient, hygienic measure. Buy an earwash kit at the drugstore, or visit your doctor’s office to have the nurse scrub you clean. They can’t take accurate impressions if your ear canals are occluded.

I had my impressions made at the offices of H.E.A.R. in San Francisco, when they had a staff audiologist who donated an evening a week to preventing the next generation of great San Francisco musicians — and me, too — from going deaf.

His first step was to insert a tiny disk of rigid foam into my ear canal. Way in. Like, about four inches. He kept pushing, way past what I would have guessed was the point where the ear canal ends and the brain begins. When he put in the left side, I think it moved the disk on the right. It was scary, and loud, too. And if I recall correctly, it hurt, so he was pretty much batting zero for three. But that was the worst part.

The audiologist then mixed up some kind of rubber impression compound. He loaded it into a large-bore syringe and injected it into each ear, and then packed it in with his fingers. At least, I think he used his fingers.

And then I sat, for fifteen minutes, while the impression compound set up. I couldn’t hear anything except my own breathing, which was ragged and somewhat desperate. If you’ve never been fitted for earplugs, you won’t be able to relate, I realize. Suffice to say the entire process was uniquely unpleasant.

The modeling compound pulled out easily. I was ecstatic to see the little foam disk stuck on the end of the mold — I couldn’t imagine the audiologist going in after it with a really long needle or forceps or something. Or, rather, I could imagine it, which is why I was so glad to see it come out on its own.

The molds were sent off to a lab, and two weeks later I picked up my earplugs. They were tough to put in the first time, but immediately useful. “They make my drums sound better,” I wrote at the time.

Hey, maybe they’ll make my band sound better, too. (Just kiddin’, guys!)

Update — see also the High Fidelity Ear Filters from Hearos. There’s a nice review at the website of the Enemy. I can’t vouch for these plugs, and in my opinion a 20db cut is too much attenuation, but for the price you could certainly afford to experiment.

New Zealand’s Marlborough region features a “44-mile-long, inn-to-inn walking trail that seems custom-built for wine enthusiasts”. John Flinn scores by terming it the “Appellation Trail.” Ha!

Here’s the whole story: Walking New Zealand’s Wine Country

It reminds me of the Rotweinwanderweg.

It took most of two days, but I managed to get that failing disk drive replaced. By all rights I should be relieved.

I spent most of Saturday practicing RAID rebuilds on my test bench. I did nearly a complete dry run: booted with one functional drive, rewrote the partition map of the “new” drive, synchronized the RAID. It worked beautifully. Of course, it was just a demo.

The drive replacement on this server began at 10:00 AM today. We started with the usual problems: figuring out how to get the drives out of the server’s 1U rackmount chassis… determining that SCSI ID 0 must be the failing drive, but rebooting only with ID 1 to find out that ID 0 was the good one after all… magic, inexplicable SCSI ID failures, such that the ID configuration identical to what worked for eight months now refused to boot…

After about two hours, we began rebuilding the RAID. There were six partitions, the biggest of which took 30 minutes to synchronize. So we were standing around, buffeted by the sound of 1000 cooling fans and the 65° colo breeze, for a long time. We checked status frequently; the command ‘cat /proc/mdstat’ shows the ETC for the synchronization.

The process was on the 6th of 6 partitions, at about 99% complete. We were literally one minute away from being past the risky part: in one minute, we’d be able to put the lid back on, re-rack the server, reboot, test, and walk away. It would have been a by-the-book RAID repair.

But the ‘cat /proc/mdstat’ command hung. I’d become sensitive to the timing of things, because any pause usually meant trouble. Sure enough, after a few seconds the screen filled with RAID error messages. Not only did the last partition fail to synch, but the RAID software knocked three other partitions offline too, effectively erasing the past hour’s work. We shut down quickly and began the tedious process of investigation and repair.

Our best guess is that the RAID software got confused, maybe due to memory corruption, maybe due to the fact that the swap partition is also on RAID-1. (I didn’t set it up that way, and wouldn’t recommend it.) So, when we rebuilt the /var partition after rebuilding the swap partition, maybe the RAID software had a memory fault. Or maybe it was due to re-synchronizing 37gb worth of disk space. I’ll likely never know.

Upon reboot, /var was hosed. The data was corrupt. This was my worst nightmare — that repairing a redundant system, designed to prevent data loss, would in fact be the cause of data loss. We spent two hours recovering, and for most of those two hours, never really knew if we’d succeed. Fortunately, the other member partition of the RAID held a virgin copy of the missing data.

Ultimately, we erased and reformatted (mke2fs) the bad RAID partition, and copied the contents of /var onto it from a backup. This introduced some problems, but nothing that can’t be fixed. I assume.

Then we had to rebuild all the RAID partitions again. We went to lunch this time. I have, burned into visual memory, the image of approaching the system after lunch, wondering whether I’d find that the RAID rebuild had succeeded or failed, whether it had hosed a few more partitions on its way down. That was a turning point. The process had worked. (We’d opted not to rebuild the swap partition, just in case that had caused the initial failure.)

We buttoned everything up, racked the server, rebooted. The boot process hung on the network init. We realized we’d plugged the colo’s ethernet feed into the wrong port on the server, so I moved it. Ten seconds later, the server froze.

I’ve never seen a Linux machine freeze, but this was locked up tight. The screen froze at the login prompt. The keyboard was nonresponsive. We couldn’t access the machine over the wire. Despair, never far since the initial RAID rebuild failure, descended like a rusty axe toward my jugular.

I realize I’ve gotten too worked up about all this. I can imagine a thousand problems worse than a disk drive failure. But I don’t have to imagine the feeling of being dipped in stress, accompanied by a cold sweat and a stomach full of acid, because I had that already. I guess the bottom line is that if all else failed I could just go buy a new 1U server, copy data there from my backups, spend a day reconfiguring, and I’d be back online. Somehow that vision didn’t comfort me.

We used the reset button. The system rebooted successfully. Of course all the RAID partitions had been marked as out-of-synch, and the system began recovery on its own. So we left it. There didn’t seem to be any point standing around for yet another hour to watch. We’d been working for six hours on a simple (hah!) drive replacement; it was time to go home. And all the way home, I wondered if the system would be online when I got there.

Whenever I’m working on computer hardware, I notice the trend: things are making sense, or they aren’t. The trend is connected to confidence in the repair. For example, if a RAID rebuilds successfully, as it did on my test bench, I’d be very confident in the result. But if it pukes at the 99% point, I lose confidence. Having it ultimately work, whether by chance, or by careful avoidance of the things that might have triggered the failure, rebuilds some of that confidence… but if the machine later mysteriously freezes, I’m back to square zero.

And so it was that I fixed the problem that I’d been losing sleep over for a week, only to have even less confidence afterwards that the server was stable.

A long time ago I mentioned an article on web-writing tips at A List Apart. The article recommends writing during the bad times as well as the good.

Well, there you have it.

I was falling for it — the name-calling, the feigned indignation, the party line. Politics. What a pain.

In an effort to make this space a little more salty, a little more sweet, a bit less sour, and a lot less bitter, I have decided not to write about politics for a while. Maybe a long while.

But what about umami, I hear you asking. Yes, I’m going for more umami too. Someone pass me the Accent. Better a little MSG than more of the he-said, she-said crap. Jon Carroll’s plea for a return to the issues played a part in my decision. The war of personalities is an irritating distraction; it belittles the entire process. I’m done with it.

I’ll still write about environmental issues, even if they nearly always have a political edge. The environment is the thing I really care about. If not for that, I’d be utterly apolitical, and happier for it.

I got an email from Robert Redford. It’s actually a solicitation from the Kerry campaign, but the introduction is a strong statement that ties up my beliefs in a few clear sentences:

Today is Earth Day and I am afraid. For three decades, we have been fighting to protect the environment and have been proud of the great strides our country has taken. If George Bush is reelected we can count on the continuation of his agenda to undo all the advances we have struggled to achieve.

To put it simply, George Bush’s environmental policies endanger our health, loot our natural resources, and destroy the possibility of a secure energy future. George Bush may claim his environmental policies promote “healthy forests” and “clear skies” but those labels are both disingenuous and false. Most of his initiatives are nothing more than payoffs for wealthy campaign contributors.

Oh, yeah: happy Earth Day. Celebrate it now, while we’ve still got one.