Disaster

unfortuntely, this story isn't as unusual as it should be. without slagging individuals, or even specific companies, the industry as a whole has moved to having very inexperienced people man the front lines of tech support. I do as you on my server, and maintain a second drive that is used purely for backups. But even with that, the risk of something like this happening again is not eliminated. also, it's possible to have a failure that can take both drives and shove them into the land where monkies don't shine. it's for this reason I now recommend people like yourself make use of this extremely affordable service. http://bqbackup.com/ I'm moving to using this service right now, and recommend it based on reports from other server admins who attest to it's usefulness. It's possible competing services are available, but I've not looked for any at this point.

glad you're back. and I'm glad it wasn't the evil red eye bizarreo monkey force that was out to get you.

This reminds me of the Great Mail Loss of Aught Aught. Back in 2000, I upgraded my mail client to Entourage. Cleverly, I backed up all my mail data to a separate hard drive. Unfortunately, I didn't think to disconnect and unplug that hard drive. I started up Entourage, and it promptly hunted down and deleted all previous e-mail I had on all connected drives. Kind of ingenious, in an evil sort of way.

I guess this also explains the duplicated RSS feeds.

Funny - just last Friday, the LAN guys at work told me my laptop had been hijacked and they had to wipe it. 'Tis the season.

Ouch. I feel your pain, man.

For those of us who felt the terrible loss of connection to JoCo this last weekend, I'm posting the five (5) stages of mourning. This is something I commonly do, as a programmer, for my users who refuse to backup their data.

Denial and Isolation: The first reaction to learning of terminal illness or death of cherished data is to deny the reality of the situation. It is a normal reaction to rationalize overwhelming emotions. It is a defense mechanism that buffers the immediate shock. We block out the words and hide from the facts. This is a temporary response that carries us through the first wave of pain.

Anger: As the masking effects of denial and isolation begin to wear, reality and its pain re-emerge. We are not ready. The intense emotion is deflected from our vulnerable core, redirected and expressed instead as anger. The anger may be aimed at inanimate objects, complete strangers, friends or family. Anger may be directed at our data or IT People. Rationally, we know the data is not to be blamed. Emotionally, however, we may resent it for causing us pain or for leaving us. We feel guilty for being angry, and this makes us more angry. The IT Tech who told us of the data loss might become a convenient target. IT professionals deal with data loss every day. That does not make them immune to the suffering of their patients or to those who grieve for them. Do not hesitate to ask your IT Tech to give you extra time or to explain just once more the details of your server's issues. Arrange a special appointment or ask that he telephone you at the end of his day. Ask for clear answers to your questions regarding technical diagnosis and treatment. Understand the options available to you. Take your time. Both you and your IT Tech person will find that honest and open communication now are an invaluable long-term investment.

Bargaining: The normal reaction to feelings of helplessness and vulnerability is often a need to regain control. If only we had backed up our data sooner. If we got a second opinion from another technician. Secretly, we may make a deal with God or our higher power in an attempt to postpone the inevitable. This is a weaker line of defense to protect us from the painful reality.

Depression: Two types of depression are associated with mourning. The first one is a reaction to practical implications relating to the loss. Sadness and regret predominate. We worry about the cost of data loss and recovery. We worry that, in our grief, we have spent less time with other programs that depend on us. This phase may be eased by simple clarification and reassurance. We may need a bit of helpful cooperation and a few kind words. The second type of depression is more subtle and, in a sense, perhaps more private. It is our quiet preparation to separate and to bid our data farewell. Sometimes all we really need is a hug.

Acceptance: Reaching this stage of mourning is a gift not afforded to everyone. Data loss may be sudden and unexpected or we may never see beyond our anger or denial. It is not necessarily a mark of bravery to resist the inevitable and to deny ourselves the opportunity to make our peace. This phase is marked by withdrawal and calm. This is not a period of happiness and must be distinguished from depression.

So does this mean you'll be doing a non-demo version of "Screwed" with an extra verse about backups?

Octopus appears to be missing from the songs page. I know a murder when I see one.

Dean, you missed a stage. Fucking outside.

Jack O'Neill:

Jonnie Coulton was / feeling nervous, he
Switched his server and backed up his site,
Bought a drive from his new hosting service, said
Now my data's all right

Glad to see the site running again, JC, and also glad to see you so calm about it.

General O'Neill is a JoCo fan? I guess I shouldn't be surprised.

I just realized all the old comments seem to be gone. D'oh.

Best wishes for a speedy data recovery.

I'm glad you're back! I had a weekend of strumming on some tunes from your on-site tabs collection lined up, and then it was down, and man did I ever cry. I had to look around for old, or inferior musics to try to fill the void, or prove to myself how bad I am at remembering chords by going it tabless.

Now I'm wondering if any of your horde has an entire local backup of the site through something like a recursive wget, or WinHTTrack. I feel guilty for not having done this myself, as it's sort of a secret nerd hobby of mine. I was so close to being a hero, if not mildly, yet perhaps forgivably creepy.

It's also possible one or more savants visited the site. If, say, Kim Peek had happened through the entirety of your server this past week, you could probably get him to sing the entire thing back to you, though you'll want to bring along a fast typist, maybe a court stenographer when you visit him.

Welcome back!

I tried to post an abridged summary of the comments to the PopSci video contest blog entry, but it's "awaiting moderation". Perhaps the software didn't like the looks of my hyperlink! But click my name on this post and you will see the playlist I've made of all the video contest entries I've found.

Welcome Back, Kotter^WJoCo!

Your root death story reminds me of the first -- and last -- time I completely killed a production server. I was a young kid on an internship at an ISP, learning the rudiments of BOFHism, when I managed to delete /etc/passwd from the mail server -- a big old cranky Sun monster running Solaris 2.51 -- which I also managed to reboot before realizing my error. Whoopsie! As my boss fed Solaris install CDs into the drive, he said to me "Don't worry, everybody gets to destroy one server. This is yours." The abject terror I felt that day 12 years ago is the best friend a sysadmin could have.

Welcome back. I missed this place over the weekend. :)

Man, I feel for ya. I used to write code, on a mainframe, and I've pulled some doozies. My first week on the job, my boss asked me to "load" a data cartridge and pointed to the console. I was just a bit-fiddler and had never handled the hardware, so like a dummy I stuck the tape in and happened to see a button on the front of the CPU unit that said "IPL/LOAD." Well, I thought, that loads the tape! That's the day I learned that IPL means you're rebooting the entire mainframe. With 100+ users attached, and a day's financial data being crunched. How I kept that job I still don't know.

I know the feeling (erasing the database).
Once I was cleaning up the database of the temporarily deleted records of a dBASE3 database and I mistakenly typed in (in DOS) "ZAP" instead of "PACK". Just a big "Oooops".

Anyway, I thought for a moment there you were abducted by musically inclined aliens. I'm glad your site is up and running. God Bless.

freaky "theplanet.com" did the same thing to us...

Wow! What's happening? One of my favorite site about airsoft at www.filairsoft.com also had a problem on their database (http://www.filairsoft.com/modules.php?name=Forums)
And had to divert to a proboards domain.
Man, this is weird. Too much of a coincidence.

In response to, "let me know if there’s a page or a link that’s not working," it looks like the various tabs on the song details pages aren't working. Neither is the submit user content function.

Kind of gives a whole new meaning to "Aftermath", doesn't it?

Sometimes I wish that there could be a "data loss event" in my life.. then I could rebuild the bits that worked and forget the bits that didn't. Imagine, no record of the screw-ups in your life.. just pick up the pieces you like and dust them off, build something new and move on.

Long ago, as the mainframe system programmer at a first-rank graduate school of business, I accidentally screwed up the file allocation area of a disk drive, causing the disappearance of the entire multi-million-dollar alumni fund database. There was a lot of swearing and screaming, and an entire alumni fund staff sitting around unable to work. Cold sweat time, my friends.

Thank goodness I was able to repair the problem within a few hours, much quicker than it would have taken to restore the data from backup tapes. And since they didn't know it was my fault in the first place, I was acclaimed a hero when I got everything back online. Whew!

Don't tell anyone I told you this.

As long as that competent tech wizard didn't start off with, "All your base are belong to us!" you'll be just fine.

I just went through a failed data recovery off a Powerbook G4 after my hard drive wouldn't boot.

I got screwed using a backup drive that was formatted for Windows and not Mac, so I thought I had lost everything. All I really cared about where the pictures of my two-year-old daughter...and JC's songs, of course.

But the Spirits in the Material World took mercy upon me and allowed my hard drive to boot as a slave, and all was recovered.

Sure, back up all your data. But for the love of Soterios Johnson, test a data recovery sometime BEFORE the hard drive gets all "soft rocked."

Greg

Glad you're back! I'm not making light of your situation, because I can only imagine what a pain this all must be, but...

In my long ago newbie code monkey days, I once purged all accounts in a production database with expiration dates greater than that date. I meant to purge the accounts with expiration dates less than that date.

All the blood left my body when I read how many rows were affected. And, given the cascade on delete properties, we lost lots.

I immediately yet casually asked my boss, "Hey, do we have a backup of XYZ?" To which he replied, "Backup?"

We managed to recover lots of stuff, and no one really knew what happened, so all was mostly well. To this day, "Backup?" and "Greater than" are our shorthand codes for something going FUBAR. :-)

Ooooh. That sucks, dude. Probably an RIAA ninja trying to silence you for your anti-DRM rants. Fight the power, man.

But on a practical matter, my recommendations for you, and everyone else:

1. Use mozy.com to backup your personal machine. $5/month for unlimited backups. PC version is shipping, Mac in late beta. Mozypro.com backs up servers, but you pay per GB. Both versions just run in the background uploading files soon after they change, and you can get back older versions for 30 days if you overwrite something. Highly recommended.

2. If you have major data loss on a Windows machine, GetDataBack for FAT/NTFS works great. I had a hard drive get badly scrambled and it got back nearly everything AFTER reformatting the drive. It was seriously like magic.

Good luck, JC. Everyone else, start backing up!

Oh, and "Vis Major", been there, done that. I now always make it a habit to begin a transaction before doing even the most trivial maintenance task. If I screw something up like you did (and I do, believe me), I just ROLLBACK and all is well.

Yeah. At my age, you can't afford to be too high strung. Besides that, it's good to know someone else has experienced an office crush.

Hey, JC! I'm sorry to hear about your computer troubles, that sounds awful. I guess, at the least, you can add your own verse to 'Screwed', right?

In other news, I recently purchased and listened to the Areas of My Expertise audiobook, and I have to say it was great. I wish I could see you and Hodgman perform live, you two sound great together. My favorite bit, besides that awesome theme song of yours, is the banter about omens, portents, and the prediction of Ragnarok.

Just thought I'd share that. I really hope everything works out for you!

Last week my daughter needed to edit a film for school on my laptop and Windows Movie Maker just about melted down my harddrive...

Then this week my Palm Desktop ate all my appointments and then overwrote my handheld. Thanks God I'm a redundant pencil-in-the-book appointment keeper, or I would have just had to sit in my office all week wondering who would drop in next and then act all normal: "Sure, have a seat while I get your chart... I was just finishing my lunch." I think I lost my dentist appointment though. Must call to recover.

Missed you.

All the info/video/etc sections for songs are missing. Or at least, all the ones I've checked.

HAHAHA. I committed a similar gaffe involving del *.* - it was in the C:\Program Files directory. When it started wiping out Common Files and other important folders, I immediately closed the command prompt window averting further disaster (Yes I was using DOS XP *blahgh*)

I missed all the excitement while I was away. Just aswell it didn't break the pages I wrote this time. A week without the song details page, how would you all cope?

Backups 4tw :)

Personally, I never let a week pass w/o backing up my laptop.

For me, I always have my portable harddisk with me all the time, just in case I got big files to keep or there's important files to backup. Worse if I am in Windows XP, so many patches goes to my C: drive.