Damn you, technology!

[prideslayer]

tl;dr : Always make sure your backups are up to date, and fully test your backup software before relying on it.

 

I don't pop into off-topic much, but I have had an irritating experience the past two days and just felt like ranting/sharing.

 

I'm a professional developer, I work from home. My desktop PC is my life. I have a laptop as well that I use on the rare occasions I must go to the office, or when my workstation is down, but I don't turn it on very often. My workstation has onboard SATA RAID, as most do. AMD 785G/710SB if you must know. I have a caddy in a 5.25" external bay that accepts four 2.5" laptop drives, and I run 4x 500GB seagate drives in it in a RAID-10. I also have a few external drives, the main one being another seagate 2TB "goflex" that I use for downloads and backups.

 

I also use Norton Ghost to do an automated full backup every week, and an incremental backup every morning, to the external drive. The *really* important stuff is also backed up to a 16G usb stick that I sync to daily. I've restored individual files and such from the ghost backups before, but had never done a full restore on this PC.

 

One of the drives in the array started a periodic ticking a few days ago. The RAID controller said everything was fine, nothing showing up in SMART, but again -- it's my life / livlihood. I generally don't take chances with failing hardware. So I decided the time had finally come to replace them all, and why not go a bit larger too. They're a bit old anyway (Momentus 7200.4's, about 5 years old) and usually when one fails, the others are not far behind.

 

I poked around the web, and before anyone rants at me, I did seriously consider going to SSDs. The price for the size is still prohibitively expensive though, and even if they're a million times more reliable than spinning disks, I would still run a RAID-10. Four 500G SSDs is in the ballpark of $1500 -- I almost bought them and then slapped myself for being an idiot.

 

Instead I bought four new WD's, 750G Black's. WD7500BPKX if you're curious. They arrived here on Monday.

 

Towards the end of the work day, I got a head start on the upgrade. Being a RAID-10 in external drive bays, this is something I can do while the machine is up and running, while I'm still working. I let the drives warm up for a few hours so there would be no possible issues with condensation or thermal expansion, and then did what I always do -- popped one of the drive caddies out, replaced the drive inside it, put it in, and told the array to rebuild.

 

First mistake, I did not verify that my backups were going off as scheduled.

 

After the array rebuilt and was again healthy, I popped the next drive out and repeated the process.

 

Second mistake, I did not replace the "correct" drive. Disks 1&2 are a raid-1 volume, as are disks 3&4. I replaced disk 1 first, and then did disk 2, when I should have done disk 3 (or 4) next.

 

About 10% in, the rebuild failed due to an escalating number of bad sectors. I thought it might be a fluke due to not being seated all the way, dust/stuff on the connectors, whatever. I pulled the drive out, cleaned it (and the slot), reseated it, and tried again. Same thing.

 

Now I'm a bit irritated, because these are brand new, and supposedly WD tests them all in a rigorous burn-in before shipping to consumers or resellers.

 

I can't leave my array in a degraded/critical state for long, it makes me too nervous, so replace this with one of the other new drives. I figure I'll let it run with two seagates and two WDs until the defective one can be warranty replaced.

 

This rebuild also fails, and I go from irritated to irritated and nervous.

 

Further investigation reveals that I did not get two defective drives, just one -- the first one. The one already in place in a degraded RAID-1, that was written to and read from just fine during the array rebuild. Now I'm even more irritated, and dreading the long process of restoring from backup, but know it will have to be done.

 

I fire up ghost and tell it to make an up to date backup. It can't. Too many errors trying to read from that disk. Great. Well, at least there's the morning backup. I won't have lost anything of note except time.

 

So I put the original drives back into the array, recreate it, and boot up off the ghost recovery CD. Time to start a restore from backup.

 

What's this? The last successful backup was December 28th?? WTF. Well, nothing to do about it but soldier on.

 

12 hours later, I find the restore did not work. I need to download the 32bit drivers for my RAID controller and load them up before doing the restoreBecause Ghost 15 is built on a 32bit version of windows. During the restore it worked just fine, but behind the scenes, it was writing to individual disks in the array since the driver wasn't present.

 

Things didn't really get better from here. Three restores later, and a boot off the Win7 CD to repair the bootmgr/MBR, and I'm back in business on my desktop. I've uninstalled ghost and am on a trial version of their new software which is, finally, available with a 64bit recovery CD so no driver downloads needed -- it will use the ones you have installed when you make a custom recovery disk.

 

On the upside, I have not lost very much. A few weeks worth of emails and downloads of random crap. It's nothing compared to what would've been lost with no backups at all, or if I didn't already keep important projects offsite -- like the sexout/fomm/nx sourcecode I recently moved here to LL.

 

I just finished testing the new norton restore disk and it works fine. I also used WDs tools to test all four drives, and so far, only the one is bad -- it won't even run their quicktest to completion, let alone run it and fail. Shortly I will be doing the raid upgrade dance again with working drives. I'm just going to do two of them, and wait for the RMA drive to come in to do the other two.

 

I've also setup the new backup software to email me its results every day, which I should've done with the old one. I have no idea why or how the backups started failling shortly after christmas, but now at least I'll know that if I don't get an email, something has gone wrong.

 

*shakes fist at the sky*

[Carida]

It was like reading a foreign language.. LOL I barely understood a thing. And its rare that I feel that unbright. Sorry to hear you had a hard time tho. Hope it gets better

 

-C

[Halstrom]

LOL, reminds me of the week I went through for this SCADA/HMI project.

Had to set up WinCC V6.0 for home use, (Current released version is V11 )

Spent 2 days proving it was impossible to get this XP based software to install on my Main Windows 8 PC

Another day trying to install it on my Backup PC run XP before I worked out it needed XP Professional not XP home

The only another 2 days setting up a 3rd PC with Windows XP Professional and 200 odd Windows updates to get the WinCC software to install & function but half the features still wouldn't work and it was getting confused with newly created tags being on my PC instead of the nominated Server, so not accessible by the screens.

So I went on site and got my own folder backup of the project as I usually did rather than using the WinCC archive function that the Sparky on site used as shown on the WinCC course.

Then magically all the rest of the functions worked again and I could start work on the project

[prideslayer]

Haha Hal.. it's real horror some days. I wonder occasionally why I don't just become an anti technology unabomber.

 

So after testing everything, I've swapped the first drive in the array again, and it is rebuilding... again. 30% done and then on to the next one.

[Halstrom]

And just to make it worse  the resolution options with the WinCC program are screwed & distorted in XP on my 22" monitor and my usual Magnification software and windows font settings won't play nice with it, thank gawd I think it's the last project I'll ever do with it, I told them another 12 months eyesight deterioration and I probably won't be able to use the software. I updated the original Project 5 years ago, but I'm possibly the only experienced WinCC programmer in NE Australia and I also know the machine inside out. I wish they would just spend some money and update to a later version or something more industry standard 

[RitualClarity]

This might be a silly question from a non-techie like myself, but why not enterprise level hard drives?

[prideslayer]

This might be a silly question from a non-techie like myself, but why not enterprise level hard drives?

There really aren't any yet in the 2.5" form factor (laptop size) space, and the few that are are super expensive. A WD XE 900G drive is $540. That's well over what I paid for all four of these.

 

It also somewhat defeats the purpose of RAID. Everyone seems to forget the "I" stands for "Inexpensive"

[Halstrom]

I went to 500gb SSD on my Windows 8 PC as a main drive and it's great, so much quicker and quiet, have to check glow from the blue ricer fan light to know if it's running, whether they last longer who knows, solid state tech isn't what it used to be, I've brought digital 2 clock radios that ran fast by around 20minutes a week 

[RitualClarity]

 

This might be a silly question from a non-techie like myself, but why not enterprise level hard drives?

There really aren't any yet in the 2.5" form factor (laptop size) space, and the few that are are super expensive. A WD XE 900G drive is $540. That's well over what I paid for all four of these.

 

It also somewhat defeats the purpose of RAID. Everyone seems to forget the "I" stands for "Inexpensive"

 

 

"I" should be "Inexpensive" but I have also see it used interchangeably with "Independent" as well.

 

Guess enterprise level 2.5's would be prohibitively expensive I wouldn't know. I can't even replace the drives that died on me in the last year.  More than 4 tb's worth of drives have bit the dust. Not normal failures. No clicking and no warning. Just silent corruption and final failure to access. Because of this I have resulted in "Selective" backup. That is accepting the reality that some stuff may just be gone if another drive fails.  Fortunately I don't rely on my computer as you do Pride.

 

I only mentioned enterprise level hard drives because they are suppose to be more resiliant and long lasting for purposes along the line mentioned in the OP.

[prideslayer]

I don't mind that a drive failed.. buying a brand new one that is defective sucks but is not a big deal.

 

I am astounded that during the whole raid rebuild process which copied 500G from an old drive to the new one, with verification, there wasn't a single error -- then 'bad sectors' as soon as I tried to read from it about an hour later.

 

Also was pretty upset that ghost didn't tell me the backups had been failing for the past 3 weeks.

[RitualClarity]

That really sucks that all that work only to realize that the product is crap in the end. The drive must have started to crap out right at the end. If that is the case whatever brief testing they did at the plant most likely would never have recognized a defective drive.

 

Oh and make sure you label the drives soon before you decide to change out the wrong drive again... .. Seriously.. like right now...   

 

A very common mistake, and sometimes a very costly mistake as well. All my drives and even the sata cables are labeled. Not pretty but works well when I start to dig into the machine. I even have notes on when the drive was bought. When it starts to get about three or so years old I move it from a primary to some other data drive. I did that with a drive that died recently. All I had to do is re-backup some data to another drive.

 

All of this must have put you behind on your professional and private projects... Hope you get it all sorted out soon and never have to go through this again..Best wishes.

 

[prideslayer]

I know which one is which now, and too late for labels anyway. 2nd drive is already in and the array is about 35% done rebuilding. When it's done, I'm firing off another backup, and then popping the "last" one in. Have to wait on the replacement drive to do the last one.

 

I'm very behind on private projects, have lost all the "Surprise!" work I've been doing for NX, Sexout, and Sewerslave. Not behind on anything professional, I'm extremely anal retentive about that because it's my ass if it gets lost. Everything I have locally is backed up manually, and almost all my work is actually done remotely anyway; e.g. when I'm coding and stuff I'm working on files opened from the office over a VPN.

 

I also have the important office stuff backing up nightly to a server I have here, in case of fire or terrorists or whatever.

[panthercom]

 Clocks that run too fast are probably a result of using cheap crystals for the timekeeper circuit; they come in different tolerances

and the ones rated for wristwatch type application cost more. The cost differential is fairly minimal. Of course, if they're made in China

the cockroach they use for a microprocessor probably gets impatient after a while

 

 I'm very selective about which technologies I embrace; I drive a 49 year old car and I loathe cell phones. I like powerful computers

very much, as long as I can play games on them

 

Class D audio amplifiers are nasty pieces of work; I've had the opportunity to look at the outputs with a spectrum analyser and see

just how dirty the output signal is. It's a well-kept secret that they rely on the HUMAN Brain and auditory system to filter out all the

high frequency garbage they emit; just about crapped my pants when I read the data sheet for a popular IC that may very well be

in your computer speakers and understood the implications of what was said. I asked one of the engineers where I work if it was true

and he just laughed and nodded. Just remember, newer is always better.

 

That's the end of my neo-Luddite rant...  and if you care about sound quality you should use vacuum tubes, dammit!!

 

I went to 500gb SSD on my Windows 8 PC as a main drive and it's great, so much quicker and quiet, have to check glow from the blue ricer fan light to know if it's running, whether they last longer who knows, solid state tech isn't what it used to be, I've brought digital 2 clock radios that ran fast by around 20minutes a week 

 

[Rayblue]

Using RAID has its shares of power and pitfalls. As much as it's desirable to have it for the sake of data redundancy, because hard drives are expensive (at least in my country), an inept RAID setup and/or poor usage practices could cause bad sectors to be written.

 

BTW, a couple of 500gb use hard drives I got for free had to be first wiped clean, checked and verified before I could use them, and that took 48 hours.

[Halstrom]

The cool thing we did find about ghosting is it copies licences as well, hence I now have a $5k duplicated 64k tag licence for WinCC on my development machine by some constructive use of backups and the dumb 90's tech Transfer Licence tool

[prideslayer]

Well, the nightmare continues.

 

I restarted the "upgrade" process yesterday and again, first drive went in and the array rebuilt fine. Moved onto the second one and was getting close to bedtime, so I let it go. This morning, I find the rebuild failed @ 83% done. Too many bad sectors, again. At least this time it was not the reading drive in a RAID1 pair that failed, so no need to restore from backup again -- yet.

 

WD gave me another RMA for the 2nd failed drive, and popped the 4th one in in it's place. It's rebuilding right now. After thinking about it for a few minutes though I decided I don't want these even if they work, I've lost faith in them. So I've got an authorized RMA from newegg for all four and will be sending them all back and buying something else once the two from WD arrive here.

 

I might actually bite the bullet on the SSDs this time instead. The cost is a lot higher, but I have little in the way of choice for good quality 2.5" spinning drives. As I mentioned, the enterprise ones cost *more* than SSDs.

 

Rayblue, I have good power here, and all my computers are on good UPSes with power conditioning, with good power supplies. I've had enough problems myself due to bad power in the past that I don't skimp there either.

[prideslayer]

Clocks that run too fast are probably a result of using cheap crystals for the timekeeper circuit; they come in different tolerances

A/C powered clocks typically do not use crystals or other timing circuits, they use the AC power cycle rate. In the US, the power companies regulate this so it's within some tolerance every day that's better than all but the best crystals and the circuitry is much cheaper.

 

See here and here..

 

It's a well-kept secret that they rely on the HUMAN Brain and auditory system to filter out all the high frequency garbage they emit;

This is true not just of amps, but of even the MP3 format (and others). "psychoacoustic analysis" is the term, and it means "the average human brain will filter out the screwed up bits and fill in the gaps."

[prideslayer]

Annnnnd another failure at about the 80% mark.

 

Which is no big deal at this point. Was just waiting for it to finish so I could eject it and start moving back onto the old drives anyway, so I guess the failure just gave me a head start.

 

Fuck WD, now and forever.

 

I did find some 1T enterprise Seagate drives, so those are on order now. I've always preferred them or IBM (how hitachi or whatever) anyway. The 36G 10k RPM SCSI drives I have are still cranking along almost 20 years later. Also Seagate. I think I have some 18GB 15k RPM Ultrastars that are still spinning just fine as well. Too bad capacity is so slow, they were *fast*.

[Rayblue]

Been using Seagates for years -- hardly had a problem with them as they last up to 4 years or even more depending how they're maintained and used. 

 

[Guest endgameaddiction]

Seagate has been really reliable for me as well. My current rig uses a Seagate 1TB HDD and I got this PC back from 2009. Haven't had any issues yet and I've put a lot of stress on it throughout these last 2 years. For some people it varies. It's kinds hard to say which HDD is reliable and what not. I won't ever touch another Hitachi again. even though I currently have one as a secondary drive. 

[prideslayer]

Yeah I don't know why I grabbed these WDs in the first place. Just looked like a good deal I guess.

[RitualClarity]

Not behind on anything professional, I'm extremely anal retentive about that because it's my ass if it gets lost. Everything I have locally is backed up manually, and almost all my work is actually done remotely anyway; e.g. when I'm coding and stuff I'm working on files opened from the office over a VPN.

 

I also have the important office stuff backing up nightly to a server I have here, in case of fire or terrorists or whatever.

Good to hear this didn't screw up your professional work.

 

Well, the nightmare continues.

 

WD gave me another RMA for the 2nd failed drive, and popped the 4th one in in it's place. It's rebuilding right now. After thinking about it for a few minutes though I decided I don't want these even if they work, I've lost faith in them. So I've got an authorized RMA from newegg for all four and will be sending them all back and buying something else once the two from WD arrive here.

That should never happen... Are the drives from the same batch /date/ firmware? If so WD might have totally screwed the pooch with this and the entire run of HD's might be screwed.

[prideslayer]

I'm not sure. I'm still having difficulties and I'm starting to suspect the motherboard.

 

After the array failed to rebuild today back to one of the OLD drives, I started testing all of them with a usb/sata adapter on my laptop. The one for-sure known bad WD failed, as it did before when tested on my workstation. All of my other drives are passing the tests -- my old seagates, and the other two WDs. The 4th WD is still in the array, and I did not pop any of those drives out to test them. Double and triple checked cables, replaced the drive cage power cable, and the rebuild is running again as we speak.

 

Instead of failing at 5-10% like the bad WD does, this failure crops up at 85% or more done though, so I won't know for a while if it's still having problems.

 

Maybe the southbridge is going... south.

 

If this rebuild fails I'll just delete the array, pop the original drives back in, recreate it, and restore from backup again. It's not like any data is at risk, since I haven't even bothered starting to reinstall stuff yet.

 

EDIT: Re batches/serials. That's one possible explanation. Another is the UPS dudes playing football with my package.

[RitualClarity]

The batches cold be an issue. If a batch is bad it is likely all of those will fail soon.

 

 

Another is the UPS dudes playing football with my package.

Not going there...

 

Another possible issue however if the packages are OEM or retail from the manufacture plant. the one with the plastic holder of the drive etc , and there is no damage to the box it should be safe. The actuator arm should have been in home location and the vibration even drops and tosses should be no problem. They are designed for those drives. I could drop one off a small staircase and it would still work.. Don't ask me how I know this... As long as that box doesn't have damage to it you should be safe. Added to the fact that it is a notebook drive which adds more safety to vibration and shakes when unpowered.

 

However if it was in any other box it is likely the cause of the problems. I don't know of any drive that survived retail delivery in a non - standard box. They usually fail in about a month or less. many retail stores ship them in totes or loose boxes. These allow vibrations that aren't allowed with the proper packing. These vibrations and shakes cause stress on a hard drive and that accelerates the failure.

 

For everyone

If the drive isn't in a proper OEM / Retail box.. don't buy it People. Just don't.

 

 

 

 

[prideslayer]

They're from newegg, of course they don't come in OEM packaging.

 

Newegg feels they must open up every single one, put it in their special bubblewrap/aircushion packaging, and stick it in a little newegg box. Then they pack all those into a BIG newegg box and send it out. They don't want you running out of cardboard or aircushion things.

 

Really though only one is actually known to be bad right now. Their serial #s are wildly different. Two start with WXM, one with WXA (not sure on the 4th), and it's one of the WXM's that's bad for certain.