Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Thecus N5550 RAID 5 - 2nd Drive Failure during rebuild!
#1
Hi all,

Last week I had a drive failure on my Thecus NAS N5550. I have 5 2TB Seagate drives (all models in my NAS are compatible according to the HDD compatibility list for the N5550) After the failure happened, I immediately turned off the NAS and ordered a replacement 2TB HDD.

The 2TB HDD arrived today and I put the drive in the tray and booted the NAS. For whatever reason, the RAID 5 did not start to rebuild automatically (my experience with other Thecus products was the RAID usually automatically rebuilds). I looked up the issue online and saw that if I choose the new disk as a spare disk and hit apply, the array will start to rebuild. About 30 minutes into the process, I suddenly hear the NAS beeping like it had when the disk failed last week. I looked at the NAS to find that now the 3rd disk in the array was blinking red, indicating a failure.

This of course happened during the RAID rebuild, and I am well aware of the fact that RAID 5 can only withstand a single drive failure. I did a bad block scan on the hard drive in question, and 4 bad blocks were found. Is there a repair tool or some way to fix these bad blocks so I can continue rebuilding the RAID? After the bad block scan was complete, I shut the unit down and started it back up, only to find that now the web interface and the front panel is saying that there is no RAID array found. Lots of fun.

I am fully well aware that I may have lost my data, but I was wondering what (if any) possible things can I try to do to fix the bad blocks on the disk that failed during the raid rebuild? Is there anything I can do? Are there any applications (such as spinrite) that I could use to try to recover the bad blocks?

I should mention I do have the SSH module installed on the NAS, and I am very comfortable with using the command line in Linux so if there is anything I can do via SSH I am fine with it. I was considering just going ahead and re-creating the RAID array, but I have a LOT of data on this NAS and am willing to do anything I can in order to recover whatever I can before I resign myself to losing everything.

Let me know if you need any additional technical details..

Thanks
Richard
Reply
#2
Hi all,

Last week I had a drive failure on my Thecus NAS N5550. I have 5 2TB Seagate drives (all models in my NAS are compatible according to the HDD compatibility list for the N5550) After the failure happened, I immediately turned off the NAS and ordered a replacement 2TB HDD.

The 2TB HDD arrived today and I put the drive in the tray and booted the NAS. For whatever reason, the RAID 5 did not start to rebuild automatically (my experience with other Thecus products was the RAID usually automatically rebuilds). I looked up the issue online and saw that if I choose the new disk as a spare disk and hit apply, the array will start to rebuild. About 30 minutes into the process, I suddenly hear the NAS beeping like it had when the disk failed last week. I looked at the NAS to find that now the 3rd disk in the array was blinking red, indicating a failure.

This of course happened during the RAID rebuild, and I am well aware of the fact that RAID 5 can only withstand a single drive failure. I did a bad block scan on the hard drive in question, and 4 bad blocks were found. Is there a repair tool or some way to fix these bad blocks so I can continue rebuilding the RAID? After the bad block scan was complete, I shut the unit down and started it back up, only to find that now the web interface and the front panel is saying that there is no RAID array found. Lots of fun.

I am fully well aware that I may have lost my data, but I was wondering what (if any) possible things can I try to do to fix the bad blocks on the disk that failed during the raid rebuild? Is there anything I can do? Are there any applications (such as spinrite) that I could use to try to recover the bad blocks?

I should mention I do have the SSH module installed on the NAS, and I am very comfortable with using the command line in Linux so if there is anything I can do via SSH I am fine with it. I was considering just going ahead and re-creating the RAID array, but I have a LOT of data on this NAS and am willing to do anything I can in order to recover whatever I can before I resign myself to losing everything.

Let me know if you need any additional technical details..

Thanks
Richard
Reply
#3
Hi Richard,

I'm sorry to hear of your RAID issues. Having a RAID failure is something that scares me too. I had issues with my N5550 when I first bought it 3.5 years ago, but thankfully the newer OS versions fixed the RAID dropping issues.

In my experience with Thecus and RAID rebuilds, the rebuild only starts automatically if the NAS is still powered on (i.e. disk fails and you install a new one whilst the unit is still running). Thankfully I haven't needed to test this.

Unfortunately I cannot help you with the data recovery part of your question.
Reply
#4
Hi Richard,

I'm sorry to hear of your RAID issues. Having a RAID failure is something that scares me too. I had issues with my N5550 when I first bought it 3.5 years ago, but thankfully the newer OS versions fixed the RAID dropping issues.

In my experience with Thecus and RAID rebuilds, the rebuild only starts automatically if the NAS is still powered on (i.e. disk fails and you install a new one whilst the unit is still running). Thankfully I haven't needed to test this.

Unfortunately I cannot help you with the data recovery part of your question.
Reply
#5
Dear Sir,

Following FAQ for your reference:
<!-- m --><a class="postlink" href="http://thecus.kayako.com/default_import/News/NewsItem/View/47/some-tips-for-tech-support">http://thecus.kayako.com/default_import ... ch-support</a><!-- m -->
Yvon.
Reply
#6
Dear Sir,

Following FAQ for your reference:
<!-- m --><a class="postlink" href="http://thecus.kayako.com/default_import/News/NewsItem/View/47/some-tips-for-tech-support">http://thecus.kayako.com/default_import ... ch-support</a><!-- m -->
Yvon.
Reply
#7
I just wanted to give a quick update on my original post regarding having 2 drives fail in a RAID 5 array. I contacted Thecus Support and had someone log into my NAS remotely and he found that not only did the 3rd drive have issues, but so did the 4th drive! The support person (Chase Lin, thank you btw!) suggested that I do the following to try to recover my RAID 5 array. The actual text from the ticket is below

Quote:According to your description, the raid was rebuilding with the new 1st disk but the rebuilding failed due to the bad 3rd disk.
I have checked all the status of all the disks on your NAS and I found the information below:
The 3rd disk:
======================
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 48
======================

The 4th disk:
======================
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 88
======================

These 2 disks are unstable disks which will cause the failure of the rebuilding process.
Unfortunately, we do not have related tool to fix the bad blocks of these disks.
We recommend to clone the 1st, 3rd and 4th disks and then we can try to recover the raid back.
But We will also need to confirm if there is any file system errors on your raid.
If there is file system errors, we will need to run file system check to fix the errors.
But we can't guarantee the process will be successful if the file system has too may errors.

If you accept the chance to be failed, please follow steps below and we will try to help recover the raid:
1. Power off your NAS
2. Remove the new 1st, 3rd, and 4th disks from your NAS.
3. Please clone the original 1st 3rd and 4th disks with healthy disks.
4. Once the cloning disks is successful, please add the newly cloned healthy disks back into your NAS by original orders.
For example:
The disk cloned by the original 1st disk should be added into the 1st slot of your NAS
5. Power on your NAS.
6. Please inform us and we will try to help to recover your original raid back.

If you have any problem, please let me know.
Thank you.

I purchased 2 additional 2TB Seagate hard drives and when they arrived I used a disk cloning application that I have a Technicians License for called HD Clone v6 (go to <!-- m --><a class="postlink" href="http://www.miray.de">http://www.miray.de</a><!-- m -->, there is a freeware version available but it limits the copy to 20MB/s.) to clone disk 3 and 4. I did encounter some read errors during the copy process, but as long as most of the data is still intact and readable you should be good. The process to copy the disks was LONG because of the read errors. I would say at least a full day per drive. If you are in a similar situation with two disks failing in a RAID 5 and decide to use HD Clone, just let the process complete even if it looks like it is stuck. When I was copying disk 3, I did get an error at the end saying there was some data that was unreadable, but I still had a copy of all of the readable parts of the disk.

After cloning both disks, I put everything back into the array and powered it up. My RAID 5 is rebuilding as I type this

[Image: Untitled.png]

If anyone ends up in a similar situation and needs some advise, feel free to reply to this thread and I will do my best to help.

Thanks
Richard
Reply
#8
I just wanted to give a quick update on my original post regarding having 2 drives fail in a RAID 5 array. I contacted Thecus Support and had someone log into my NAS remotely and he found that not only did the 3rd drive have issues, but so did the 4th drive! The support person (Chase Lin, thank you btw!) suggested that I do the following to try to recover my RAID 5 array. The actual text from the ticket is below

Quote:According to your description, the raid was rebuilding with the new 1st disk but the rebuilding failed due to the bad 3rd disk.
I have checked all the status of all the disks on your NAS and I found the information below:
The 3rd disk:
======================
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 48
======================

The 4th disk:
======================
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 88
======================

These 2 disks are unstable disks which will cause the failure of the rebuilding process.
Unfortunately, we do not have related tool to fix the bad blocks of these disks.
We recommend to clone the 1st, 3rd and 4th disks and then we can try to recover the raid back.
But We will also need to confirm if there is any file system errors on your raid.
If there is file system errors, we will need to run file system check to fix the errors.
But we can't guarantee the process will be successful if the file system has too may errors.

If you accept the chance to be failed, please follow steps below and we will try to help recover the raid:
1. Power off your NAS
2. Remove the new 1st, 3rd, and 4th disks from your NAS.
3. Please clone the original 1st 3rd and 4th disks with healthy disks.
4. Once the cloning disks is successful, please add the newly cloned healthy disks back into your NAS by original orders.
For example:
The disk cloned by the original 1st disk should be added into the 1st slot of your NAS
5. Power on your NAS.
6. Please inform us and we will try to help to recover your original raid back.

If you have any problem, please let me know.
Thank you.

I purchased 2 additional 2TB Seagate hard drives and when they arrived I used a disk cloning application that I have a Technicians License for called HD Clone v6 (go to <!-- m --><a class="postlink" href="http://www.miray.de">http://www.miray.de</a><!-- m -->, there is a freeware version available but it limits the copy to 20MB/s.) to clone disk 3 and 4. I did encounter some read errors during the copy process, but as long as most of the data is still intact and readable you should be good. The process to copy the disks was LONG because of the read errors. I would say at least a full day per drive. If you are in a similar situation with two disks failing in a RAID 5 and decide to use HD Clone, just let the process complete even if it looks like it is stuck. When I was copying disk 3, I did get an error at the end saying there was some data that was unreadable, but I still had a copy of all of the readable parts of the disk.

After cloning both disks, I put everything back into the array and powered it up. My RAID 5 is rebuilding as I type this

[Image: Untitled.png]

If anyone ends up in a similar situation and needs some advise, feel free to reply to this thread and I will do my best to help.

Thanks
Richard
Reply
#9
Richie086 Wrote:Hi all,

Is there a repair tool or some way to fix these bad blocks so I can continue rebuilding the RAID?

Thanks
Richard

Steve Gibson's Spinrite https://www.grc.com/sr/spinrite.htm is often able to work miracles. It usually recovers unreadable and even undetectable HDDs. The current version 6.0 can take a long time (hours!) to scan and repair 2TB drives. You would have to run it on a x86 box that than read the HDDs. The underlying OS / file structure is unimportant, as the software works at the sector level.

He's working on version 6.1, a more modern version, which will/does use more RAM, more modern CPU capabilities, in order to cut down the scan/repair times for today's huge drives. It will be a free upgrade when it's available.
Thecus N5550 (CPU: D2550 @ 1.86GHz, BIOS: 2.05.14.5.cdv X64)
Disks: 5 x 3TB Seagate NAS-class; 2 x 3TB-RAID1 plus Hot Spare
Memory: 8125 MB (2 x 4 GB Corsair 204-Pin DDR3 SO-DIMM 1333)
Apps: Too many to count....
Reply
#10
Richie086 Wrote:Hi all,

Is there a repair tool or some way to fix these bad blocks so I can continue rebuilding the RAID?

Thanks
Richard

Steve Gibson's Spinrite https://www.grc.com/sr/spinrite.htm is often able to work miracles. It usually recovers unreadable and even undetectable HDDs. The current version 6.0 can take a long time (hours!) to scan and repair 2TB drives. You would have to run it on a x86 box that than read the HDDs. The underlying OS / file structure is unimportant, as the software works at the sector level.

He's working on version 6.1, a more modern version, which will/does use more RAM, more modern CPU capabilities, in order to cut down the scan/repair times for today's huge drives. It will be a free upgrade when it's available.
Thecus N5550 (CPU: D2550 @ 1.86GHz, BIOS: 2.05.14.5.cdv X64)
Disks: 5 x 3TB Seagate NAS-class; 2 x 3TB-RAID1 plus Hot Spare
Memory: 8125 MB (2 x 4 GB Corsair 204-Pin DDR3 SO-DIMM 1333)
Apps: Too many to count....
Reply
#11
Agreed, I own a copy of SpinRite 6 and it is scanning one of the failed drives now. It has been going for days, and it is not even half way done though Sad
Reply
#12
Agreed, I own a copy of SpinRite 6 and it is scanning one of the failed drives now. It has been going for days, and it is not even half way done though Sad
Reply
#13
Richard,

I now have the same issue on a N4100Pro (4x Seagate 1Tb).
I have tried Clonezilla, but it doesn't seem to copy the data, just the partition.
I notice you mention the Technician version of HD Clone, do you know if the Standard edition will work with Linux RAID 5?

Thanks
David
Reply
#14
Is is possible to use the Clone function built into my N5550 OS to clone the data from a failing drive to a new identical drive? I am in the same boat with 2 problematic drives that happened simultaneously in a RAID 5. Thanks much.




Jeff
Reply
#15
(06-06-2017, 12:11 AM)soniccake Wrote: Is is possible to use the Clone function built into my N5550 OS to clone the data from a failing drive to a new identical drive? I am in the same boat with 2 problematic drives that happened simultaneously in a RAID 5. Thanks much.

Jeff

My best advice is to purchase a copy of SpinRite and run it on those drive.  Don't wait.

My next best advice is in the form of a question: Are those HDDs NAS-class drives?  If not, upgrade them to NAS-class.  Get bigger drives so that you can replicate your RAID as a four-drive RAID, and install the fifth drive as a Hot Spare (so the NAS rebuilds the RAID as soon as it detects a degraded RAID, or better still use RAID6, or best, IMHO, use RAID10 instead of RAID5 and have a Hot Spare.

Third: the Clone function, from what I've read, (I've never used it) is to clone drive through, say, the USB port.  Its function is similar to the Norton Ghost utility.  What you're looking for, in Thecus, is the 'Raid Volume Replication'.  Problem is, you need as many spare slots/HDDs as your old RAID is composed of.
Thecus N5550 (CPU: D2550 @ 1.86GHz, BIOS: 2.05.14.5.cdv X64)
Disks: 5 x 3TB Seagate NAS-class; 2 x 3TB-RAID1 plus Hot Spare
Memory: 8125 MB (2 x 4 GB Corsair 204-Pin DDR3 SO-DIMM 1333)
Apps: Too many to count....
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Amazon Cloud Drive for Thecus NAS (like N5550) ? Dandy 16 4,075 09-03-2016, 06:42 PM
Last Post: Dandy
  Failed harddisk in Raid 1, rebuild OK and long SMART test OK vdhaeyere 32 6,959 12-24-2012, 10:50 PM
Last Post: tbt
  Failed harddisk in Raid 1, rebuild OK and long SMART test OK 0 3,484 Less than 1 minute ago
Last Post:
  Amazon Cloud Drive for Thecus NAS (like N5550) ? 0 1,110 Less than 1 minute ago
Last Post:
  Thecus N5550 RAID 5 - 2nd Drive Failure during rebuild! 0 587 Less than 1 minute ago
Last Post:

Forum Jump:


Users browsing this thread: 1 Guest(s)