corrupt disk: *urgent*

First response from swsoft support:

You have corrupted filesystem on the physical server. This most likely related to bad disks / disk controller. The first report about the broken filesystem appeared in logs about 3 hours ago:

Mar 31 09:31:02 server001 kernel: scsi: Device offlined - not ready after error recovery: host 0 channel 0 id 1 lun 0 Mar 31 09:31:03 server001 kernel: SCSI error : <0 0 1 0> return code = 0x8000002 Mar 31 09:31:03 server001 kernel: Info fld=0x0, Current sdb: sense key Aborted Command Mar 31 09:31:03 server001 kernel: end_request: I/O error, dev sdb, sector 57402769 Mar 31 09:31:03 server001 kernel: Buffer I/O error on device sdb1, logical block 28701353 Mar 31 09:31:03 server001 kernel: lost page write due to I/O error on sdb1 Mar 31 09:31:03 server001 kernel: scsi0 (1:0): rejecting I/O to offline device

As filesystem (and journal) is marked as corrupted filesystem can be operated in read-only mode.

Please do NOT reboot the node - there is a good chance that all files on the filesystem will be lost. The best bet would be to manually rsync all VPSs data to another Virtuozzo box, start them there and only then reboot and start fsck.

Again, very important - do NOT reboot the server until you will be able to save / migrate all the data.


Second response:


Migration / data copy won't work as system doesn't show any files or directories in /vz/ directory anymore:

[root@server001 root]# ls -la /vz
total 0
[root@server001 root]#

While it reports some disk usage there:

[root@server001 root]# df -h /vz
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 245G 23G 208G 10% /vz
[root@server001 root]#

The disk subsystem is not working well, there is a huge number of messages like below:

scsi0 (1:0): rejecting I/O to offline device scsi0 (1:0): rejecting I/O to offline device scsi0 (1:0): rejecting I/O to offline device scsi0 (1:0): rejecting I/O to offline device


So, the only way would be to reboot the server, check controller / disks status and then force fsck with a hope that it can repair the filesystem. However the chance that fsck will success are quite low.
If fsck won't help, then only offsite backups (if any) restoration will help to bring VPSs data back.


What is recommended in this case without taking chances of losing all data. There is no backup as it is a VPS server

 

 

 

 

Top