Buffer I/O error on device loop0

Submitted by rubenzsolt on Fri, 12/27/2013 - 05:10

I have installed Cloudmin 7.4.kvm, I create on vps with regular disk but issue appear if the vps is created with LVM disk, and I receive next errors every 5 minutes on syslog, I think when cloudmin or virtualmin check: Dec 27 11:50:17 server kernel: [ 8721.244784] quiet_error: 35 callbacks suppressed Dec 27 11:50:17 server kernel: [ 8721.244792] Buffer I/O error on device loop0, logical block 1310704 Dec 27 11:50:17 server kernel: [ 8721.245883] Buffer I/O error on device loop0, logical block 1310704 Dec 27 11:50:17 server kernel: [ 8721.246543] Buffer I/O error on device loop0, logical block 1310704 Dec 27 11:50:17 server kernel: [ 8721.247226] Buffer I/O error on device loop0, logical block 1310718 Dec 27 11:50:17 server kernel: [ 8721.247852] Buffer I/O error on device loop0, logical block 1310718 Dec 27 11:50:17 server kernel: [ 8721.253895] Buffer I/O error on device loop0, logical block 1310718 Dec 27 11:50:17 server kernel: [ 8721.254742] Buffer I/O error on device loop0, logical block 1310719 Dec 27 11:50:17 server kernel: [ 8721.255318] Buffer I/O error on device loop0, logical block 1310719 Dec 27 11:50:17 server kernel: [ 8721.255843] Buffer I/O error on device loop0, logical block 1310719 Dec 27 11:50:17 server kernel: [ 8721.256630] Buffer I/O error on device loop0, logical block 1310719 Dec 27 11:50:17 server kernel: [ 8721.277205] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Fri, 12/27/2013 - 15:24

Hmm, that's an unusual error! Are you seeing any problems other than that error in the logs though?

And just to clarify -- are you saying that error is only reported when using LVM, it doesn't occur when the disk isn't using LVM?

Submitted by rubenzsolt on Fri, 12/27/2013 - 17:04

the error appear just when the vps is ON, if I turn OFF the vps the issue remain until I not restart the server, and after I restarted the server the error not appear just if I start one vps. I tested with regular file or LVM same errors on log.

Submitted by JamieCameron on Fri, 12/27/2013 - 20:15

Also, what caching mode do you have selected for this VM's disks? You can see this at Resources -> Manage Disks -> first disk, in the "Caching mode" field.

Submitted by rubenzsolt on Sat, 12/28/2013 - 18:26

now i use just regular img files on vps and it is set to writethrough.

Submitted by JamieCameron on Sun, 12/29/2013 - 13:11

Does changing the cache mode stop the errors from happening?

Submitted by rubenzsolt on Tue, 01/14/2014 - 02:41

I make a lot of tests, I changed the cache mode, but nothing changes, the error appear in every 5minutes.

Submitted by rubenzsolt on Tue, 01/14/2014 - 02:52

I make a lot of tests, I changed the cache mode, but nothing changes, the error appear in every 5minutes.

Submitted by JamieCameron on Tue, 01/14/2014 - 11:40

So other than the warning, is this causing any problems with your VMs?

Submitted by rubenzsolt on Tue, 01/21/2014 - 17:20

In general not causing another problems, not find another problems, but on shell apear this warnings and fills the screen with this warning.

Jan 21 23:50:01 server /USR/SBIN/CRON[19154]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi) Jan 21 23:50:01 server /USR/SBIN/CRON[19157]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi) Jan 21 23:50:20 server kernel: [11600.333296] quiet_error: 157 callbacks suppressed Jan 21 23:50:20 server kernel: [11600.333303] Buffer I/O error on device loop0, logical block 1310704 Jan 21 23:50:20 server kernel: [11600.333854] Buffer I/O error on device loop0, logical block 1310704 Jan 21 23:50:20 server kernel: [11600.334397] Buffer I/O error on device loop0, logical block 1310718 Jan 21 23:50:20 server kernel: [11600.334927] Buffer I/O error on device loop0, logical block 1310718 Jan 21 23:50:20 server kernel: [11600.335499] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:50:20 server kernel: [11600.336176] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:50:20 server kernel: [11600.336784] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:50:20 server kernel: [11600.337328] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:50:20 server kernel: [11600.337849] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:50:20 server kernel: [11600.338377] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:50:20 server kernel: [11600.348896] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:50:20 server kernel: [11600.482092] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:50:20 server kernel: [11600.521292] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:50:20 server kernel: [11600.622756] EXT4-fs (loop3): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:55:01 server /USR/SBIN/CRON[20340]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi) Jan 21 23:55:01 server /USR/SBIN/CRON[20341]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi) Jan 21 23:55:20 server kernel: [11900.044173] quiet_error: 171 callbacks suppressed Jan 21 23:55:20 server kernel: [11900.044180] Buffer I/O error on device loop0, logical block 1310704 Jan 21 23:55:20 server kernel: [11900.044722] Buffer I/O error on device loop0, logical block 1310704 Jan 21 23:55:20 server kernel: [11900.045253] Buffer I/O error on device loop0, logical block 1310718 Jan 21 23:55:20 server kernel: [11900.045770] Buffer I/O error on device loop0, logical block 1310718 Jan 21 23:55:20 server kernel: [11900.046282] Buffer I/O error on device loop0, logical block 1310704 Jan 21 23:55:20 server kernel: [11900.046834] Buffer I/O error on device loop0, logical block 1310718 Jan 21 23:55:20 server kernel: [11900.047372] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:55:20 server kernel: [11900.047902] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:55:20 server kernel: [11900.048686] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:55:20 server kernel: [11900.049311] Buffer I/O error on device loop0, logical block 1310719 Jan 21 23:55:20 server kernel: [11900.072758] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:55:20 server kernel: [11900.289407] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:55:20 server kernel: [11900.290122] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:55:20 server kernel: [11900.337525] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Jan 21 23:57:51 server postfwd2/master[3857]: Jan 22 00:00:01 server /USR/SBIN/CRON[22073]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi) Jan 22 00:00:01 server /USR/SBIN/CRON[22072]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi) Jan 22 00:00:10 server kernel: [12190.734222] quiet_error: 158 callbacks suppressed Jan 22 00:00:10 server kernel: [12190.734229] Buffer I/O error on device loop0, logical block 2621424 Jan 22 00:00:10 server kernel: [12190.734767] Buffer I/O error on device loop0, logical block 2621424 Jan 22 00:00:10 server kernel: [12190.735297] Buffer I/O error on device loop0, logical block 2621438 Jan 22 00:00:10 server kernel: [12190.735818] Buffer I/O error on device loop0, logical block 2621438 Jan 22 00:00:10 server kernel: [12190.736658] Buffer I/O error on device loop0, logical block 2621439 Jan 22 00:00:10 server kernel: [12190.737187] Buffer I/O error on device loop0, logical block 2621439 Jan 22 00:00:10 server kernel: [12190.737798] Buffer I/O error on device loop0, logical block 2621439 Jan 22 00:00:10 server kernel: [12190.738341] Buffer I/O error on device loop0, logical block 2621439 Jan 22 00:00:10 server kernel: [12190.738869] Buffer I/O error on device loop0, logical block 2621439 Jan 22 00:00:10 server kernel: [12190.739397] Buffer I/O error on device loop0, logical block 2621439 Jan 22 00:00:10 server kernel: [12190.753578] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Jan 22 00:00:11 server kernel: [12190.829028] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null) Jan 22 00:00:11 server kernel: [12190.951613] EXT4-fs (loop2): mounted filesystem with ordered data mode. Opts: (null) Jan 22 00:00:11 server kernel: [12190.991530] EXT4-fs (loop3): mounted filesystem with ordered data mode. Opts: (null)

Submitted by sennalb on Tue, 06/10/2014 - 16:24

Also I have the same problem with clodumin 7.7 on wheezy with LVM PowerEdge T620 with PERC H710 controller.

Is there same new ? Thanks

Submitted by sennalb on Fri, 06/13/2014 - 12:34

All kernel error messages are stopping if I umount /kvm partition, ed if i remount /kvm partition, where is all my virtual machine, the message reappears

Any ideas ? Thanks

Submitted by JamieCameron on Fri, 06/13/2014 - 13:48

Perhaps corruption or filesystem errors on the device underlying /kvm?

Submitted by andreychek on Fri, 06/13/2014 - 13:50

Doing some research on those kernel errors, I see a number of folks posting in various communities on the Internet that they receive those errors -- however, there isn't a clear cause, or resolution.

You could try updating the firmware of your controller, or updating your kernel, as those helped some people. But I'm not sure that doing so would help.

Since things are otherwise working properly, and the main issue is just that it's filling up your console with log messages, you could always tweak your system to prevent those kernel messages from appearing on the console.

To do that, we can tweak the "kernel.printk" line in /etc/sysctl.conf.

What is the output of this command:

grep printk /etc/sysctl.conf

Submitted by sennalb on Fri, 06/13/2014 - 14:59

Hi Jamie,

I have formatted file system with ext 4 twice with -c option for badblocks and all is OK, file sistems are on RAID 5 with PERC 710H Dell Controller.

Thanks

Submitted by sennalb on Fri, 06/13/2014 - 15:11

Hi andr,eychek

the output is:

~# grep printk /etc/sysctl.conf

kernel.printk = 3 4 1 3

root@srv1:~#

I should add that I note the same problem with another server with the same software configuration: Debian wheezy Base configuration + kvm + clodumin 7.7 hw but with completely different hardware.

Tanks sennalb

Submitted by andreychek on Fri, 06/13/2014 - 15:18

Okay, let's try this as a test -- try running this command on your server as root:

echo "4 4 1 7" > /proc/sys/kernel/printk

After that, do you receive any more kernel errors on the console? Or does that suppress those?

If that works properly, then try editing the /etc/sysctl.conf file, and change that kernel.printk line to read as follows:

kernel.printk = 4 4 1 7

Submitted by sennalb on Fri, 06/13/2014 - 16:05

Now my sysctl.con is

root@matrix2v:~# cat /etc/sysctl.conf

net.ipv4.ip_forward=0

kernel.printk = 4 4 1 7

root@matrix2v:~#

and then I reboot the machine and appear the same massages:

Jun 13 22:30:10 matrix2v kernel: [ 131.532148] Buffer I/O error on device loop0, logical block 2621424 Jun 13 22:30:10 matrix2v kernel: [ 131.532172] Buffer I/O error on device loop0, logical block 2621424 Jun 13 22:30:10 matrix2v kernel: [ 131.532190] Buffer I/O error on device loop0, logical block 2621438 Jun 13 22:30:10 matrix2v kernel: [ 131.532202] Buffer I/O error on device loop0, logical block 2621438 Jun 13 22:30:10 matrix2v kernel: [ 131.557277] Buffer I/O error on device loop0, logical block 2621424 Jun 13 22:30:10 matrix2v kernel: [ 131.557587] Buffer I/O error on device loop0, logical block 2621438 Jun 13 22:30:10 matrix2v kernel: [ 131.557615] Buffer I/O error on device loop0, logical block 2621439 Jun 13 22:30:10 matrix2v kernel: [ 131.557633] Buffer I/O error on device loop0, logical block 2621439 Jun 13 22:30:10 matrix2v kernel: [ 131.557645] Buffer I/O error on device loop0, logical block 2621439 Jun 13 22:30:10 matrix2v kernel: [ 131.557738] Buffer I/O error on device loop0, logical block 2621439 Jun 13 22:30:10 matrix2v kernel: [ 131.934181] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Jun 13 22:30:11 matrix2v kernel: [ 132.083766] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)

Tanks a lot for yor help

sennalb

Submitted by andreychek on Fri, 06/13/2014 - 16:09

Just to verify, are those messages appearing on the console itself? Or are those in the log files and/or dmesg output?

Submitted by sennalb on Fri, 06/13/2014 - 16:38

Those are from syslog, I work from remote and now I have not console access

Submitted by andreychek on Fri, 06/13/2014 - 16:53

Ah, yes, that's expected.

The changes we made aren't preventing the log message -- as we don't know what's causing them. The change we made is only preventing them from being pushed onto the console.

There isn't a clear cause, or fix, for those errors. It could be related to the controller, the kernel version, or some other device-related issue.

You don't appear to be seeing a Cloudmin issue though.

If you aren't seeing any filesystem corruption, and things are otherwise working properly for you -- then all we can do is prevent those error from being sent to the console.

Just to verify though -- which kernel version are you using, is it 3.5.x? You can determine that by running the command "uname -a".

Submitted by sennalb on Fri, 06/13/2014 - 17:16

Hi andreychek,

Also for me seems not a Cloumin issue, but only on this forum I have found a post of the same exact problem on the same s.o. and Cloudmin.

I work on Cloudmin with old stable Debian servers by two years and I have no problem!

On google there is a lot about "Buffer I/O error on device" but not for loop0/1 on KVM server.

Then my uanme -a is:

Linux matrix2v1 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u2 x86_64 GNU/Linux

Tanks a lot

sennalb

Submitted by sennalb on Thu, 06/19/2014 - 15:51

Hi,

I have solved the problem renaimg mount dir from /kvm to another name

thanks to all

sennalb

Submitted by sennalb on Fri, 06/20/2014 - 11:05

HI,

message is back again, after some hour !! It is not solved!!!

sennalb

Submitted by andreychek on Fri, 06/20/2014 - 11:47

The errors you're receiving are kernel errors -- and we aren't sure what's causing them, since other than the log messages there's no problem occurring.

Since they aren't Cloudmin errors, and you don't seem to be experiencing any problems, our suggestion is to just ignore the errors.

The only other person we saw in the forums with those same errors was using the same kernel version as you're using -- so those messages may just be an issue with that particular kernel. If that's the case, it may be corrected when your distribution publishes a kernel update.

Since it's the kernel producing an error message, but there otherwise aren't any problems you're seeing, there unfortunately isn't anything we can do to prevent those messages from occurring.

Submitted by sennalb on Fri, 06/20/2014 - 13:05

Hi, andreychek

I agree with what you say, it was only to correct the incorrect information that I have report for the other users of this forum

Thanks sennalb