System Memory not Recognised

Submitted by Tempest on Wed, 04/22/2015 - 08:32 Pro Licensee

Hi, we just spooled up a new server to migrate over to, which has 16GB of ram, after installing the base Centos6.6_x64 system the memory is recognised, Once the xen kernel is installed, cloudmin and only reports 1024mb available thus making it useless to create new vm's impossible, I know that xen is dead and its now xl, I'm wondering if cloudmin isn't using the right commands to get available info updated.. after doing some back peddling with the xen host kernel and using nothing newer than Linux 3.10.63-11.el6.centos.alt.x86_64 on x86_64 I can know create vm's however i cannot shut them down using cloudmin as xend is deprecated and xl is used. Any news on cloudmin using xl toolset? I saw a post about this dated 2015-03-07 08:20 and another dated 2013-08-16 15:31 What OS should we be using until this is fixed?

Status: 
Active

Comments

Submitted by andreychek on Wed, 04/22/2015 - 09:25

Howdy -- CentOS 6 is a good distro/version to use. I hadn't heard of an issue with it not detecting all the available RAM before.

What is the output of this command --

free -m

Submitted by Tempest on Wed, 04/22/2015 - 09:34 Pro Licensee

Hi Andy,

[root@ch1 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           925        852         73          0          5        588
-/+ buffers/cache:        257        668
Swap:         4093          0       4093

Submitted by Tempest on Wed, 04/22/2015 - 09:33 Pro Licensee

and xl info

[root@ch1 ~]# xl info
host                   : ch1.tws99.co.uk
release                : 3.10.63-11.el6.centos.alt.x86_64
version                : #1 SMP Tue Jan 6 10:23:44 CST 2015
machine                : x86_64
nr_cpus                : 8
max_cpu_id             : 15
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2800
hw_caps                : bfebfbff:28100800:00000000:00003b40:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 16375
free_memory            : 11039
sharing_freed_memory   : 0
sharing_used_memory    : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 2
xen_extra              : .5-37.el6
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : dom0_mem=1024M,max:1024M loglvl=all guest_loglvl=all
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Mon Oct 20 08:33:55 CDT 2014
xend_config_format     : 4

Submitted by andreychek on Wed, 04/22/2015 - 09:39

Yeah that definitely doesn't seem right!

It looks like it's only seeing 1GB of RAM.

Are you saying that if you boot into a different kernel, that all your RAM is appearing?

If so, could you try that, and then paste in the output you receive from "free -m" when it's working properly?

Submitted by Tempest on Wed, 04/22/2015 - 09:41 Pro Licensee

the main issues i had was upon install the kernel was 3.10.68-11.el6.centos.alt after downgrading the kernel the memory was reporting fine and xen version that was installed was 4.4.1-10.el6 which killed cloudmin's admin of xen I managed to downgrade the xen as well - it seems to be fine, but now obviously scared to do any updates on the host incase something breaks these are the packages i have managed to use to get things to run with cloudmin so far so good, but any newer packages - i found it isn't happening

[root@ch1 ~]# yum --showduplicates list xen | expand
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.mirrors.ovh.net
* extras: mirror0.babylon.network
* updates: centos.mirrors.ovh.net
Installed Packages
xen.x86_64                  4.2.5-37.el6.centos.alt                  @Xen4CentOS
Available Packages
xen.x86_64                  4.2.5-37.el6.centos.alt                  Xen4CentOS
xen.x86_64                  4.2.5-38.el6.centos.alt                  Xen4CentOS
xen.x86_64                  4.4.1-5.el6                              Xen4CentOS
xen.x86_64                  4.4.1-7.el6.centos.alt                   Xen4CentOS
xen.x86_64                  4.4.1-8.el6                              Xen4CentOS
xen.x86_64                  4.4.1-10.el6                             Xen4CentOS
[root@ch1 ~]# yum --showduplicates list kernel | expand
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.mirrors.ovh.net
* extras: mirror0.babylon.network
* updates: centos.mirrors.ovh.net
Installed Packages
kernel.x86_64               2.6.32-504.12.2.el6                      @updates  
kernel.x86_64               3.10.63-11.el6.centos.alt                @Xen4CentOS
kernel.x86_64               3.10.68-11.el6.centos.alt                @Xen4CentOS
Available Packages
kernel.x86_64               2.6.32-504.el6                           base      
kernel.x86_64               2.6.32-504.1.3.el6                       updates   
kernel.x86_64               2.6.32-504.3.3.el6                       updates   
kernel.x86_64               2.6.32-504.8.1.el6                       updates   
kernel.x86_64               2.6.32-504.12.2.el6                      updates   
kernel.x86_64               2.6.32-504.16.2.el6                      updates   
kernel.x86_64               3.10.56-11.el6.centos.alt                Xen4CentOS
kernel.x86_64               3.10.63-11.el6.centos.alt                Xen4CentOS
kernel.x86_64               3.10.68-11.el6.centos.alt                Xen4CentOS

Submitted by Tempest on Wed, 04/22/2015 - 09:53 Pro Licensee

The free-m i posted above is whats being reported with the downgraded kernel and through cloudmin interface is reporting that i have 16gb less host (to save scrolling up ;-) )

[root@ch1 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           925        844         81          0          7        577
-/+ buffers/cache:        258        667
Swap:         4093          0       4092

seems i'm unable to attach a screen shot of the cloudmin interface showing so i've uploaded it to http://www.tws99.com/downloads/cloudmin.png

Submitted by JamieCameron on Wed, 04/22/2015 - 22:06

This could be because you have only 1GB reserved for the host system itself, with the rest available to create VMs. However, Cloudmin should detect this case, and allow you to use the remaining 15 GB for creating VMs.

Actually, based on your screenshot it does appear that this is working fine - you can see this in the "Real memory available for Xen hosting" field.

Submitted by Tempest on Thu, 04/23/2015 - 05:44 Pro Licensee

Hi Jamie,
yes it's working now only because I managed to downgrade the kernel and the XEN packages to
xen-4.2.5-37.el6.centos.alt and kernel-3.10.63-11.el6.centos.alt

I do understand that certain amount of ram is reserved for the host, and
when I first installed everything, by default xen-4.4.1-5.el6 and kernel-3.10.68-11.el6.centos.alt was installed, and it was under those versions that cloudmin couldn't see the memory,
tbh I think it is either kernels fault or cloudmin because it couldn't fully control a vps as it came back with an error saying xend was deprecated etc and to use XL when trying to reboot a vps

I have managed to retrieve the original outputs before i downgraded the system (love my mac terminal ;-))
[root@ch1 ~]# cat /proc/meminfo
MemTotal: 947764 kB
MemFree: 143904 kB


CentOS release 6.6 (Final)
Linux ch1.tws99.co.uk 3.10.68-11.el6.centos.alt.x86_64 #1 SMP Fri Feb 6 10:40:16 CST 2015 x86_64 x86_64 x86_64 GNU/Linux

server : 133750
hostname : ch1.tws99.co.uk
eth0 IPv4 :

[root@ch1 ~]# htop
-bash: htop: command not found
[root@ch1 ~]# top

top - 00:17:45 up 35 min, 1 user, load average: 0.08, 0.09, 0.11
Tasks: 149 total, 1 running, 148 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 947764k total, 803624k used, 144140k free, 10060k buffers
Swap: 4093944k total, 0k used, 4093944k free, 583584k cached

[root@ch1 ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 143520 10072 583588 0 0 87 87 55 43 1 0 98 1 0

Initializing cgroup subsys cpuacct
Linux version 3.10.68-11.el6.centos.alt.x86_64 (mockbuild@sclbuild) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Fri Feb 6 10:40:16 CST 2015
[root@ch1 ~]# head -1 /proc/meminfo
MemTotal: 947764 kB
[root@ch1 ~]# dmidecode -t17 | grep -e Size -e 'Locator: DIMM'
Size: No Module Installed
Locator: DIMM1A
Size: No Module Installed
Locator: DIMM1B
Size: 8192 MB
Locator: DIMM2A
Size: No Module Installed
Locator: DIMM2B
Size: 8192 MB
Locator: DIMM3A
Size: No Module Installed
Locator: DIMM3B
Size: 4096 kB
[root@ch1 ~]# free -m
total used free shared buffers cached
Mem: 925 790 134 0 11 570
-/+ buffers/cache: 208 717
Swap: 3997 0 3997
[root@ch1 ~]# uname -a
Linux ch1.tws99.co.uk 3.10.68-11.el6.centos.alt.x86_64 #1 SMP Fri Feb 6 10:40:16 CST 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@ch1 ~]# cat /proc/meminfo
MemTotal: 947764 kB
MemFree: 137300 kB
Buffers: 11900 kB
Cached: 584512 kB
SwapCached: 0 kB
Active: 91396 kB
Inactive: 612480 kB
Active(anon): 7676 kB
Inactive(anon): 100020 kB
Active(file): 83720 kB
Inactive(file): 512460 kB
Unevictable: 20 kB
Mlocked: 20 kB
SwapTotal: 4093944 kB
SwapFree: 4093944 kB
Dirty: 48 kB
Writeback: 0 kB
AnonPages: 107504 kB
Mapped: 16968 kB
Shmem: 220 kB
Slab: 69564 kB
SReclaimable: 19832 kB
SUnreclaim: 49732 kB
KernelStack: 1456 kB
PageTables: 3508 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4567824 kB
Committed_AS: 340540 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 274888 kB
VmallocChunk: 34359450620 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 1048988 kB
DirectMap2M: 0 kB


after rebooting back to the stock centos.6.6 kernel
Last login: Wed Apr 22 01:26:29 2015 from cache.ovh.net
CentOS release 6.6 (Final)
Linux ch1.tws99.co.uk 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

server : 133750
hostname : ch1.tws99.co.uk
eth0 IPv4 : 188.165.245.190
eth0 IPv6 : 2001:41d0:2:c0be::/64

[root@ch1 ~]# cat /proc/meminfo
MemTotal: 16455168 kB
MemFree: 16137584 kB
[root@ch1 ~]# dmidecode -t17 | grep -e Size -e 'Locator: DIMM'
Size: No Module Installed
Locator: DIMM1A
Size: No Module Installed
Locator: DIMM1B
Size: 8192 MB
Locator: DIMM2A
Size: No Module Installed
Locator: DIMM2B
Size: 8192 MB
Locator: DIMM3A
Size: No Module Installed
Locator: DIMM3B
Size: 4096 kB
[root@ch1 ~]# free -m
total used free shared buffers cached
Mem: 16069 311 15757 0 6 80
-/+ buffers/cache: 225 15843
Swap: 4093 0 4093

Submitted by JamieCameron on Thu, 04/23/2015 - 18:52

Did you happen to run xm info before downgrading?