I cannot figure what's wrong. DB not starting at all.....
service mariadb status -l
Redirecting to /bin/systemctl status -l mariadb.service
● mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-01-27 19:50:43 CST; 30min ago
Process: 897 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 896 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 868 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 896 (code=exited, status=0/SUCCESS)
Jan 27 19:50:42 ns03.bislinks.com systemd[1]: Starting MariaDB database server...
Jan 27 19:50:42 ns03.bislinks.com mysqld_safe[896]: 170127 19:50:42 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Jan 27 19:50:43 ns03.bislinks.com mysqld_safe[896]: 170127 19:50:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jan 27 19:50:43 ns03.bislinks.com mysqld_safe[896]: 170127 19:50:43 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Jan 27 19:50:43 ns03.bislinks.com systemd[1]: mariadb.service: control process exited, code=exited status=1
Jan 27 19:50:43 ns03.bislinks.com systemd[1]: Failed to start MariaDB database server.
Jan 27 19:50:43 ns03.bislinks.com systemd[1]: Unit mariadb.service entered failed state.
Jan 27 19:50:43 ns03.bislinks.com systemd[1]: mariadb.service failed.
Appreciate help.
Update: This is actually a retired server, behind ATT router. But I need to backup dbs from this server to their counterparts on DreamHost. I do have full access to it, locally.
Status:
Active
Comments
Submitted by andreychek on Fri, 01/27/2017 - 23:19 Comment #2
Howdy -- you may want to take a look at the MariaDB log file, located in /var/log/mariadb/mariadb.log.
That should contain info regarding why it's not starting up.
Here are the last few lines from mariadb.log.
170125 11:42:12 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 09:53:49 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170127 9:53:54 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
170127 9:53:54 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 2708 ...
170127 9:53:56 InnoDB: The InnoDB memory heap is disabled
170127 9:53:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170127 9:53:56 InnoDB: Compressed tables use zlib 1.2.7
170127 9:53:56 InnoDB: Using Linux native AIO
170127 9:53:57 InnoDB: Initializing buffer pool, size = 128.0M
170127 9:53:57 InnoDB: Completed initialization of buffer pool
170127 9:53:58 InnoDB: highest supported file format is Barracuda.
170127 9:54:47 InnoDB: Waiting for the background threads to start
170127 9:54:48 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-38.0 started; log sequence number 1259827222
170127 9:54:50 [Note] Plugin 'FEEDBACK' is disabled.
170127 9:54:53 [Note] Server socket created on IP: '0.0.0.0'.
170127 9:54:55 [Note] Event Scheduler: Loaded 0 events
170127 9:54:55 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
170127 10:28:18 [Note] /usr/libexec/mysqld: Normal shutdown
170127 10:28:18 [Note] Event Scheduler: Purging the queue. 0 events
170127 10:28:19 InnoDB: Starting shutdown...
170127 10:28:20 InnoDB: Shutdown completed; log sequence number 1259827222
170127 10:28:20 [Note] /usr/libexec/mysqld: Shutdown complete
170127 10:28:20 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 10:42:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 10:42:11 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 10:59:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 10:59:30 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 13:18:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 13:18:04 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 13:19:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 13:19:14 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 13:22:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 13:22:10 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 13:23:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 13:23:37 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 13:24:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 13:24:28 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 13:36:22 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 13:36:22 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 14:56:21 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 14:56:21 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170127 19:50:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170127 19:50:43 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170128 09:23:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170128 09:23:30 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170128 09:25:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170128 09:25:20 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170128 09:28:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170128 09:28:27 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170128 09:30:27 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170128 09:30:27 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
170128 09:31:47 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: /usr/libexec/mysqld: cannot execute binary file
170128 09:31:47 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Submitted by andreychek on Sat, 01/28/2017 - 12:04 Comment #4
Hmm, was a MySQL/MariaDB recently installed from a third party, or otherwise modified in some unusual way?
What is the output of these commands:
rpm -qa | grep -i marials -l /usr/libexec/mysqld
file /usr/libexec/mysqld
uname -a
rpm -qa | grep -i maria
mariadb-server-5.5.52-1.el7.x86_64
mariadb-libs-5.5.52-1.el7.x86_64
mariadb-devel-5.5.52-1.el7.x86_64
mariadb-5.5.52-1.el7.x86_64
ls -l /usr/libexec/mysqld
-rwxr-xr-x 1 root root 14287200 Nov 14 17:17 /usr/libexec/mysqld
file /usr/libexec/mysqld
/usr/libexec/mysqld: data
uname -a
Linux ns03.bislinks.com 3.18.25-19.el7.x86_64 #1 SMP Wed Mar 16 19:07:40 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Submitted by andreychek on Sat, 01/28/2017 - 20:52 Comment #6
Hmm, I thought I had responded to this earlier today, but I don't see the comment here. Chances are someone else got your comment... which will probably be a bit odd for them :-)
What I'm seeing above suggests something very odd is going on. It looks like there's a problem with the database binary.
Can you run these commands... the first two may produce gibberish, but it'd be good to double-check:
head /usr/libexec/mysqldtail /usr/libexec/mysqld
dmesg | tai -50
dmesg | tail -50[44344.599384] XFS (dm-1): Unmount and run xfs_repair
[44344.599386] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599388] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599389] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599391] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599392] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599413] XFS (dm-1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x15c22c0
[44344.599415] XFS (dm-1): Unmount and run xfs_repair
[44344.599417] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599419] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599420] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599421] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599423] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599443] XFS (dm-1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x15c22c0
[44344.599445] XFS (dm-1): Unmount and run xfs_repair
[44344.599447] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599449] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599450] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599451] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599453] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599473] XFS (dm-1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x15c22c0
[44344.599475] XFS (dm-1): Unmount and run xfs_repair
[44344.599477] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599479] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599480] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599482] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599483] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599503] XFS (dm-1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x15c22c0
[44344.599505] XFS (dm-1): Unmount and run xfs_repair
[44344.599507] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599509] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599510] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599512] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599513] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599533] XFS (dm-1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x15c22c0
[44344.599535] XFS (dm-1): Unmount and run xfs_repair
[44344.599537] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599539] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599540] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599541] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599543] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599563] XFS (dm-1): Metadata corruption detected at xfs_inode_buf_verify+0x7d/0xe0 [xfs], block 0x15c22c0
[44344.599565] XFS (dm-1): Unmount and run xfs_repair
[44344.599567] XFS (dm-1): First 64 bytes of corrupted metadata buffer:
[44344.599569] ffff88017605e000: 00 04 15 03 46 4d 66 6e fd 32 40 dc 1f b3 0f 76 ....FMfn.2@....v
[44344.599570] ffff88017605e010: 04 f2 3a 01 00 00 00 00 00 00 00 00 00 00 00 00 ..:.............
[44344.599572] ffff88017605e020: 00 00 00 00 00 00 00 00 28 20 e8 a5 8f 8d ff ff ........( ......
[44344.599573] ffff88017605e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[44344.599579] XFS (dm-1): metadata I/O error: block 0x15c22c0 ("xfs_trans_read_buf_map") error 117 numblks 16
[44344.599582] XFS (dm-1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
Submitted by andreychek on Sat, 01/28/2017 - 23:21 Comment #8
Ah, there we go... that explains it, but it's not good news :-)
I think your disk may be going bad.
Or at the very least, you're seeing filesystem corruption.
What you'd likely need to do, is boot a system or repair disk, and run a filesystem repair tool on all of your partitions.
It will make an attempt to fix the disk problems as well as it can -- if it's only filesystem corruption, it may be able to fix it. If the disk really is going bad, there might not be much that can be done, unfortunately.