Mysql restarting frequently ?

If you are seeing/getting complaints that the mysql is down on the server and when you check this from your end you can see everything is fine. This might be due to frequent automatic restarts.Check the mysql uptime and also let see the what the log says.

root@server [~]# mysql -e “\s”

This will display the mysql uptime like below.You can see the uptime on the bottom.

===

mysql Ver 14.14 Distrib 5.1.73, for unknown-linux-gnu (x86_64) using readline 5.1

Connection id: 3304
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ”
Using delimiter: ;
Server version: 5.1.73-cll MySQL Community Server (GPLv2)
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: latin1
Db characterset: latin1
Client characterset: latin1
Conn. characterset: latin1
UNIX socket: /var/lib/mysql/mysql.sock
Uptime: 4 min 3 sec

====

From this you can conclude the mysql is just started and no one manually done that.No check the logs.

tail -f  /var/lib/mysql/hostname.err (replace hostname with your server hostname) . You may be seeing an error like below. Usually this type of reboot is caused when there is an incorrect entry on my.cnf or due to innodb crash. Below is the error.

=============<<<<<<<<<<<<<>>>>>>>>>>>>>================

This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=157286400
read_buffer_size=1048576
max_used_connections=10
max_threads=500
thread_count=5
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1182947 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.

Thread pointer: 0x1cfa8f20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 2b7da1cf4f08 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x86e6be]
/usr/sbin/mysqld(handle_fatal_signal+0x380)[0x6c0630]
/lib64/libpthread.so.0[0x2b7d8e91aca0]
/lib64/libc.so.6(gsignal+0x35)[0x2b7d8f431265]
/lib64/libc.so.6(abort+0x110)[0x2b7d8f432d10]
/usr/sbin/mysqld(fil_io+0x3be)[0x78876e]
/usr/sbin/mysqld[0x772d04]
/usr/sbin/mysqld(buf_read_page+0x200)[0x773200]
/usr/sbin/mysqld(buf_page_get_gen+0x144)[0x76c974]
/usr/sbin/mysqld(btr_cur_open_at_rnd_pos+0x227)[0x75fbb7]
/usr/sbin/mysqld(btr_estimate_number_of_different_key_vals+0x1ae)[0x75ffce]
/usr/sbin/mysqld(dict_update_statistics_low+0x75)[0x775765]
/usr/sbin/mysqld(dict_table_get+0x156)[0x77a436]
/usr/sbin/mysqld(_ZN11ha_innobase4openEPKcij+0x133)[0x755713]
/usr/sbin/mysqld(_ZN7handler7ha_openEP8st_tablePKcii+0x3f)[0x6b37ef]
/usr/sbin/mysqld(_Z21open_table_from_shareP3THDP14st_table_sharePKcjjjP8st_tableb+0x54b)[0x62cafb]
/usr/sbin/mysqld[0x6205cb]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootPbj+0x683)[0x624db3]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjj+0x372)[0x625712]
/usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x1e)[0x625a4e]
/usr/sbin/mysqld[0x6df12b]
/usr/sbin/mysqld(_Z14get_all_tablesP3THDP10TABLE_LISTP4Item+0x4fb)[0x6df68b]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x1d0)[0x6d7650]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x439)[0x64d7a9]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1b9)[0x64f799]
/usr/sbin/mysqld(_Z13handle_selectP3THDP6st_lexP13select_resultm+0x169)[0x6500a9]
/usr/sbin/mysqld[0x5de5f4]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x545)[0x5e1c05]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x418)[0x5e7be8]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x732)[0x5e8322]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x120)[0x5e9300]
/usr/sbin/mysqld(handle_one_connection+0x7ed)[0x5dae9d]
/lib64/libpthread.so.0[0x2b7d8e91283d]
/lib64/libc.so.6(clone+0x6d)[0x2b7d8f4d627d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (1cfb4e90): SELECT data_free FROM tables WHERE ENGINE = ‘InnoDB’ LIMIT 1
Connection ID (thread ID): 332
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140920 07:30:09 mysqld_safe Number of processes running now: 0
140920 07:30:09 mysqld_safe mysqld restarted
140920 7:30:09 [Warning] ‘record_buffer’ is deprecated and will be removed in a future release. Please use ‘read_buffer_size’ instead.
140920 7:30:09 [Note] Plugin ‘FEDERATED’ is disabled.
140920 7:30:09 InnoDB: Initializing buffer pool, size = 8.0M
140920 7:30:09 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
140920 7:30:09 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…

================>>>>>>>>>>>><<<<<<<<<<<====================

To resolve this we need to add the below parameters on the /etc/my.cnf file .

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0
innodb_thread_concurrency=0

and then restart the mysql service. Please monitor the log again to see if there is any further issue. Hope this help you 🙂

 

Check whether the server support Innodb.

-Check For InnoDB Support

Using phpMyAdmin:

1. Open up phpMyAdmin
2. Click on the Storage Engines link on the main screen
3. If InnoDB is enabled, it will show up in blue color

You can also check if InnoDB is enabled from shell. Connect to MySQL and run the following query:

mysql> SHOW ENGINES;
+————+———-+—————————————————————-+
| Engine     | Support  | Comment                                                        |
+————+———-+—————————————————————-+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance         |
| MEMORY     | YES      | Hash based, stored in memory, useful for temporary tables      |
| InnoDB     | YES      | Supports transactions, row-level locking, and foreign keys     |
| BerkeleyDB | NO       | Supports transactions and page-level locking                   |
| BLACKHOLE  | YES      | /dev/null storage engine (anything you write to it disappears) |
| EXAMPLE    | YES      | Example storage engine                                         |
| ARCHIVE    | YES      | Archive storage engine                                         |
| CSV        | YES      | CSV storage engine                                             |
| ndbcluster | DISABLED | Clustered, fault-tolerant, memory-based tables                 |
| FEDERATED  | YES      | Federated MySQL storage engine                                 |
| MRG_MYISAM | YES      | Collection of identical MyISAM tables                          |
| ISAM       | NO       | Obsolete storage engine                                        |
+————+———-+—————————————————————-+
12 rows in set (0.00 sec)

If InnoDB is enabled, it will have a YES next to it.

InnoDB Crash Recovery Guide

Make sure you have free disk space in /home and /var/lib/mysql or where ever your mysql databases are stored. This guide assumes they are in /var/lib/mysql.

FOR CPANEL

Step 1: Disable tailwatchd (chkservd on 11.23) to prevent mysql from being restarted. Stop any other processes that may access mysql including munin, backups, and the webserver if possible.

killall -TERM tailwatchd

Step 2: Add

innodb_force_recovery = 3
to /etc/my.cnf

Step 3: Restart mysql

/etc/init.d/mysqld restart

Step 4: Verify mysql is up

Step 5: Put this script in ~/recover.pl and run it

======================
#!/usr/bin/perl

BEGIN { unshift @INC, ‘/usr/local/cpanel’; }

use Cpanel::MysqlUtils         ();

mkdir(‘/home/innodb_dumps’,0700);

chdir(‘/home/innodb_dumps’) || die “Could not create /home/innodb_dumps”;

my $pwd = `pwd`;
chomp($pwd);
my $hasinnodb = 0;
my $dbdir     = Cpanel::MysqlUtils::getmysqldir();
my $dbcheckok = -d $dbdir . ‘/mysql’ ? 1 : 0;
if ($dbcheckok) {
opendir( my $sql_dh, $dbdir );
while ( my $db = readdir($sql_dh) ) {
next if ( $db =~ m/^\.+$/ );
next if $db eq ‘mysql’;    # mysql db should never have InnoDB on
next if $db eq ‘horde’;    # Horde session table is the only one that uses InnoDB, so this is not a problem
next if ( !-d $dbdir . ‘/’ . $db );
my $ms = sqlcmd(“show table status from `$db`;”);
if ( !$ms ) {
$dbcheckok = 0;
last;
}
elsif ( $ms =~ m/\s+InnoDB\s+/m ) {
print “Saving to $pwd/$db.sql…”;
system “mysqldump -c $db > $db.sql”;
print “Done\n”;
}
}
closedir($sql_dh);
}

sub sqlcmd {
my ($cmd) = @_;
my $result;

my $mysql = Cpanel::MysqlUtils::find_mysql();
my $pid = IPC::Open3::open3( \*WTRFH, \*RDRFH, “>&STDERR”, $mysql, ‘-N’ );
print WTRFH “show status like ‘uptime’; $cmd\n”;    #make sure we already get something back so we know mysql is up
close(WTRFH);
while () {
$result .= $_;
}
close(RDRFH);
waitpid( $pid, 0 );
return $result;
}
======================================
If the script fails, remove your backup attempt in /home/innodb_dumps, increase the innodb_recovery_level in /etc/my.cnf, restart MySQL, and rerun the script. Repeat until you complete a backup without errors

Step 6: Stop MySQL

Step 7: Do this:

mkdir /var/lib/mysql/INNODB_BACKUPS
mv /var/lib/mysql/ib* /var/lib/mysql/INNODB_BACKUPS

Step 8: For each database that was dumped to /home/innodb_dumps move (do not copy, completely move, or the database may not be properly recreated from the backups you made) the /var/lib/mysql/DBNAME folder to the /var/lib/mysql/INNODB_BACKUPS directory

Step 9: Create folders for each database that was moved in /var/lib/mysql and chown them to mysql:mysql

Step 10: Remove the following from my.cnf and start mysql:

innodb_force_recovery = 3

Step 11: Start mysql and restore all databases in /home/innodb_dumps to their respective database

Step 12: Check the mysql server logs to ensure there were no errors

Step 13: Restart mysql and party.

FOR ALL OTHER LINUXES

Step 1: Add

innodb_force_recovery = 3
to /etc/my.cnf

Step 2: Restart mysql

/etc/init.d/mysqld restart

Step 3: Verify mysql is up

Step 4: Dump all innodb databases to a directory i.e /root or /root/recovery

Step 5: Stop MySQL

Step 6: Do this:

mkdir /var/lib/mysql/INNODB_BACKUPS
mv /var/lib/mysql/ib* /var/lib/mysql/INNODB_BACKUPS

Step 7: For each database that was dumped move the /var/lib/mysql/DBNAME folder to the /var/lib/mysql/INNODB_BACKUPS directory

Step 8: Create folders for each database that was moved in /var/lib/mysql and chown them to mysql:mysql

Step 9: Remove the following from my.cnf and start mysql:

innodb_force_recovery = 3

Step 10: Start mysql and restore all databases in /home/innodb_dumps to their respective database

Step 11: Check the mysql server logs to ensure there were no errors

Step 12: Restart mysql and party