Automatically Backup MySQL Databases on Windows (WAMP)

The best way to create and automate backups of MySQL databases is to:

1. Use the Windows Task Scheduler to automatically run a backup task every day or week.

2. Have the task run a BATCH file containing the “mysqldump” and “makecab” commands to export and compress the databases.

3. For additional recovery, use MySQL’s Binary Log files to record transactions between backup jobs (to rebuild the database to the last transaction recorded).

This solution will work for everything from Windows 10, down to XP, and Server 2003. No external tools are required.

Backup MySQL Databases with Batch File

Create a mysql-backup.bat file to export all the databases (or only select databases), using a DATE-TIME file-name stamp, and compress the SQL file…

@ECHO OFF

set TIMESTAMP=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%

REM Export all databases into file C:\path\backup\databases.[year][month][day].sql
"C:\path-to\mysql\bin\mysqldump.exe" –-user=username –-password=password --all-databases --result-file="C:\path-to\backup\databases.%TIMESTAMP%.sql"

REM Change working directory to the location of the DB dump file.
C:
CD \path-to\backup\

REM Compress DB dump file into CAB file (use "EXPAND file.cab" to decompress).
MAKECAB "databases.%TIMESTAMP%.sql" "databases.%TIMESTAMP%.sql.cab"

REM Delete uncompressed DB dump file.
DEL /q /f "databases.%TIMESTAMP%.sql"

Why use CAB instead of ZIP?…

MS-CAB files have almost 50% better compression ratios over ZIP (especially for single files), and the MAKECAB/EXPAND commands exist on all Windows versions.

Scheduled MySQL Backup Task

Create a Windows Task to run the above BATCH file every day or every week.

Make sure that:

1. The user has rights to Log on as a batch job.

2. If the drive/path you are exporting to is a mapped drive, to use the UNC path.

3. If the drive/path is a shared folder, the user the task is ran on has the correct permissions.

Use MySQL Binary Log

Verify that my.ini has the Binary Log enabled, set to either a MIXED or ROW mode, and does not expire between backup tasks (make it 2x the frequency of the backup task schedule +1 day).

log-bin=mysql-bin
binlog-format=MIXED
expire_logs_days=15

This way you can restore to the last transaction recorded by replaying the log over the last backup job.

MySQL Won’t Start Because of InnoDB Table Corruption (Repair and Recover Database)

Hardware failures and power outages are the usual cause of databases crashes, corrupted tables, and bad data.

Normally, as MySQL starts it will check itself and attempt to automatically recover from the previous crash (by redoing incomplete transactions using ib_logfiles). But when MySQL encounters an abnormal issue or corrupt InnoDB data, it will refuse to start and will let the user decide what to do next.

InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…

InnoDB: Error:…

Before attempting to repair the existing database: if you have a recent database backup (either a full SQL dump-file or a directory copy), you can rebuild the database to the last transaction recorded by using MySQL’s Binary Log (as long as the backup is not older than the Binary Log’s “expire_logs_days” value).

If the issue is that an InnoDB table has become corrupt, you have to attempt to get as much good data from it, and then manually reload it…

If the corrupt table is not important, you can skip all the steps, and just delete its IBD file (\Database\mysql-data\mysql\table-name.ibd).

1. First you have to make a backup of the entire \Database directory (as data and files are going to be changed and mistakes can happen easily).

“7zip” is a good compression utility, that will properly zip/unzip NTFS Junction Point / folder-links (some apps will not), which this folder contains.

2. Open file Config\Mysql\my.ini (in Notepad), and in section “[mysqld]” add in line:

innodb_force_recovery = 1

Save file. Attempt to start MySQL.

The value above (“1″) specifies which level of regular startup checks MySQL will skip over – so it can start. The value goes from 1 to 6, 4 + is considered dangerous, and the minimum value should be used to get MySQL to start…

This will also put the Database into a predominantly read-only mode (but you can still DROP tables).

3. If MySQL starts, mysqldump (export) the corrupt table, then DROP (remove) it…

Open the command-line (Systems tab, button: Command Line).

Export the specific table into an SQL file –

mysqldump -u root -p database.name table.name > database.table.sql

(* it will ask you for the root password, if it is not set then just leave the “-p” switch out)

The above exported table copy will only contain table rows/entries that MySQL could read (and it is possible that no data could be recovered from the table).

Then access the MySQL shell –

mysql -u root -p

(* it will ask you for the root password, if it is not set then just leave the “-p” switch out)

Drop (remove) the corrupt table which is preventing MySQL from starting –

use database.name;
drop table table.name;

Exit the MySQL shell –

exit;

4. Restart MySQL in normal recovery mode (undo the my.ini edit), and re-import the “recovered” table. This table will only contain non-corrupted rows/entries; it will likely have some data missing…

Remove from file my.ini, section “[mysqld]” –

innodb_force_recovery = 1

Save file. Start MySQL.

Open the command-line.

Import the table –

mysql database.name < database.table.sql

5. If more than 1 table is corrupted, check MySQL log and repeat the above process.

6. If the corrupt table issues are fixed, but MySQL still has some problems starting due to “binary log files errors”…

Move out the “mysql-bin.*” files that are in \Database\mysql-data\ (but not the ib_logfile*, nor ibdata* files).

7. Once MySQL is running, you can also perform a general check and repair of all databases –

mysqlcheck -u root -p --auto-repair --check --optimize --all-databases

MySQL Error “Impossible to write to binary log since BINLOG_FORMAT = STATEMENT”

A user trying to install a webapp reported this MySQL error –

Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.

This error is a result of using a mode for MySQL’s Binary Log Format that is not compatible with TRANSACTION/COMMIT type statements when using InnoDB tables…

The above error is specifically mentioned here

If you are using InnoDB tables and the transaction isolation level is READ COMMITTED or READ UNCOMMITTED, only row-based logging can be used.

MySQL has 3 BINLOG-FORMAT modes:

  • STATEMENT, statement-based logging: every SQL statement is recorded.
  • ROW, row-based logging: every SQL result is recorded.
  • MIXED, mixed logging: if data modification is non-deterministic uses ROW, otherwise STATEMENT.

While the Binary Log is only used for Replication and Point-in-Time Recovery (i.e., nothing to do with installing single-DB webapps), when enabled + using the above transactional statements, instead of SQL statements, results must be recorded.

Edit my.ini (via notepad) –
C:\WampDeveloper\Config\Mysql\my.ini

Modify the Binary Log Format from –

# binary logging format
binlog-format=STATEMENT

To –

binlog-format=MIXED

Save file. Restart MySQL.

If this does not fix the issue, then instead of “MIXED” use value “ROW”.

Installing StartCom’s StartSSL SSL Certificates for Apache (on WAMP)

Before proceeding, if you have downloaded or received a zip file that contains all the various certificate and key files, you should – right click the zip file, select Properties, and click “Unblock” (if that is displayed, otherwise Windows won’t let you use some of the files after unzipping).

Extract/place the provided files into the website’s certs folder –

C:\WampDeveloper\Websites\www.example.com\certs\

If the Private Key (ssl.key) was generated by StartCom for you, you’ll need to remove the password that is set on it (as it’s not needed, and also Apache on Windows cannot use password protected Keys) –

openssl rsa -in ssl.key -out ssl.key

This will ask you for the password, remove it from the key, and save the key back into the same file.

Create the Bundle file that packages all intermediate certificates (as 1_root_bundle.crt, but only if this file has not been already provided)…

copy /B intermediate.crt + root.crt 1_root_bundle.crt

This will create a bundle file named “1_root_bundle.crt” containing the needed CA (Certificate Authority) intermediate certificates which establish the chain between your public certificate up to the root CA certificate.

Then update the website’s SSL VirtualHost file with –

SSLCertificateFile "C:/WampDeveloper/Websites/www.example.com/certs/ssl.crt"
SSLCertificateKeyFile "C:/WampDeveloper/Websites/www.example.com/certs/ssl.key"
SSLCertificateChainFile "C:/WampDeveloper/Websites/www.example.com/certs/1_root_bundle.crt"

Usually nothing more needs to be changed in the VirtualHost (as WAMPs such as WampDeveloper Pro have all other parts of SSL pre-configured).

Save the VirtualHost file and restart Apache.

How to enable cURL and curl_exec in PHP

Client URL library (cURL) is not enabled in this server. cURL is needed to perform URL fetching.

curl_exec() function is disabled in this server. This function must be enabled in php.ini

The PHP extension “cURL” is enabled (usually by default) in php.ini –

extension=php_curl.dll

But it is also possible to disable individual functions, such as curl_exec(), in php.ini (“disable_functions” cannot be set in VirtualHost nor .htaccess files).

For example, php.ini might be using disable_functions to disable curl_exec() from being used –

disable_functions =  ...,curl_exec,...

If this is the case, the website’s PHP and HTTP error log files will contain this warning –

Warning: curl_exec() has been disabled for security reasons

Edit the value of “disable_functions” to remove “curl_exec”, save php.ini, and restart Apache.

Enabling Mass Virtual Hosting

Mass Virtual Hosting allows you to automatically create 100s of websites by simply creating a “full.domain.name” folder. Each of these folders represent the DocumentRoot of a website.

You do not have to create any VirtualHost files, you do not have to restart Apache, you do not have to interact with your server.

This works well for command-line, and scripted/automated, creation and deletion of websites.

Enable Mass Virtual Hosting

To enable Mass Virtual hosting:

1. Load the Dynamic Mass Virtual Hosting configuration into Apache by editing file Config\Apache\httpd.conf and un-commenting line:

Include C:/WampDeveloper/Config/Apache/extra/wampd-vhosts-mass.conf

2. Remove the wildcard “*” ServerAlias entry from the LOCALHOST VirutalHost files:

Config\Apache\extra\wampd-localhost.http.vh.conf
Config\Apache\extra\wampd-localhost.ssl.vh.conf

Otherwise localhost will catch (and respond to) all the unassigned mass domain names.

Create New Website

To create a new website, simply create a new folder:

Websites\dynamic.vhosts\webroot\full.domain.name\

Make sure that the domain name is resolved to an IP address via DNS or the Hosts file.

Limits

1. WampDeveloper Pro does not generally interact with these websites.
a) Websites Tab does not see these websites.
b) LocalDNS is not updated for these websites. You will need to manually update the Windows Hosts file to resolve full.domain.name to 127.0.0.1.

2. All logs go into one master/global log file.
a) Websites logs cannot be segmented into individual files: per domain or sub-domain. Log entries ARE formatted with the correct Virtual Host [sub.domain.tld] fields.

3. The normal fallback/default website (localhost) will no longer get used for non-defined (*) ServerName and ServerAlias.

4. All sites share a common \cgi-bin folder.

5. All sites share the same “outside webroot” folders (ex: \certs, \private, etc).

6. All sites share the same SSL certificate, which will need to be a wildcard cert or have multiple SAN (subjectAltName) entries.

7. Each website has only 1 domain name. For domain aliases, you will need to create a new website folder with an .htaccess file containing a redirect line: “Redirect / http://primary.domain.name/”

Notes

1. PHP and all other configurations are inherited due to use of the standard webroot path/structure:
Websites\dynamic.vhosts\webroot\full.domain.name\site-files

Installing mod_cloudflare Apache Module To Get Real Visitor IP Addresses

If your Apache server is using CloudFlare for security, or to take advantage of their CDN network, you’ll notice that all client requests now come-in from CloudFlare IP addresses – and the real visitor IP address is hidden.

This creates problems as you can no longer do IP based access controls (big problem!), and also you no longer have correct access logs of your visitors. There are also other issues with rewrite rules, various scripts, configurations, and web applications.

mod_cloudflare fixes the issues by restoring the real IP of each connection. It will provide Apache and PHP with the original client IP address.

But there is a twist to all this, as you don’t really want to use a 3rd party Apache module (mod_cloudflare) when there is already a perfectly good native solution… Apache’s mod_remoteip will do the same job as mod_cloudflare, except even better.

mod_remoteip will pull the original client IP address from the CF-Connecting-IP Header provided in each CloudFlare-based request, and use it as-so (after doing some verification).

The full configuration for mod_cloudflare/mod_remoteip is provided by WampDeveloper Pro, and can be loaded into Apache by un-commenting the loading of wampd_cloudflare.conf in httpd.conf.

For everyone else, here is the full CloudFlare configuration for Apache:

# WampDeveloper Pro CloudFlare Integration

# mod_remoteip configuration documentation - http://httpd.apache.org/docs/2.4/mod/mod_remoteip.html
# CloudFlare IP Ranges from -
#   https://www.cloudflare.com/ips
#   https://github.com/cloudflare/mod_cloudflare/blob/master/mod_cloudflare.c
#

#
# To use, just enable your domain name in your CloudFlare account.
# This module and configuration will correctly report the client's true IP / Remote IP (instead of the Proxy IP)
# This fixes issues with web applications, scripts, access and rewrite configurations, and logs
#

<IfModule !mod_remoteip.c>
	LoadModule remoteip_module modules/mod_remoteip.so
</IfModule>

<IfModule mod_remoteip.c>
	# CloudFlare Header
	RemoteIPHeader CF-Connecting-IP
	
	# Trusted Proxy List
	# note - using RemoteIPTrustedProxy instead of RemoteIPInternalProxy
	# note - RemoteIPTrustedProxy does NOT trust Header provided private intranet addresses (local and LAN addresses)
	# note - RemoteIPInternalProxy is a security risk when using an external Proxy
	
	# CloudFlare IPv4 Address Ranges
	RemoteIPTrustedProxy 103.21.244.0/22
	RemoteIPTrustedProxy 103.22.200.0/22
	RemoteIPTrustedProxy 103.31.4.0/22
	RemoteIPTrustedProxy 104.16.0.0/12
	RemoteIPTrustedProxy 108.162.192.0/18
	RemoteIPTrustedProxy 141.101.64.0/18
	RemoteIPTrustedProxy 162.158.0.0/15
	RemoteIPTrustedProxy 172.64.0.0/13
	RemoteIPTrustedProxy 173.245.48.0/20
	RemoteIPTrustedProxy 188.114.96.0/20
	RemoteIPTrustedProxy 190.93.240.0/20
	RemoteIPTrustedProxy 197.234.240.0/22
	RemoteIPTrustedProxy 198.41.128.0/17
	RemoteIPTrustedProxy 199.27.128.0/21
	
	# CloudFlare IPv6 Address Ranges
	RemoteIPTrustedProxy 2400:cb00::/32
	RemoteIPTrustedProxy 2405:8100::/32
	RemoteIPTrustedProxy 2405:b500::/32
	RemoteIPTrustedProxy 2606:4700::/32
	RemoteIPTrustedProxy 2803:f800::/32
</IfModule>

With correcting Apache’s reported client IP, and PHP’s reported $_SERVER['REMOTE_ADDR'], this also secures the process by only trusting the Header-provided IP data from only the CloudFlare servers IP range.