Dealing PHP-FPM Crash & Nginx 502 Bad Gateway Error with Perl FastCGI

Last updated: 5th October, 2015, 11:51 AM

Continuing from my previous post, “Dealing PHP-FPM Crash & Nginx 502 Bad Gateway Error with Shell Scripts“…

501.pl fail to restart PHP-FPM.

So I worked on further trying to improvise this. Then an idea struck me, why not catch the error and try to solve it before giving up? With this, I came up with additional method. By introducing a simple Perl script into the chain, it would be able to catch the 502 Bad Gateway errors beforehand and try to fix it, thus making it transparent to visitors. Although there’ll be a slight delay (PHP5-FPM restarting and re-caching the Opcache/APCU), hopefully I won’t have to lose a hit. Right now I have both Shell and Perl scripts running hand in hand. I wouldn’t say this is hundred percent 502 Bad Gateway errors proof but it does help lessen the issue.

502.pl custom themed.

If somehow the restart process failed (usually multiple site hits per second), visitors will be presented with a modified HTTP 502 Bad Gateway error page. To be able to achieve this setup, firstly we’ll need to have Perl FastCGI enabled in Nginx which you’ll obviously need root access. I’m not covering this here as there’s already an official page on how to have it running, Setting up Perl FastCGI with Nginx. But if you have installed Nginx from my latest WDMyCloud WebHosting Mods at the WD Forum here, [APP] WebHosting for firmware V4+, Perl FastCGI is already enabled.

Disclaimer: As I’m frequently updating the original guides and installers here on TeaNazaR.com, I will not be responsible for any brick issues if you were to follow my obsolete guides copied elsewhere. Thus subscribe to this post to get latest updates. Modifying any part of a device may void its warranty.

After Perl FastCGI has been enabled, continue reading the 502.pl script’s notes on how to get it installed. You could also easily blend in the final error with your site’s theme. Just point the “$template” variable to a plain html file on your server root, then add a string “#502.pl#” anywhere you wish in the body content.

502.pl

#!/usr/bin/perl
use strict;
use warnings;

# 502.pl v1.2 by Nazar78 @ TeaNazaR.com
#######################################
# Simple script to catch Nginx 502 error due to PHP-FPM crash.
# It will try to restart PHP-FPM then redirect the results.
#
# History:
# v1.0 - 20150124 - 1st release.
# v1.1 - 20150301 - Added delay option before checking and restarting PHP-FPM.
# v1.2 - 20150307 - Prevent infinite request loop and few enhancements/fixes.
#
# PS: Feel free to distribute but kindly retain the credits (-:
#######################################

# Begin settings
# PHP-FPM process priority during restart, highest -20 to lowest 19, default is 0.
my $nice = 0;
# Delay in secs before checking and restarting PHP-FPM, 5secs or more.
# Subsequent 502 error within this delay will be redirected after 1sec.
# If after this delay and still unsuccessful, PHP-FPM restart attempts will halt for 
# another period of this value and users will see the 502 error within this period.
my $delay = 5;
# Email address to display on the final 502 error page, leave blank to hide.
my $contact = 'webmaster@teanazar.com';
# Optional absolute path of custom html template containing #502.pl# string.
# Default 502 error template will be presented this file is not readable.
my $template = '/var/www/html/502.htm';
# End settings

=Notes
User running PHP-FPM e.g. 'www-data' needs sudo access to restart PHP-FPM i.e.:
/etc/sudoers:www-data ALL=(ALL) NOPASSWD: ALL

Nginx needs to be setup to run Perl FastCGI. See sample below but I won't explain details here.
For details setting up Perl FastCGI with Nginx: http://nginxlibrary.com/perl-fastcgi/.

If you like to include your site's theme in the final error, set '$template' to the absolute path 
of the html file. Body should contain a #502.pl# string which will be replaced with the results.

Place this script in the server root, chmod 755 and add below to Nginx's vhost then reload i.e.:
server {
	;...other settings.
	;Redirect 502 error to 502.pl script.
	error_page 502 =307 /502.pl;
	;Sample of Perl FastCGI.
	location ~ \.pl$ {
		include fastcgi_params;
		fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
		fastcgi_pass unix:/tmp/fastcgi-wrapperl.sock;
	}
	;...other settings.
}
=cut

my $program = '502.pl v1.2 by Nazar78 @ TeaNazaR.com';

$| = 1;

$ENV{REQUEST_URI} = '' if !$ENV{REQUEST_URI};

$ENV{QUERY_STRING} = '' if !$ENV{QUERY_STRING};

$delay = 5 if !int($delay) || $delay < 5;

if ($ENV{REQUEST_URI} =~ /^\/502\.pl/) {
	if ($ENV{QUERY_STRING} =~ /^502=(.+)&req=(.+)$/) {
		$ENV{QUERY_STRING} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
		$ENV{QUERY_STRING} =~ /^502=(.+)&req=(.+)$/;
		status($1, $2);
	}
	else {
		error('ERROR: Invalid request.');
	}
}
else {
	my $tmp = (stat('502.dat'))[9] || 0;
	if (time - $tmp <= $delay) {
		sleep 1;
		goto end;
	}
	error('ERROR: PHP-FPM could not restart. Previous restart attempt failed.') if (time - $tmp <= ($delay * 2));
	open(TMP, '>502.dat') or error("ERROR: Could not write to '502.dat'. " . $!);
	print TMP time;
	close(TMP);
	my $result = system("sudo nice -n$nice \$(ls /etc/init.d/php*fpm) restart");
	error('ERROR: PHP-FPM could not restart. Service failed.') if $result;
	end:
	print "Status: 307 Temporary Redirect\nLocation: $ENV{REQUEST_URI}\n\n";
}

sub error {
	$ENV{REQUEST_URI} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
	my $result = "Status: 307 Temporary Redirect\nLocation: /502.pl?502=" . shift . "&req=$ENV{REQUEST_URI}\n\n";
	print $result;
	die $result;
}

sub status {
	local $/ = undef;
	my ($result, $req) = @_;
	my $status = '502 Bad Gateway';
	$req =~ s/502\.pl//g;
	$req =~ s/(\?|&)502=.+//g;
	my $href = $req;
	$href = 'here' if $req eq '/';
	my $rand = '?502=' . rand;
	$rand = '&502=' . rand if $req =~ /\?/;
	print "Status: $status\n";
	print "Pragma: no-cache\n";
	print "Cache-Control: no-store, no-cache, must-revalidate\n";
	print "Expires: Sat, 8 Jul 1978 00:00:00 GMT\n";
	print "Content-Type: text/html\n\n";
	my $theme = '';
	if ($template ne '') {
		if (open(THEME, "<$template")) {
			$theme = <THEME>;
			close(THEME);
		}
		else {
			warn "Unable to read theme template '$template'. " . $! . "\n";
			$result = "$result - Also unable to read theme template.";
		}
	}
	if ($contact) {
		$result = "mailto:$contact?subject=$status&body=This is what happened...$result";
		$result =~ s/ /%20/g;
		$result =~ s/"/&#34;/g;
		$result =~ s/@/&#64;/g;
		$contact = " Please help to notify us by <a href=\"$result\">email</a>.";
	}
	$result = <<_RESULT_;
	<center><h1>$status</h1><hr>Something bad happened and your request failed.$contact<p>You could retry this link again later, "<a href="$req$rand">$href</a>".</center>
	<hr><center>$ENV{SERVER_SOFTWARE} <font size="1">Powered with $program</font></center>
_RESULT_
	if ($theme ne '' && $theme =~ /#502\.pl#/) {
		$theme =~ s/#502\.pl#/$result/;
		print $theme;
	}
	else {
		print <<_EOL_;
		<html>
		<head><title>$status</title></head>
		<body bgcolor="white">$result</body>
		</html>
_EOL_
	}
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

#!/usr/bin/perl

use strict;

use warnings;

# 502.pl v1.2 by Nazar78 @ TeaNazaR.com

#######################################

# Simple script to catch Nginx 502 error due to PHP-FPM crash.

# It will try to restart PHP-FPM then redirect the results.

# History:

# v1.0 - 20150124 - 1st release.

# v1.1 - 20150301 - Added delay option before checking and restarting PHP-FPM.

# v1.2 - 20150307 - Prevent infinite request loop and few enhancements/fixes.

# PS: Feel free to distribute but kindly retain the credits (-:

#######################################

# Begin settings

# PHP-FPM process priority during restart, highest -20 to lowest 19, default is 0.

my $nice = 0;

# Delay in secs before checking and restarting PHP-FPM, 5secs or more.

# Subsequent 502 error within this delay will be redirected after 1sec.

# If after this delay and still unsuccessful, PHP-FPM restart attempts will halt for

# another period of this value and users will see the 502 error within this period.

my $delay = 5;

# Email address to display on the final 502 error page, leave blank to hide.

my $contact = 'webmaster@teanazar.com';

# Optional absolute path of custom html template containing #502.pl# string.

# Default 502 error template will be presented this file is not readable.

my $template = '/var/www/html/502.htm';

# End settings

=Notes

User running PHP-FPM e.g. 'www-data' needs sudo access to restart PHP-FPM i.e.:

/etc/sudoers:www-data ALL=(ALL) NOPASSWD: ALL

Nginx needs to be setup to run Perl FastCGI. See sample below but I won't explain details here.

For details setting up Perl FastCGI with Nginx: http://nginxlibrary.com/perl-fastcgi/.

If you like to include your site's theme in the final error, set '$template' to the absolute path

of the html file. Body should contain a #502.pl# string which will be replaced with the results.

Place this script in the server root, chmod 755 and add below to Nginx's vhost then reload i.e.:

server {

;...other settings.

;Redirect 502 error to 502.pl script.

error_page 502 =307 /502.pl;

;Sample of Perl FastCGI.

location ~ \.pl$ {

include fastcgi_params;

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

fastcgi_pass unix:/tmp/fastcgi-wrapperl.sock;

}

;...other settings.

}

=cut

my $program = '502.pl v1.2 by Nazar78 @ TeaNazaR.com';

$| = 1;

$ENV{REQUEST_URI} = '' if !$ENV{REQUEST_URI};

$ENV{QUERY_STRING} = '' if !$ENV{QUERY_STRING};

$delay = 5 if !int($delay) || $delay < 5;

if ($ENV{REQUEST_URI} =~ /^\/502\.pl/) {

if ($ENV{QUERY_STRING} =~ /^502=(.+)&req=(.+)$/) {

$ENV{QUERY_STRING} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;

$ENV{QUERY_STRING} =~ /^502=(.+)&req=(.+)$/;

status($1, $2);

}

else {

error('ERROR: Invalid request.');

}

else {

my $tmp = (stat('502.dat'))[9] || 0;

if (time - $tmp <= $delay) {

sleep 1;

goto end;

}

error('ERROR: PHP-FPM could not restart. Previous restart attempt failed.') if (time - $tmp <= ($delay * 2));

open(TMP, '>502.dat') or error("ERROR: Could not write to '502.dat'. " . $!);

print TMP time;

close(TMP);

my $result = system("sudo nice -n$nice \$(ls /etc/init.d/php*fpm) restart");

error('ERROR: PHP-FPM could not restart. Service failed.') if $result;

end:

print "Status: 307 Temporary Redirect\nLocation: $ENV{REQUEST_URI}\n\n";

}

sub error {

$ENV{REQUEST_URI} =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;

my $result = "Status: 307 Temporary Redirect\nLocation: /502.pl?502=" . shift . "&req=$ENV{REQUEST_URI}\n\n";

print $result;

die $result;

}

sub status {

local $/ = undef;

my ($result, $req) = @_;

my $status = '502 Bad Gateway';

$req =~ s/502\.pl//g;

$req =~ s/(\?|&)502=.+//g;

my $href = $req;

$href = 'here' if $req eq '/';

my $rand = '?502=' . rand;

$rand = '&502=' . rand if $req =~ /\?/;

print "Status: $status\n";

print "Pragma: no-cache\n";

print "Cache-Control: no-store, no-cache, must-revalidate\n";

print "Expires: Sat, 8 Jul 1978 00:00:00 GMT\n";

print "Content-Type: text/html\n\n";

my $theme = '';

if ($template ne '') {

if (open(THEME, "<$template")) {

$theme = <THEME>;

close(THEME);

}

else {

warn "Unable to read theme template '$template'. " . $! . "\n";

$result = "$result - Also unable to read theme template.";

}

if ($contact) {

$result = "mailto:$contact?subject=$status&body=This is what happened...$result";

$result =~ s/ /%20/g;

$result =~ s/"/"/g;

$result =~ s/@/@/g;

$contact = " Please help to notify us by <a href=\"$result\">email</a>.";

}

$result = <<_RESULT_;

<center><h1>$status</h1><hr>Something bad happened and your request failed.$contact<p>You could retry this link again later, "<a href="$req$rand">$href</a>".</center>

<hr><center>$ENV{SERVER_SOFTWARE} <font size="1">Powered with $program</font></center>

_RESULT_

if ($theme ne '' && $theme =~ /#502\.pl#/) {

$theme =~ s/#502\.pl#/$result/;

print $theme;

}

else {

print <<_EOL_;

<html>

<head><title>$status</title></head>

<body bgcolor="white">$result</body>

</html>

_EOL_

}

A point to take note is that having PHP-FPM always restarting on 502 errors does take a hit on your server’s performance, especially if Nginx is mainly serving static or cached files. This script should be used as a last resort.

TeaNazaR.com

Dealing PHP-FPM Crash & Nginx 502 Bad Gateway Error with Perl FastCGI

Like this:

Related:

Leave a Reply Cancel reply

…_\m/

Share this on:

Like this:

Related:

Leave a Reply Cancel reply

…_\m/