Nginx and PHP-FPM suddenly permission denied
Seemingly, without cause, one of our web applications began exhibiting strange behavior. It had run without needing much intervention for years. But suddenly files were not fully downloading, they were only getting a small partial download when the file download stalled.
So I went into debug mode. First checking the obvious, how was the load on the machine, was it failing on downloads because of a memory or cpu issue? I didn't think so, the load at that time was below average. I restarted the webserver just in case. Problem still present.
Next I took a look at nginx-error.log and there was an interesting string showing up in there [some details removed]:
[datetime] [crit] 63972#0: *149793 open() "/var/tmp/nginx/fastcgi_temp/0/59/0000005590" failed (13: Permission denied) while reading upstream, client: [client ip], server: [domain], request: "GET [url]. HTTP/1.1", upstream: "fastcgi://unix:/tmp/php-fpm.sock:", host: "www.rfpdb.com", referrer: "[referrer]"
Permission denied, always the devil on a unix box, nine out of ten times some permission thing is the cause of things-not-working-right. This is all good to know, the files (for some reason) no longer have the correct permissions to be read.
] ls -l files/ # ... -rwxrwx--- 1 www developers 25901 Jun 30 2010 10252 # ...
All files are still owned by www, which is to be expected since the webserver is running as www. The webserver should be able to read these. The webserver is still running as www right?
]$ ps -aux | grep nginx root 5581 0.0 0.4 15528 5100 ?? Is 8:28PM 0:00.03 nginx: master process /usr/local/sbin/nginx nobody 5661 0.0 0.5 15528 7100 ?? S 8:40PM 8:03.49 nginx: worker process (nginx) tleen 18810 0.0 0.1 9124 1168 0 S+ 6:13PM 0:00.01 grep nginx
My Nginx worker was running a nobody not www, which explains why that file cannot be read. That's not right. How did that happen?
Time to head over to the Nginx config and see what is happening. One thing I always do is Git All The Things!, including this Nginx configuration. I had just done an update to the Nginx configuration while updating a SSL certificate. I didn't really see why that would result in a permissions change for the webserver, but it wouldn't take long just to check.
[/usr/local/etc/nginx]# git diff 18e5:nginx.conf 3e74:nginx.conf diff --git a/18e5:nginx.conf b/3e74:nginx.conf index 1d60814..62515c7 100755 --- a/18e5:nginx.conf +++ b/3e74:nginx.conf @@ -1,4 +1,4 @@ -#user nobody; +user nobody; worker_processes 1; # more SSL stuff
Looks like while I was updating the config file I somehow unintentionally removed the hash from the line setting the nginx user to "nobody", so when I restarted with the new SSL setup Nginx started running as the new user.
One small character, big consequences.