| Posted by GumSlone, 01-12-2010, 06:11 AM | | Hello,
is it possible to restart httpd automatically when the server load is over 100?
i have a cpanel with centos server.
|
| Posted by laswatech, 01-12-2010, 07:31 AM | | You can write a shell script and put it in cron. The logic is that it will search for the load by top command and then restart httpd by using /etc/init.d/httpd restart command if it is over 100. The cronjob can be run every 5 minutes depending on your requirement.
|
| Posted by RHS-Chris, 01-12-2010, 08:54 AM | | I think the bigger question here is why is the load so high?
|
| Posted by laswatech, 01-12-2010, 08:56 AM | | If the load is due to http, then there are every possibility that you might be under ddos attack. You got to find the ips and block them...On a nutshell you need to find out why the load is high rather than restarting httpd.
|
| Posted by GumSlone, 01-12-2010, 09:25 AM | | after the httpd restart the server load goes back to minimum avarage 0,5 its value remains for several hours at minimum but then happens something what causes a high server load of 250 so i needed to restart httpd to get it back to the minimum, i have installed nginx reverse proxy to apache what helped alot to reduce the load, without nginx the server load was all the time high load.
|
| Posted by mattle, 01-12-2010, 10:03 AM | | I'd prefer 'w -u' over 'top -s'...
But really, your load should never be at 100! Sounds like something's going wrong. You may want to look at your apache configuration, or think about new hardware. I'd start there instead of starting with a hack.
|
| Posted by hosting_we3cares, 01-12-2010, 10:55 AM | | When the load is high run this command and let us know the output
===========
netstat -n | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
==========
|
| Posted by GumSlone, 01-14-2010, 09:27 AM | | here is a output:
the server monitoring staff restarts httpd about 15 minutes after they detect that apache is crashed, and 15 minutes is too long for me, i thought if its possible to create a cron which will check the server load every minute and if its higher than 100 it should restart httpd. I have found this shell script on web: http://bash.cyberciti.biz/web-server...-shell-script/ but it doesnt check for server load value.
Last edited by GumSlone; 01-14-2010 at 09:33 AM.
|
| Posted by net, 01-14-2010, 09:37 AM | | Man, that is not the solution.
You need to find out why you are reaching that load.
|
| Posted by GumSlone, 01-14-2010, 09:51 AM | | i have 3 managed servers (2 at liquidweb and 1 at singlehop) and in 3 days period of time there is always 1 or more apache crash because of high server load on every server.
the server which output ive posted above has Load Averages: 0.55 9.81 83.74 after httpd restart
Last edited by GumSlone; 01-14-2010 at 09:54 AM.
|
| Posted by Scott.Mc, 01-14-2010, 01:10 PM | | If it's consistently occurring then your management are not doing a very good job to identify the source, restarting/temporarily mitigating is NOT the answer.
|
| Posted by mattle, 01-14-2010, 01:24 PM | | I'm a little concerned about some of those IPs with double-digit simultaneous connections. You've always got some bastard blasting your server with requests for common admin tool URLs. It could either be that, or your apache children are not dying properly and you have hung processes.
Try this command to look for people abusing the system:
grep " 404 " /path/to/access_log | awk '{ print $1 }' | sort | uniq -c | sort -n
You can either add them to iptables directly, or do something like this:
Figure out your average lines/min in your access_logMultiply that by 5 or 10Write a script that uses this command:
tail -N /path/to/access_log | grep " 404 " | awk ... (same as above)
Where "N" is the number of lines you calculatedParse the output from that command and automatically add an iptables rule for anyone who has an obnoxious amount of 404 errorsSchedule the job to run every 5/10 minutes
I also like to write a process that will block a whole C-class block of IPs if I get more then 3 IPs in the same range that are behaving badly.
(A job like this analyzing your maillog can be very useful as well)
If you want to get really sophisticated, you could also look for superhuman accesses as well (e.g., 100 requests, valid or not, from the same IP in one minute). That will help prevent simple DOS (not DDOS) attacks.
Finally, you'll want to look at those apache children. Each server instance is a little different, but I'd google "optimizing apache" and see how your config settings compare to the recommended ones. There's a lot to look at, but if you have any questions about your settings, post back here and we'll take a look.
Hope that helps,
Matt
|
| Posted by RHS-Chris, 01-14-2010, 03:07 PM | | There are some bugs in cPanel right now, and there is a large discussion going on in the cPanel forums. This is happening to a lot of people right now, and it seems to effect really busy servers. I have downgraded eAccelerator and have change my statistic processing to off peak times in order to try to keep the server load from spiking. Take a look at these threads:
http://forums.cpanel.net/f5/11-25-cu...ur-134281.html
http://forums.cpanel.net/f5/httpd-defunct-142509.html
|
| Posted by Website themes, 01-15-2010, 12:51 AM | | No there is nothing wrong with those IPs with double digit connections. Its all quite normal. The reasons are:
* Proxies used by ISPs and large organisations
* Keep alive off in apache means multiple simultaneous connections for requests to multiple webpage elements
Even 3 digit (100-200) connections from the same IP are OK if your site is popular among office workers/schools.
To the OP your problem is being caused by a script that is badly coded. What you need to do is turn on mod_status and log into your server and view the output of that /server-status during high load times. That will tell you which script is the cause of the high load.
Last edited by Website themes; 01-15-2010 at 12:57 AM.
Reason: formatting,keep alive off
|
| Posted by mattle, 01-15-2010, 11:31 AM | | That may be a premature presumption, not knowing the expected traffic/user base of the site...I still contend that even if the IPs you're seeing is legitimate traffic, you aren't doing any harm by adding firewall rules for IPs generating lots of 404 statuses.
I do agree, however, that a hanging script is a more likely culprit.
|
| Posted by Website themes, 01-15-2010, 09:39 PM | | No not at all premature. Lots and lots of organisations and ISPs use proxies. So its not uncommon at all. Also apache with keepalive off does not allow browsers to reuse the same connection for multiple web page elements. So browsers have to make multiple connections to the same server. Again very common.
As far as blocking IPs based on 404 errors you need to be very careful. Often CMS are quick to generate 404 errors. Also sometimes if there is an inadvertent change in the permalink of a popular page you will see lots of 404 errors as visitors from external sites come to your site and get that error. Also if your embedding a script or image in every page of your site and that script/image file is missing it will generate 404 errors for every pageview of your site!
Last edited by Website themes; 01-15-2010 at 09:41 PM.
Reason: pageview not request
|
| Posted by xserverx, 01-15-2010, 10:34 PM | | I have same problem in 3 servers
its cpanel 11-25 bug
I advice you to downgrade cpanel to 11.24 stable
|
| Posted by GumSlone, 02-14-2010, 12:07 AM | | Thanks everyone who tried to help me, i think i have found what i was looking for:
http://zedomax.com/blog/2009/08/17/w...ervice-script/
|
| Posted by DashV, 02-15-2010, 12:53 AM | | I suggest you use apachetop to get a realtime view of your incoming requests.
If you find out you're serving too many static files (Images, CSS, JavaScript) then Varnish and/or NGINX in front of Apache would do.
I'm ready to help install and configure these for you. PM me for details.
|
| Posted by Dr:linux, 02-15-2010, 10:24 PM | | Please check this script that restarts Apache when the load is over 100.Cron this script to run every 10 min or customize it.
Feel free to drop in any queries further.Have a nice day.
|
| Posted by bhavicp, 02-15-2010, 11:35 PM | | Good short-term solution. If your servers are at LW/SH open a ticket (if you have managed services) and ask them why this could be happening. You really need to get to the root of the problem!
|
|
 Add to Favourites
 Print this Article
|
