| Posted by jcisio, 04-01-2010, 05:43 AM | | Hello,
I have a dedicated X3220 2GB of RAM. The situation hasn't been better after upgrading to 4GB.
The server gets high load at random time, multiple times per day. In my PHP script, I set to kill it if the load is too high, so the problem doesn't affect as much as it did before (completed down, slowly restart to fill the cache). But it does!
Here are some stats with munin. No certain thing with Apache, MySQL, memory... but only with load (and netstats with many fail at that time). I can't understand why the load is high.
Thanks.
Attached Thumbnails
|
| Posted by jcisio, 04-01-2010, 05:45 AM | | New post because of the 5-attachment limit.
Thanks.
Attached Thumbnails
|
| Posted by activelobby4u, 04-02-2010, 02:36 AM | | What does your top command say when there is the peak usage?
|
| Posted by UNIXy, 04-02-2010, 03:09 AM | | By the time you check for the load in your PHP, a lot of resources have already been consumed. You might want to to hook the load check before apache's spawning mechanism. What does mysql -u root -e "show full processlist" produce during load peak? I have a feeling it's the bottleneck. Have you thought about using an accelerator in front of Apache like nginx or varnish?
Regards
Joe / UNIXY
|
| Posted by jcisio, 04-02-2010, 03:53 AM | | No more than 30 PHP processes with less than 30m memory each and a few dozens of httpd (a few MBs each, I'm using suphp). I have more than 2 GBs for buffer/cache, so I don't think it's a memory problem.
Thanks for your input. I'm using mytop to show the list, but nothing there! (except for the SHOW PROCESS query). I use memcached to offload the MySQL. Here are what I have with MySQL:
The "highest connection usage" is normally less than 30 during peak hour, so I set the value to be 100. Here, it's 101. So it seems that you are correct. If MySQL is the bottleneck, is it the disk access problem (it's SATA II, I upgraded memory instead of HDD because it's much cheaper)?
I use a fine tuned Apache. I don't have an accelerator in front of Apache, but I have static file cache that bypasses PHP (Drupal 6 with boost, FYI). So I still don't understand why it can take the server down easily at any time, without too many connections. When I list the connected IP by
No IP has more than 20.
|
| Posted by jcisio, 04-02-2010, 05:54 AM | | A top on a relative high load.
And here is the `ps aux` right after that:
|
| Posted by activelobby4u, 04-02-2010, 06:00 AM | | It is the apache process
try these
1. enable suphp
2. Verify the results of netstat to confirm DOS attack
3. mysqladmin proc stat - to find the db user causing trouble
|
| Posted by jcisio, 04-02-2010, 06:09 AM | | 1. It's enabled already! There are a few php5 processes with 30-40MB memory each
2. Max connections per IP now is 16 (sometimes it was 20-25, but not more than that). Maybe a DDoS. But when the load is too high, my PHP script is die automatically, and the effect ends soon (return to normal). A very small/short attack?
3. Only one db user. Only one website in this dedicated box.
Apache just has small memory usage. I don't use lighttpd for static content (even I plan to) because I'm not sure that's the problem. I want to use port 80, so I need a second IP. But in Vietnam IP address is not cheap
|
| Posted by activelobby4u, 04-02-2010, 06:21 AM | | You apache still runs as nobody.
Its not the memory that increases the load, but the cpu. I suggest you tweak your apache for performance.
|
| Posted by jcisio, 04-02-2010, 06:35 AM | | I can modify httpd.conf so that Apache run as another user. But I think it doesn't help. PHP run already as the right account (ttcn). suphp just changes uid of the process executing the PHP interpreter, doesn't it?
I can't imagine Apache at less than 100 req/s can cause high CPU load. I'll investigate on this.
|
| Posted by jcisio, 04-02-2010, 06:42 AM | | The actual load is 7, and I have this:
So I don't think Apache is the cause. It takes less than 1/10000 CPU load.
|
| Posted by UNIXy, 04-04-2010, 03:23 AM | | It's not Apache but it's the PHP processes. You'll need to run eAccelerator or any of the opcode cachers to lower the load and switch to fastcgi w/suexec. suphp has to go through all the stages of script transformation to opcode every time it executes a script. There's a considerable performance penalty as a result.
Regards
Joe / UNIXY
|
| Posted by jcisio, 04-04-2010, 04:08 AM | | Yes, eAccelerator was enabled, too. The whole stack:
- PHP, Suhosin, eAccelerator
- Memcached
- Apache, MPM Prefork (I know Worker is better in this case, but I will leave the default cPanel choice for a while), suphp, FastCGI
- MySQL with cache enabled
So, not the op code cache problem. However, I found that there were many 404 error (each needs an PHP call, even it returns quickly), I fixed most of them. But the problem still happens.
I've tested your recommendation in #2 with nginx in front of Apache and the performance increase was a hit. I'll switch to it, when everything ready, for a nice gain.
Another problem, maybe related, is that the ad code has recently loaded very slowly and kept the whole page stuck. I don't know in that case, if the PHP memory is released before the page is completed transfered to the client or not. If not, problem might be there.
|
| Posted by nwmcsween, 04-04-2010, 07:36 PM | | QUOTE=jcisio;6729332]A top on a relative high load.
...
How many disks does this server have? It looks like IO wait is the issue meaning something is hammering your disks and my best guess is mysql. Run something like iotop or atop and check for disk utilization. To fix it I would look into hiring someone or taking a weekend to work on it as it's most likely more than one thing that needs tuning.
|
| Posted by jcisio, 04-04-2010, 08:09 PM | | Nice catch.
On this top it had 16%us, sometimes very high.
There is only one 250GB SATA2. Is there a big performance gain in this case if I use SAS? There were no problem on a VPS 768 MB RAM (but 15K rpm, RAID 1+0), but I upgraded to dedicated 4 GB, 7200 rpm. I think in a VPS, disk access is much slower?
To tune the disk usage, except for hardware upgrade, is there anything to do? I don't have much chance to modify MySQL queries/schemas. The cache is large enough.
|
| Posted by RDOSTI, 04-05-2010, 04:07 PM | | Sounds like you need server optimization.
Suggest seeing which type of websites / scripts your users are accessing.
My guess their accessing forums or database/php run portals or sites.
Good option is to optimize php (http) and mysql.
Its also worth looking into security (including mod_security) rules and a good firewall.
Software/scripts being run should be updated to latest versions as well.
There are tons of things beyond the server itself that can be looked upon.
Put a step by step plan together and proceed ahead. Checklist the required to be seen areas/items.
|
| Posted by gate2vn, 04-06-2010, 09:43 AM | | No, a VPS on raid10 15K disks array will give you much better disk speed than your single SATA drive.
|
| Posted by jcisio, 04-06-2010, 09:53 AM | | Even when there are 20 VPSs on a single box (the VPS costed $50, so I imagine...)? Not sure with MySQL, but a simple 'mv' of 10 GB log data from one partition to another on my VPS was much slower. However that's just what I feel.
|
|
 Add to Favourites
 Print this Article
|
