| Posted by mileka, 09-12-2011, 12:35 PM | | So quick and dirty explanation, i'm running a shared cPanel server and every day while daily backups are running the server seems to have many "timeouts" and disconnects. It will take about 30-60 seconds longer to load any type of page, compared to if i kill the cpbackup process it loads fine.
My iousage is averaging about 16.5% and load is hardly anything. With backup running it goes up to about 2gb of ram (12gb on server).
I'm not running any type of RAID array on the server, just two 2 TB 7.2k drives, one for backups.
Now i've read all around on google and here and been using iotop to try and "track down" what is causing the problem.
So my original question would be, does anybody have any suggestions on how to track down IO usage and determine what may be causing the problem? I mean even to the point of logging daily to get some type of report?
I know you can use IOTOP to show specific users, etc, but it's hard for me to really put my "finger" on what is causing the problem.
I have found many users that have over 500k inodes and i'm working on writing a shell script to scan the entire home directory and print out inode usage so i can remove those specific users from backups, but does anybody have any other suggestions on what i could do to help this?
The server is a very powerful 8 core server so load is not the issue at all, will i gain much out of setting up a RAID array? I'm using RAID1+0 on my VPS server and it works like a charm, my IO usage is less than 1%, but i want to try and stay away from installing a RAID card and more drives if i don't have to.
Any suggestions on how to optimize the server to reduce IO usage? Maybe it's not even IO usage that's causing the problem when the backups run but i've exhausted everything i can think of...
Any help is greatly appreciated, thanks guys!
|
| Posted by Martin-D, 09-12-2011, 12:42 PM | | Are you using the cPanel backup option?
The zip/tar process will be taking a lot of time and can cause bottlenecks. With regards to the inodes, write a quick script to check for 'new' emails in default mail accounts. Many people forget about these and they very quickly fill up.
|
| Posted by mileka, 09-12-2011, 01:07 PM | | Yes it is setup as cPanel internal backup system, using this:
Incremental Backup
Compression: Disabled
Backup Config Files: Enabled
Backup MySQL: Per account and entire directory
Use Hard Links: Enabled
Optimized Account Backup Tool: Enabled
And yes i'm going to be looking for default emails and catchall accounts, but the script im writing will just output each directory under that user so i can tell if it's in mail directory from there, maybe soon i will update it to include auto removal from backups, etc...but that's a long term plan, for now i just want to scan all the directories so i have an output file that shows them all instead of having to do it one by one.
So does the process used by cpanel backup bottleneck on the IO? Could that be what's causing the IO to be up so high? I haven't check it in a bit but about a month or two ago this server was running with only around 6% IO now it's at 16%...hmmm, any ideas?
|
| Posted by GOT, 09-12-2011, 01:49 PM | | With incremental selected, this should not cause a huge problem unless you have a ton of changes being made every day. This is typically the solution to when backups are causing high IO.
If it were me, I would try turning the two experimental options off which you have on.
|
| Posted by instantDS, 09-12-2011, 01:52 PM | | Think about disabling cpanel backups, automate mysql dumps and rsync the data of the users. This will not affect your I/O and does not produce a high load. You can also build in a cron that checks if load is > 5 THEN pause rscync/mysqldump
|
| Posted by barbus, 09-12-2011, 03:04 PM | | If you have MyISAM tables they will be locked.
And yes, you might need to use mysqldump for backups instead of cPanel built-in tools.
|
| Posted by GOT, 09-12-2011, 03:25 PM | | cPanel's backup routines does create dump files, so that's not a problem.
There's no reason that cpanel's backup systems can't be used, its just a matter of tracking down the source of the problem. Watching top while the backup is running should tell you a lot.
|
| Posted by EclipzeComputing, 09-12-2011, 03:49 PM | | Check your CPU %wait while the backup is running using top.
With only one drive for 2 TB of data there is a strong chance that your drive is getting thrashed and constantly seeking back and forth, causing a bottleneck for the rest of your system.
|
|
 Add to Favourites
 Print this Article
|
