| Posted by lovelynetworks, 07-07-2008, 04:53 PM | | I am facing some strange problems with one of my cPanel/WHM server.
It's a dedicated server with Quad Core processor and 4gb ram.
First i thought it was a ddos attack but now i think it's not.
I am keep getting ftpd failed emails every 5-10 mins.
server is extremely slow and load is over 10 for last 2 days.
it shows that httpd, mysql, ftp is up but none of our site's are loading.
1 thing i noticed when i logged into WHM is that it says WHM VPS Optimized.
Attachment 13075
I tried updating cPanel but same problems.
SSH Top:
I tried everything i knew and frustrated now.
Any Help is appreciated.
Thanks
|
| Posted by TheITAdvisory, 07-07-2008, 05:20 PM | | Hello, You have a lot of zombie processes running, this would explain the 37.4%wa in your results from top. This means there are waiting tasks/data/processes or potentially failing hardware. To get a full snap shot list of processes, type
ps faux > results.txt This will dump the results of ps faux to a text file called results.txt, this makes it easier for you to go through the results, and find the zombie proc's. You can then use your favorite text editor (VIM is mine) for looking at the new text file.
You will find the Zombie processes by looking for something like the following,
16 root 10 -5 0 0 0 {Z} 0 0.0 0:00.00 events/2
I have Bracketed {} the Part you need to look for, The Z means Zombie, S would mean Sleeping, R would mean running, and D means waiting for something.
You can then kill -9 PID HERE the PID being that of the zombie process you found in the ps aux. It probably also wouldn't hurt to tail some of your log files in /var/log, like dmesg, messages, etc etc.
Good Luck!
|
| Posted by lovelynetworks, 07-07-2008, 06:21 PM | | I ran ps faux command and then looked through results.txt file and killed all zombie proccess.
server load is still same and wa is 44%
|
| Posted by The3bl, 07-07-2008, 06:29 PM | | Run top press the 1 button so we can see the cpus.
Then press shift key P so it is ordered by cpu usage and post the results again.
The run uname -a and post the output.
|
| Posted by lovelynetworks, 07-07-2008, 06:52 PM | |
|
| Posted by The3bl, 07-07-2008, 07:10 PM | | You are running a xen kernel. Some data centers seem to be doing this lately so they can install the VPS cpanel on a sever instead of the more expensive full cpanel/whm. Not exactly legal from the license point of view.
You have something stuck that is waiting on your disk drives. But that normally happens when it has run out of mem and using lots of swap. Does not seem to be the case on your system unless they have set up the xen wrong and not allocated you full use the system memory.
What does a netstat -n show?
Have you tried to reboot the system?
|
| Posted by lovelynetworks, 07-07-2008, 07:27 PM | | we have this server for almost a year now and we had full version of WHM/cPanel. Just for last 2 days its showing that WHM VPS Optimized in WHM. We haven't updated the server or changed whm/cpanel or anything.
Maybe our server got hacked or something.
right now i can't ssh to server, its timing out.
I have tried restarting server numerous times.
|
| Posted by The3bl, 07-07-2008, 07:35 PM | | 2.6.18-8.1.10.el5xen is a XEN vps kernel not a normal kernel.
I doubt someone hacked your server and installed a xen kernel that does not make sense. Though anything is possible I guess.
Your load is too high so the system is probably losing services like ssh.
Try a reboot.
|
| Posted by lovelynetworks, 07-07-2008, 07:41 PM | | rebooting the server.
how can i install the normal kernel?
why is the server load so high?
|
| Posted by Lsupport, 07-07-2008, 07:48 PM | | Your system looks to be clean from memory/swap issue and cpu is relaxing.But from 100.0%wa, there seems some activities happening in the hard disk. This may happen if one of the hard disk is bad and RAID is trying to rebuild it
|
| Posted by The3bl, 07-07-2008, 07:57 PM | | pico /boot/grub/grub.conf
You will see a line near the top that says default =0 or some number.
below that you will should see some kernel images example:
title CentOS (2.4.21-50.EL)
root (hd0,0)
kernel /vmlinuz-2.4.21-50.EL ro root=LABEL=/
initrd /initrd-2.4.21-50.EL.img
title CentOS (2.4.21-50.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.4.21-50.ELsmp ro root=LABEL=/
initrd /initrd-2.4.21-50.ELsmp.img
Change the number next to default to the number of the kernel you want to change to.
They are numbered 0 through to what ever number of images you have on your system.
fFr instance in the ones in posted above if default was set to =0 then it would boot into 2.4.21-50.EL if I set it to =1 it would boot into 2.4.21-50.ELsmp
If you only have one kernel listed then you will need to compile a new kernel and change the default number to it.
But before you do all this you might want to check with your data center and ask them about this xen kernel and if they installed it.
Why the load is so high is because of the wait processes what is causing that I do not know there is not enough info posted here to really tell you.
After changing the default number you will need to reboot in order to change the kernel.
|
| Posted by lovelynetworks, 07-08-2008, 12:38 AM | | for the last few hours i am unable to login through ssh. Restarted server several times.
I can login to WHM, server load is 5 currently and still it wouldn't let me ssh to server. it asks for user then pass, when i enter password it just freeze there and timeout.
i have contacted my host, they haven't installed xem kernel in our server or WHM VPS optimized version.
Is it possible that hard drive is failing?
|
| Posted by ub3r, 07-08-2008, 02:25 AM | | It could definitely be a failing drive. if ssh gets back to accessible, could you run `dmesg | tail -n 50` in ssh and send us the output?
If whm is accessible but sshd isn't, then that's pretty weird. Isn't there a way to restart the ssh daemon via cpanel?
|
| Posted by lovelynetworks, 07-08-2008, 02:55 AM | | finally got into ssh.
here is the output.
|
| Posted by lovelynetworks, 07-08-2008, 03:14 AM | | seems like harddrive is failing from these logs.
Is there anyone who can help me with this issue? i can pay
Server is with FDC and they will replace the hard drive for free.
I need someone who can transfer all data to new hard drive and get the sites up again.
|
| Posted by ub3r, 07-08-2008, 03:26 AM | | It's going to be pretty hard to get data off the drives if they're that badly damaged. Your best bet would be to get a kvm from fdc, boot the system into rescue mode linux, and attempt copying the data to a new drive that way. Just don't forget to reinstall grub over the mbr on the new disk.
Try the wht employment forums for finding someone to help. it's against the rules for people to offer to help for money here.
|
| Posted by HostingBig, 07-08-2008, 03:33 AM | | I could do that for you
but it would be best to make Cpanel backups and restore from them
you should probably run a manual fsck
and test for bad blocks
(your provider should help you do this)
|
|
 Add to Favourites
 Print this Article
|
