| Posted by bloodyman, 11-02-2014, 11:11 PM | | Hi
Today I've noticed on 2 servers, that strange files has been created, I assume - by apache process. They were owned by users from my webhosting server.
Here is stat of such example file:
File: `/tmp/undo.#prelink#.AUDlvP'
Size: 477736 Blocks: 944 IO Block: 4096 regular file
Device: 803h/2051d Inode: 92567 Links: 1
Access: (0600/-rw-------) Uid: ( 929/ *USERNAME*) Gid: ( 927/ *USERNAME*)
Access: 2014-11-03 03:30:01.000000000 +0100
Modify: 2014-11-03 00:13:01.000000000 +0100
Change: 2014-11-03 02:42:53.000000000 +0100
Access time/Change time was modified because I've moved this file to other temp folder for future investigation. Also I've marked "username" as *USERNAME* due to privacy.
On another server file has the same size/blocks/io blocks and content, but different user (also webhosting user) and creation time.
I've searched user accounts for any changed files and user apache logs, but the only common I've found that they are using outdated Joomla scripts on their servers. Also there were Joomla Login attempts (brute force) during time when files were created.
I've viewed both tiles in RAW mode and they are exactly the same.
I've attached one of those files in ZIP folder to this post.
Can anyone advice what it can be?
Attached Files
undo.#prelink#.zip
(137.3 KB, 6 views)
|
| Posted by bloodyman, 11-02-2014, 11:43 PM | | Some additional information - here are apache logs with date/time when files /tmp/undo.#prelink#. were created:
one server:
78.25.86.116 - - [03/Nov/2014:00:13:01 +0100] "POST /administrator/index.php HTTP/1.0" 303 - "-" "-"
Another server:
178.210.145.30 - - [02/Nov/2014:11:20:46 +0100] "POST /index.php HTTP/1.1" 200 226 "http://DOMAIN-NAME/index.php?option=com_content&task=view&id=5&Itemid=6" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11"
One is from Russia, other - from Ukraine.
|
| Posted by sneader, 11-02-2014, 11:47 PM | | Just wanted to let you know I'm seeing the same files. Started maybe a week ago. I've seen these files generated from sites running various scripts (Shopping cards, WordPress, etc.) so I don't think it's a Joomla thing.
|
| Posted by blade77, 11-03-2014, 11:10 AM | | Happens to me as well, is creating these prelink in / tmp files, I get just as DDoS attacks, or brute force, mine sites which use code php and codeigniter, and also as root.
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.0PWZYl
-rw------- 1 user user 477736 Nov 3 14:16 undo.#prelink#.5CbjkU
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.9SeKR9
-rw------- 1 user user 477736 Nov 3 14:15 undo.#prelink#.AqA4vr
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.CVHEF9
-rw------- 1 user user 0 Nov 2 07:31 undo.#prelink#.EdaAhX
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.hDUWL9
-rw------- 1 root root 477736 Nov 3 14:09 undo.#prelink#.j5jJzW
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.kpbqTr
-rw------- 1 root root 0 Nov 2 23:10 undo.#prelink#.NFhgkV
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.psQHI9
-rw------- 1 user user 477736 Nov 3 14:15 undo.#prelink#.rWNjJl
-rw------- 1 user user 0 Nov 3 14:15 undo.#prelink#.U8dclb
-rw------- 1 user user 477736 Nov 3 14:16 undo.#prelink#.wv9qib
any explanation ??
|
| Posted by FastServ, 11-03-2014, 11:30 AM | | I've noticed it on a couple Cpanel machines, but most interestingly -- just got some on a plain CentOS server WITHOUT PHP installed (just apache) with HAproxy sitting in front of it -- apache isn't accessible directly and only 80/443 is open to the world via HAproxy. Seems like something related to or similar to shellshock going on here, although everything's up to date... nothing lining up in apache or error logs.
Last edited by FastServ; 11-03-2014 at 11:36 AM.
|
| Posted by bloodyman, 11-03-2014, 01:19 PM | | Has anyone enough knowledge to analize this file? Seems that all of us have the same size of file.
I've read RAW format and found string libfreebl3.so in this file, but I'm not sure if this library have any connection with it.
|
| Posted by bloodyman, 11-03-2014, 01:38 PM | | of course I mean file analysis
Maybe some of you have also found this - here is some bug report in Redhat:
https://bugzilla.redhat.com/show_bug.cgi?id=584550
where I can see the same file (/tmp/undo.#prelink) and libfreebl3.so name.
Last edited by bloodyman; 11-03-2014 at 01:47 PM.
|
| Posted by FastServ, 11-04-2014, 08:30 PM | | Only thing I've come up with is something happened with the CentOS 6.6 update. Probably a bug affecting apache... surprised there isn't much info out there, I've got over 10 boxes doing this now, some with Cpanel, some without, some without PHP at all. The only common denominator is CentOS 6.6 and Apache.
|
| Posted by bloodyman, 11-07-2014, 12:43 AM | | Hi
Today this file happend again on one of the servers it happend before (so far there were 2 of my servers). It happend with different user name, here are stat details:
File: `undo.#prelink#.7NguGz'
Size: 477736 Blocks: 944 IO Block: 4096 regular file
Device: 700h/1792d Inode: 1050 Links: 1
Access: (0600/-rw-------) Uid: ( 625/ *DIFFERENT_USERNAME*) Gid: ( 623/ *DIFFERENT_USERNAME*)
Access: 2014-11-07 05:25:20.000000000 +0100
Modify: 2014-10-16 16:05:18.000000000 +0200
Change: 2014-11-07 05:23:01.000000000 +0100
Size is the same, username is different (cpanel user).
Modify time is the same as previous file with different user on the same server, here is stat for 2 files on the same server (this is different server than the one I posted in initial):
File: `undo.#prelink#.7NguGz'
Size: 477736 Blocks: 944 IO Block: 4096 regular file
Device: 807h/2055d Inode: 801260 Links: 1
Access: (0600/-rw-------) Uid: ( 625/ *USER_1*) Gid: ( 623/ *USER_1*)
Access: 2014-11-07 05:43:20.000000000 +0100
Modify: 2014-10-16 16:05:18.000000000 +0200
Change: 2014-11-07 05:43:06.000000000 +0100
File: `undo.#prelink#.Zk2AyM'
Size: 477736 Blocks: 944 IO Block: 4096 regular file
Device: 807h/2055d Inode: 801258 Links: 1
Access: (0600/-rw-------) Uid: ( 876/ *USER_2*) Gid: ( 874/ *USER_2*)
Access: 2014-11-07 05:43:20.000000000 +0100
Modify: 2014-10-16 16:05:18.000000000 +0200
Change: 2014-11-02 15:50:07.000000000 +0100
This is strange for me - modify time is 2014-10-16 while those files were shown in /tmp on 2014-11-07 05:23:01 and previously - 2014-11-02 15:50:07. Size is the same as before, content is the same.
Also - this time there is nothing in apache logs for 2014-11-07 05:23 for this user.
Last edited by bloodyman; 11-07-2014 at 12:51 AM.
|
| Posted by bloodyman, 11-07-2014, 01:00 AM | | edit]
here is the only thing in apache logs for the second file created on the same server:
173.208.205.194 - - [07/Nov/2014:05:23:01 +0100] "GET /DIRECTORY/ucp.php?mode=login HTTP/1.0" 200 9622 "http://DOMAIN/ucp.php?mode=login" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0"
This time 173.208.205.194 goes to US.
|
| Posted by UNIXy, 11-07-2014, 08:54 AM | | It's 'prelink' pre-linking libs and binaries to speed up run time.
Nothing to worry about IMHO.
|
| Posted by FastServ, 11-07-2014, 12:57 PM | | My question is why have these files never been detected until now? On some servers, they seem to be cleaned up automatically, and on others they are piling up and taking space in /tmp.
|
| Posted by UNIXy, 11-07-2014, 01:35 PM | | Prelink is an old program with ugly goto logic sprinkled all around. All it takes is a subtle change in code path execution (bug) for the file created with mkstemp() to not unlink(). And there you go the file stays in /tmp forever or until box is restarted. Someone needs to submit a bug report to Red hat as they're the maintainer.
|
| Posted by FastServ, 12-10-2014, 11:31 AM | | FYI, just got one on a Debian squeeze box. It's not Redhat specific.
|
| Posted by Steven, 12-10-2014, 12:12 PM | | I have yet to see this on any of the boxes we maintain
Anyone running snoopy on any of the affected boxes?
|
| Posted by Steven, 12-10-2014, 12:16 PM | | Since there is a connection to libfreebl3.so earlier in the thread:
Last edited by Steven; 12-10-2014 at 12:26 PM.
|
| Posted by Steven, 12-10-2014, 12:23 PM | | Did this line up with the @daily cron?
Every day /etc/cron.daily/prelink is ran, which does a re-prelink of things that are out of date.
Another thing to mention, cPanel does prelinks during upcp.
|
| Posted by FastServ, 12-10-2014, 12:29 PM | | Another thing I've noticed, they don't always stick around long. LFD may have become more sensitive in a recent update and picks up on them more quickly. I've actually put an ignore for them in csf.fignore on the most annoying machines.
|
| Posted by Steven, 12-10-2014, 12:31 PM | | I am reading the prelink documentation:
http://linux.die.net/man/8/prelink
I posted earlier that dovecot is doing a prelink command on cPanel machines:
According to the prelink documentation:
Are any of the boxes you guys are seeing this on, heavily utilized/low on available resources?
|
| Posted by Steven, 12-10-2014, 12:34 PM | | Last thing before I head off and eat breakfast. I ran this:
Which resulted in:
So to me it appears that this is from the -u flag, so what we need to determine is what is calling the -u flag on even NON cPanel boxes.
|
| Posted by Steven, 12-17-2014, 03:32 AM | | To go along with my previous suspicions looks like the nss-softokn-freebl package has issues, and one of those issues is failures during prelinking.
http://bugs.centos.org/view.php?id=7812
http://serverfault.com/questions/640...inux-is-active
Last edited by Steven; 12-17-2014 at 03:43 AM.
|
| Posted by Steven, 12-17-2014, 03:54 AM | | Looking at it further.
There is a commit in the change log:
I pulled out the patch from the SRPM which is named: nss-softokn-3.14-block-sigchld.patch
If you look at the code, they made some modifications to the prelink handling.
So it looks like the handling of sigchld in their new patch is causing these issues.
It would also be useful is someone could run yum -y downgrade nss-softokn-freebl and maybe reboot the system to confirm that what I think is happening is what is happening.
I haven't had any boxes my self doing it yet so I can't confirm my self.
Last edited by Steven; 12-17-2014 at 03:57 AM.
|
| Posted by sneader, 12-17-2014, 07:34 PM | | Hi Steven. I did the downgrade, but did not reboot, to see if it would help. It did not stop the creation of the prelink files in /tmp. I will reboot in the near future and report back.
- Scott
|
| Posted by Steven, 12-17-2014, 07:44 PM | | Scott,
Something may be holding on to the old library still. Alternatively could run a losf and see what is running it and restart that service.
|
| Posted by Srv24x7, 12-18-2014, 02:04 AM | | Hi,
It is quite strange, not to blame Apache, I got the same on the LiteSpeed web server and it was happening only occasionally.
|
|
 Add to Favourites
 Print this Article
|
