Box compromised; answers, way to prevent huge bill again?

Hello all,

We have a dedicated box that appeared to have been compromised and was sending out data full rate (10Mbps, our connection speed). We were not notified by host (another issue entirely) and we did not manually monitor RTG, as we normally only do 5-10% of our allotted bandwidth per month.

I will copy a short segment of bandwidth logs recorded through the bandwidth function of Webmin (best we have unfortunately). I have blanked out our IP. Format is <IP>_<protocol>_<internal port>_<external port>=<inbound size> <outbound size>

xxx.xxx.xxx.xxx_UDP_41498_11282=0 65508
xxx.xxx.xxx.xxx_UDP_41498_7117=0 65508
xxx.xxx.xxx.xxx_UDP_41498_2920=0 65508
xxx.xxx.xxx.xxx_UDP_41498_6639=0 65508
xxx.xxx.xxx.xxx_UDP_41498_4639=0 65508
xxx.xxx.xxx.xxx_UDP_41498_4217=0 65508
xxx.xxx.xxx.xxx_UDP_41498_17640=0 65508
xxx.xxx.xxx.xxx_UDP_41498_10950=0 65508

We have pages upon pages of this. Aside from this, no other errant traffic. Never anything inbound either on this port. Appears to me to be DOS traffic outbound. Anyone more knowledgeable than me confirm?

When I logged into the box I killed some processes that were using CPU time, a couple needed kill -9 to terminate, and the traffic ceased. The largest (70%) was running by Apache user, and the process name was simply Perl (this was through ps). Unfortunately this is all that I remember about this unfortunate 1AM Monday morning. I was mostly steamed at having seen a giant bandwidth bill, and not thinking about preserving evidence or taking screen shots.

We do also have a whopping 2 minutes (killed process at 1:02AM, bandwidth logs reset every hour Box compromised; answers, way to prevent huge bill again? ) of the regular bandwidth log showing destination IPs. IIRC, only about 3 of them, and I remember at least one was in Brazil (no where near us).

Box was running CentOS 3.4, apache, mysql, php, perl, ftp (don't recall which), and our website was running Nuke. I would not be suprised if Nuke was the source of the issue, as it was not updated very recently, (because it breaks our website damn near every time we do) and it is known to be quite insecure in the first place.

So anyway, wondering if anyone can shed some light on anything, or if there is simply not enough info.





Also, and more importantly:

Could someone point me in the right direction of some software or methods to configure firewall to throttle down connections or even disable the connection based on a ruleset? I am not talking about the regular firewall ruleset blocking ports and such, more so if we setup FTP site, and want FTP traffic to go through, but someone starts pounding the downloads.

I am already figuring the firewall ruleset in place is absolute garbage. I did not set it up initially, I sort of inherited the box, and didn't really look things over very well - my bad Box compromised; answers, way to prevent huge bill again? )

I would also like to know if it would be possible to be notified via email (sent to SMS phone) for anomalous events, any ideas there?

We will be looking for a more proactive host, who monitors their network for such strange activity as a host going from 100GB a month traffic to 100GB a day for 36 days straight Box compromised; answers, way to prevent huge bill again? but short of that, we would also like to have some knowledge about how to do things for our selves on our own box if it is capable.

Thanks much for any responses. I know there are some very knowledgeable folks posting and also lurking in these forums.

 

 

 

 

Top