High load due to deep crawling spiders....
What's up?
I need some help with my server setup..... Optimizing it specifcally....
Amd Sempron 2400+
1.5 gigs ram
80 gig hd.
First is my server-status.
This is at a load of only 20. It gets upto "100-150 requests currently being processed" when the spiders hit really hard, right now it's not too many spiders but they are still there....."32 requests currently being processed".
Server Version:
Server Built: Feb 4 2006 17:38:09
--------------------------------------------------------------------------------
Current Time: Tuesday, 07-Mar-2006 17:03:15 CST
Restart Time: Tuesday, 07-Mar-2006 13:40:12 CST
Parent Server Generation: 16
Server uptime: 3 hours 23 minutes 3 seconds
Total accesses: 137448 - Total Traffic: 1.2 GB
CPU Usage: u149.57 s25.47 cu0 cs0 - 1.44% CPU load
11.3 requests/sec - 100.1 kB/second - 8.9 kB/request
32 requests currently being processed, 17 idle servers
WW_.__W__WWWWWWW__W_WW_WWWR_WW__W_WWWW_WWWW___.....WWWW.........
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process
Server Built: Feb 4 2006 17:38:09
--------------------------------------------------------------------------------
Current Time: Tuesday, 07-Mar-2006 17:03:15 CST
Restart Time: Tuesday, 07-Mar-2006 13:40:12 CST
Parent Server Generation: 16
Server uptime: 3 hours 23 minutes 3 seconds
Total accesses: 137448 - Total Traffic: 1.2 GB
CPU Usage: u149.57 s25.47 cu0 cs0 - 1.44% CPU load
11.3 requests/sec - 100.1 kB/second - 8.9 kB/request
32 requests currently being processed, 17 idle servers
WW_.__W__WWWWWWW__W_WW_WWWR_WW__W_WWWW_WWWW___.....WWWW.........
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process
top - 16:47:27 up 19 days, 23:45, 2 users, load average: 10.35, 16.06, 17.09
Tasks: 149 total, 10 running, 139 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 18.9% sy, 62.1% ni, 0.0% id, 18.6% wa, 0.3% hi, 0.0% si
Mem: 1490040k total, 1069324k used, 420716k free, 139908k buffers
Swap: 917496k total, 2604k used, 914892k free, 139548k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16117 nobody 20 4 22400 13m 12m S 8.0 0.9 0:00.36 httpd
15834 nobody 19 4 28024 19m 12m S 7.3 1.3 0:02.77 httpd
16061 nobody 20 4 28944 19m 12m S 6.0 1.4 0:00.79 httpd
16125 nobody 20 4 28772 19m 12m S 5.0 1.4 0:00.27 httpd
13955 nobody 19 4 28980 20m 12m S 4.7 1.4 0:07.67 httpd
16052 nobody 20 4 27984 19m 12m S 4.3 1.3 0:00.78 httpd
16145 nobody 21 4 20404 11m 12m S 4.3 0.8 0:00.13 httpd
16131 nobody 20 4 24144 15m 12m S 3.0 1.0 0:00.11 httpd
16133 nobody 20 4 24456 15m 12m S 3.0 1.1 0:00.11 httpd
16044 nobody 20 4 29184 20m 12m S 2.7 1.4 0:00.88 httpd
16142 nobody 19 4 23792 14m 12m S 2.7 1.0 0:00.08 httpd
15836 nobody 19 4 27860 19m 12m S 2.3 1.3 0:02.18 httpd
16059 nobody 19 4 27836 18m 12m S 2.0 1.3 0:00.55 httpd
16091 nobody 19 4 27884 19m 12m S 2.0 1.3 0:00.60 httpd
15574 nobody 19 4 29188 20m 12m S 1.7 1.4 0:03.30 httpd
15777 nobody 19 4 29144 20m 12m S 1.3 1.4 0:02.68 httpd
16074 nobody 19 4 24240 15m 12m S 1.3 1.1 0:00.41 httpd
16137 mysql 21 6 53772 28m 3760 S 1.3 2.0 0:00.04 mysqld
16147 mysql 21 6 53772 28m 3760 S 1.3 2.0 0:00.04 mysqld
13833 nobody 19 4 28176 19m 12m S 1.0 1.3 0:08.65 httpd
14231 nobody 19 4 28584 19m 12m S 1.0 1.4 0:06.38 httpd
14527 nobody 19 4 28116 19m 12m S 1.0 1.3 0:05.99 httpd
16060 nobody 19 4 27832 18m 12m S 1.0 1.3 0:00.39 httpd
15831 nobody 19 4 24132 15m 12m S 0.7 1.1 0:00.94 httpd
16035 nobody 19 4 20524 11m 12m S 0.7 0.8 0:00.22 httpd
16058 nobody 19 4 27812 18m 12m S 0.7 1.3 0:00.37 httpd
16081 nobody 19 4 24216 15m 12m S 0.7 1.1 0:00.39 httpd
20382 mysql 21 6 53772 28m 3760 R 0.3 2.0 1:00.95 mysqld
14991 nobody 19 4 28072 19m 12m S 0.3 1.3 0:04.18 httpd
15833 nobody 20 4 27888 19m 12m S 0.3 1.3 0:03.16 httpd
15985 nobody 19 4 28084 19m 12m S 0.3 1.3 0:01.81 httpd
16126 nobody 19 4 20480 11m 12m S 0.3 0.8 0:00.13 httpd
16134 nobody 19 4 20588 11m 12m S 0.3 0.8 0:00.02 httpd
1 root 16 0 3036 552 1424 S 0.0 0.0 0:11.96 init
2 root 34 19 0 0 0 S 0.0 0.0 0:12.19 ksoftirqd/0
3 root 5 -10 0 0 0 S 0.0 0.0 0:01.24 events/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
30 root 5 -10 0 0 0 S 0.0 0.0 0:00.72 kblockd/0
43 root 13 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
31 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
42 root 16 0 0 0 0 S 0.0 0.0 4:56.09 kswapd0
116 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
183 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 ata/0
185 root 22 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
186 root 23 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
194 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord/0
203 root 15 0 0 0 0 S 0.0 0.0 8:59.57 kjournald
1101 root 6 -10 1768 512 1356 S 0.0 0.0 0:00.03 udevd
1548 root 16 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
3240 root 16 0 2656 592 1356 S 0.0 0.0 0:46.54 syslogd
3244 root 16 0 3172 468 1308 S 0.0 0.0 0:00.00 klogd
3282 root 16 0 2000 588 1360 S 0.0 0.0 0:00.36 rpc.idmapd
Tasks: 149 total, 10 running, 139 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 18.9% sy, 62.1% ni, 0.0% id, 18.6% wa, 0.3% hi, 0.0% si
Mem: 1490040k total, 1069324k used, 420716k free, 139908k buffers
Swap: 917496k total, 2604k used, 914892k free, 139548k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16117 nobody 20 4 22400 13m 12m S 8.0 0.9 0:00.36 httpd
15834 nobody 19 4 28024 19m 12m S 7.3 1.3 0:02.77 httpd
16061 nobody 20 4 28944 19m 12m S 6.0 1.4 0:00.79 httpd
16125 nobody 20 4 28772 19m 12m S 5.0 1.4 0:00.27 httpd
13955 nobody 19 4 28980 20m 12m S 4.7 1.4 0:07.67 httpd
16052 nobody 20 4 27984 19m 12m S 4.3 1.3 0:00.78 httpd
16145 nobody 21 4 20404 11m 12m S 4.3 0.8 0:00.13 httpd
16131 nobody 20 4 24144 15m 12m S 3.0 1.0 0:00.11 httpd
16133 nobody 20 4 24456 15m 12m S 3.0 1.1 0:00.11 httpd
16044 nobody 20 4 29184 20m 12m S 2.7 1.4 0:00.88 httpd
16142 nobody 19 4 23792 14m 12m S 2.7 1.0 0:00.08 httpd
15836 nobody 19 4 27860 19m 12m S 2.3 1.3 0:02.18 httpd
16059 nobody 19 4 27836 18m 12m S 2.0 1.3 0:00.55 httpd
16091 nobody 19 4 27884 19m 12m S 2.0 1.3 0:00.60 httpd
15574 nobody 19 4 29188 20m 12m S 1.7 1.4 0:03.30 httpd
15777 nobody 19 4 29144 20m 12m S 1.3 1.4 0:02.68 httpd
16074 nobody 19 4 24240 15m 12m S 1.3 1.1 0:00.41 httpd
16137 mysql 21 6 53772 28m 3760 S 1.3 2.0 0:00.04 mysqld
16147 mysql 21 6 53772 28m 3760 S 1.3 2.0 0:00.04 mysqld
13833 nobody 19 4 28176 19m 12m S 1.0 1.3 0:08.65 httpd
14231 nobody 19 4 28584 19m 12m S 1.0 1.4 0:06.38 httpd
14527 nobody 19 4 28116 19m 12m S 1.0 1.3 0:05.99 httpd
16060 nobody 19 4 27832 18m 12m S 1.0 1.3 0:00.39 httpd
15831 nobody 19 4 24132 15m 12m S 0.7 1.1 0:00.94 httpd
16035 nobody 19 4 20524 11m 12m S 0.7 0.8 0:00.22 httpd
16058 nobody 19 4 27812 18m 12m S 0.7 1.3 0:00.37 httpd
16081 nobody 19 4 24216 15m 12m S 0.7 1.1 0:00.39 httpd
20382 mysql 21 6 53772 28m 3760 R 0.3 2.0 1:00.95 mysqld
14991 nobody 19 4 28072 19m 12m S 0.3 1.3 0:04.18 httpd
15833 nobody 20 4 27888 19m 12m S 0.3 1.3 0:03.16 httpd
15985 nobody 19 4 28084 19m 12m S 0.3 1.3 0:01.81 httpd
16126 nobody 19 4 20480 11m 12m S 0.3 0.8 0:00.13 httpd
16134 nobody 19 4 20588 11m 12m S 0.3 0.8 0:00.02 httpd
1 root 16 0 3036 552 1424 S 0.0 0.0 0:11.96 init
2 root 34 19 0 0 0 S 0.0 0.0 0:12.19 ksoftirqd/0
3 root 5 -10 0 0 0 S 0.0 0.0 0:01.24 events/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
30 root 5 -10 0 0 0 S 0.0 0.0 0:00.72 kblockd/0
43 root 13 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
31 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd
42 root 16 0 0 0 0 S 0.0 0.0 4:56.09 kswapd0
116 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
183 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 ata/0
185 root 22 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
186 root 23 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
194 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord/0
203 root 15 0 0 0 0 S 0.0 0.0 8:59.57 kjournald
1101 root 6 -10 1768 512 1356 S 0.0 0.0 0:00.03 udevd
1548 root 16 0 0 0 0 S 0.0 0.0 0:00.00 kjournald
3240 root 16 0 2656 592 1356 S 0.0 0.0 0:46.54 syslogd
3244 root 16 0 3172 468 1308 S 0.0 0.0 0:00.00 klogd
3282 root 16 0 2000 588 1360 S 0.0 0.0 0:00.36 rpc.idmapd
root@hostname [~]# free -m
total used free shared buffers cached
Mem: 1455 1056 398 0 137 142
-/+ buffers/cache: 776 678
Swap: 895 2 893
total used free shared buffers cached
Mem: 1455 1056 398 0 137 142
-/+ buffers/cache: 776 678
Swap: 895 2 893
AND HERE ARE MY CURRENT httpd.conf and php.ini settings ( that is where i need to optimize yes? )
httpd.conf settings....
# Default Setup
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 150
MaxRequestsPerChild 0
# TEST 1
Timeout 25
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 35
MinSpareServers 10
MaxSpareServers 15
StartServers 10
MaxClients 100
MaxRequestsPerChild 200
# TEST 2 / CURRENT
Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 30
MinSpareServers 10
MaxSpareServers 15
StartServers 10
MaxClients 150
MaxRequestsPerChild 150
PHP.INI settings....
LoadModule rewrite_module libexec/mod_rewrite.so
LoadModule expires_module libexec/mod_expires.so
LoadModule php4_module libexec/libphp4.so
LoadModule vhost_alias_module libexec/mod_vhost_alias.so
LoadModule security_module libexec/mod_security.so
LoadModule evasive_module libexec/mod_evasive.so
max_execution_time = 30 ; Maximum execution time of each script, in seconds
max_input_time = 60 ; Maximum amount of time each script may spend parsing request data
memory_limit = 2M ; Maximum amount of memory a script may consume (8MB)
;;;;;;;;;;;;;;;;;;
; Fopen wrappers ;
;;;;;;;;;;;;;;;;;;
; Whether to allow the treatment of URLs (like http:// or ftp://) as files.
allow_url_fopen = On
; Define the anonymous ftp password (your email address)
;from="john@doe.com"
; Define the User-Agent string
; user_agent="PHP"
; Default timeout for socket based streams (seconds)
default_socket_timeout = 60
[MySQL]
; Allow or prevent persistent links.
mysql.allow_persistent = On
; Maximum number of persistent links. -1 means no limit.
mysql.max_persistent = -1
; Maximum number of links (persistent + non-persistent). -1 means no limit.
mysql.max_links = -1
; Default port number for mysql_connect(). If unset, mysql_connect() will use
; the $MYSQL_TCP_PORT or the mysql-tcp entry in /etc/services or the
; compile-time value defined MYSQL_PORT (in that order). Win32 will only look
; at MYSQL_PORT.
mysql.default_port =
; Default socket name for local MySQL connects. If empty, uses the built-in
; MySQL defaults.
mysql.default_socket =
; Default host for mysql_connect() (doesn't apply in safe mode).
mysql.default_host =
; Default user for mysql_connect() (doesn't apply in safe mode).
mysql.default_user =
; Default password for mysql_connect() (doesn't apply in safe mode).
; Note that this is generally a *bad* idea to store passwords in this file.
; *Any* user with PHP access can run 'echo get_cfg_var("mysql.default_password")
; and reveal this password! And of course, any users with read access to this
; file will be able to reveal the password as well.
mysql.default_password =
; Maximum time (in seconds) for connect timeout. -1 means no limit
mysql.connect_timeout = 60
; Trace mode. When trace_mode is active (=On), warnings for table/index scans and
; SQL-Errors will be displayed.
mysql.trace_mode = Off
[mSQL]
; Allow or prevent persistent links.
msql.allow_persistent = On
; Maximum number of persistent links. -1 means no limit.
msql.max_persistent = -1
; Maximum number of links (persistent+non persistent). -1 means no limit.
msql.max_links = -1
My server uses a decent amount of mysql from many sites, dunno what you woulc consider alot, no forums or anything like that....yet....
How can I optimize the server for apache / php and mysql ... .? Especially for spider activity since this is where it really hurts me, I don't like the idea of banning all the search engine ip ranges too much, lol, doesn't seem too smart.... ( dumb fools over at webmaster world.... ) lol..
The load is as low as 0.3 or 5.00 most of the time when there are NO spiders on my site, but when google / yahoo / msn hit.... it's on, lol, the load has been at 20+ to 35.00 all day today, litterally since I woke up until now it's been elevated past 15 which has me really worried....
How can I make it so the spiders don't effect the load so much? I haven't installed zend since my last apache rebuild, will that help at all ? how about any other additional apache or php modules/ hacks/addons ?
OTHER NOTES: ( Just wanted to make one post instead of 3 ).
I am also looking for a GOOD reliable inexpensive dedicated server host and colocation host, with impeckable uptime and very reasonable prices... 1 TB bandwidth min a month. Any suggestions? the find host feature here at WHT really sucks , sorry to say... lol, i used it many times, with very diversified prices and get no offers....
I would also need to buy server equipment for colo, and I really like the idea of owning the server and colo'ing the bandwidth..... as long as the colo is trustable and won't jack my equipment.
