Solve server load issues caused by search engine bots

(was: “Maintenance mode during maintenance”)

Today I had problems posting on a platform, Alberto mentioned planned maintenance on twitter. I think disabling editing during maintenance can prevent situation where someone writes something just to end up on error page when trying to preview/save. I find it quite stressful to write something and then worry that I can loose it because of website error, very likely other people may have similar feelings.

… but I may be wrong!

Something similar happened last week, and in that case @Matthias confirmed that the server was undergoing planned maintenance. I have no information about this case!

People, please :smiley:

Of course I use maintenance mode during maintenance of Drupal :stuck_out_tongue:

I did not do any maintenance today though, and the “similar” case from last week was automated security update installation for Ubuntu, not Drupal. It caused (for the first time) side effects, yet works well normally, and I’m willing to trade the remaining risk of uptime disturbance for getting the security updates some hours earlier and while I sleep …

The onset of the problems with timeouts you were experiencing was at 2014-12-08 3:00am, with a spike in server load from (I think) Alberto’s blog. So, one of the usual scripted attacks / login attempt scripts etc. that go on all day anyway, just not causing issues with server responsiveness usually. Since it’s an exceptional case, I’m currently not considering technical countermeasures … as I’m doing that hosting with limited resources anyway.

If it is annoying you though, feel free to reopen this task and recommend me a good login and site abuse blocker, either for Drupal and WordPress, or server-wide. I’ll install it then. And remember, “Ctrl + A, Ctrl + C” is your best friend before posting here :slight_smile:

Reopening …

Just checked the logs and in addition to “scripted attack” login attempts, it’s mostly seemingly legit traffic from search engine robots (Yahoo Slurp, YandexBot etc.). Also, same load issue again right now. So I reopen this, I need to find a request rate limiting solution for these bots …

please feel free to rename

thanks @Matthias! please feel free to rename this issue to something more appropriate

maybe we could also warn people, with short paragraph above text area, that Ctl-C may save them some headaches?