server/forums crash - Thurs 20th May

[eFestivals]

The forums were offline for a number of hours overnight, and the whole of efestivals was down for a while.

This was caused by a power failure at the datacentre where our servers are located.

There was initially a power cut just after midnight, which was successfully covered by the backup systems (UPS & generators). However, shortly after the power came back on the power control systems failed, causing loss of power from both the mains and backup systems - this second power cut caused the servers and internet access to those servers to go down. Power was restored as soon as possible after that.

Because of the power failure, some of the database tables used by the forums were corrupted, which caused he forums to not work when the server came back online. As soon as this problem was found and identified this morning, the tables were repaired and the forums brought back online. As far as I'm aware no forum posts have been lost by this having happened.

Everything should now be back to normal - apologies for any inconvenience.

The hosting service which eFestivals uses assures us that the power systems are tested regularly, and were last tested at the end of April and were problem free, and that this issue could not have been foreseen. eFestivals has been using the same hosting service for over ten years now, and this is the first time they've had a power outage in all that time. The hosting service used regularly wins "top UK hosting service" or similar in relevant magazines, and in our opinion are fully deserving of such praise - we'll be sticking with them because overall we're very happy with the service they give.

[midnight_toker]

Hi Neil, thanks for letting us know. Did this also affect the festivals email server? I haven't had any emails since about midnight last night and am waiting for some - I've tried sending myself a couple from an external email (gmail) and they haven't come through either.

Thanks!

[eFestivals]

Hi Neil, thanks for letting us know. Did this also affect the festivals email server? I haven't had any emails since about midnight last night and am waiting for some - I've tried sending myself a couple from an external email (gmail) and they haven't come through either.

Thanks!

Thanks for letting me know about this.

Having just made a quick enquiry, the situation is as follows.....

The email for the festivals.co.uk accounts goes thru one of the servers of the hosting company, for anti-virus &/or anti-spam scanning reasons (I can't remember if it's used for one or both).

Since their power cut last night, that scanning server has been out of action for some reason, and they already have an engineer looking at it. They wouldn't give a time for when it might be resolved.

I didn't push too hard for info, cos I was getting a distinct "I can't be fixing things when I'm telling you things are being fixed" vibe down the phone - I can imagine it's been a very hectic day for them. I didn't need to phone them until now, because I was able to sort out the problems I found with my own setup myself, but I'd guess that's not the case for many of their customers.

Anyway, I didn't get around to asking if mail would be lost or not, but I'm confident that they'll be doing all they can to ensure as little as possible is lost, if any.

I'll post any updates I get here as and when I know them. If I've heard nothing further by 10am tomorrow morning I'll chase it up.

Please note that when mail does start to come thru it might not come thru in date order, and the backlog might take a while to be worked thru.

[midnight_toker]

OK, cheers for looking into it.

One of the emails I'm expecting is telling me when I'm supposed to be working this weekend - if I didn't get the email on time then it looks as though I've got the weekend off!!

Thanks Neil

[eFestivals]

I've just been mailed to say that mails should now be coming thru - please post back here to say they are if they are.

[eFestivals]

I've just sent a test mail to a test inbox I have on the festivals.co.uk domain, and that got thru almost instantly, so it's defo working for new mail.

I'll find out what the score is for older mail.

[eFestivals]

The reply I got was as follows......

The remaining email will be stored on our servers or other ISP's. We can't say exactly how long they will take to come through, but I would imagine it would be in the next few hours.

While I can't say how their own servers might do things (it depends exactly what they mean by that), by "other ISPs" they're meaning other mail servers that have mail that they've been trying to deliver to festivals.co.uk accounts.

As standard, mail servers try to deliver an email at intervals, and it's normally 4 hours between delivery attempts - so I'd guess it's very likely that all missing mail will be received within 4 hours from now.

Apologies for the inconvenience, but hopefully things are resolved technically, and within a few hours everything will be back to normal.

[midnight_toker]

Hiya,

yeah, messages are beginning to trickle through now - hopefully by the end of the evening I'll have all the old ones too.

Ta