Serious Issues with RYL (includes an update on missing threads)
 

Hi Everyone,

I'm very sorry for the RYL downtime since just after 2pm this afternoon. We experienced yet another power failure to our equipment in the London datacenter at 2:08pm - because of the unexpected shutdown our database sustained damage. Despite the subsequent automatic equipment restart, this left RYL in a damaged state.

I was away with a client this afternoon and left my appointment to rush back and get this fixed ASAP - it all takes time though, so I am sorry that you should have been without RYL for so long.

This is now the third power related incident in a single month and I am now looking for some VERY good answers from our colocation company concerning the exact cause. This level of service is of course, completely unacceptable.

Thank you all for your patience, and I will post back to this thread as and when I have further information.


James

Not your fault James, thank you for getting to it so quickly =)

Chloe x

Thanks for letting us know.

thanks James

x

thank you =]

Thanks for getting back here so quickly to sort it out. That's dedication right there! =]

Chelsea xx

Bless you for leaving your appt mate !
Thanx for getting it back up so quickly.

Theere was a sudden influx to chat at the time so s'alrite. Haha.

THANK YOU!!!
james your the greatest for getting it up so quickly and for being so committed

:thumbup: :cool: :hop: :jumpin: :-p

Thank you for letting us know :)

Thankyou!! :D

Thanks for the update.

I missed missing RYL this afternoon because I was in therapy!

Thanks James, I wondered what was going on. Just one question, is this the reason rant threads are disappearing? =/

Thanks a lot, James. :)

You're ace, and I do hope that you get those very good answers!

Don't work too hard now! x

thanks james your the best.
i thought it had gone down again.

Thanks everyone :) Lanny, to answer your question, no at this stage we have no reason to believe R&V threads not linking from people's profiles is caused by any database problem. This is likely a forum permission issue, which we are currently investigating.

Okey dokey, thank you ^_^

Cheers :)

I don't suppose the Rainbow Room disappearing has anything to do with the issues?

Thank you :-)

awww James you're a sweetie. mand x

"Passes bat"

Tell the company hi, from me. :satisfied:

Dicky!

You left your appointment just for us? That's so.... sweet and yet a little crazy. but thanks, it's so great to know people work so hard to keep this site in such good health :)

Woooo for RYL resurrection!

LOL ;]

THE RAINBOW ROOM'S GONE?!

Sad times.

& James, thank you.
:)

Another beloved thread is missing too. :(

I was hoping it had been accidentally deleted, but perhaps not.
Ho hum.

My guess is that the threads have gone because they're too large.

It cant be a largness thing because the safe room is the biggest thread and still in place.
My guess is that it is the older threads missing.

The other beloved thread and the Rainbow room where made before v3 went live, but I think the safe room was made a short while afterwards.
Hopefully its not permant.

Ah Thank you James, for giving up paid work to help RYL. You really are a star.

I'm very concerned about the Rainbow Room going missing and this is, I'm afraid, undoubtedly some side-effect of our power interruption earlier today. While I have a full backup of RYL from the early hours of yesterday morning, there is no easy way to re-integrate a thread from this into a live forum.

I will be doing my utmost tomorrow to run full consistency checks on all aspects of the forum to try to resolve these problems. There is a good chance the threads should be recoverable, but to be very honest, these kind of electrical interruptions leave servers and databases as busy as RYL's in a very bad way - the fall out can be very extensive.

Please keep all discussions about missing threads to this thread. Other than the Rainbow Room, if anyone is able to let me know the definite tittle for threads they can confirm are missing, please post them here.

Thanks so much everyone for all the nice things you said :) I won't be truly happy until I can get this data back for you all though.

The worst thing is we might get all of this fixed, only to have the plug pulled on us yet again.... :(

baaaaaaaaaaad power, bad *spanks*

James, I will pm you with the name of the other thread that I know is missing.

thanks for letting us know James. and thank you for leaving your paid work just for us *hugs* it's really sweet of you, though just a bit crazy. proves you're related to Harley and Tony though!
good luck with getting everything fixed.
and at least the power was only out for a little bit over there. yesterday afternoon we had the power out for over 6hrs!

soph.

Thank-you James for everything, you truely are AMAZING

I don't know of any other thread that has disappeared... but hey, if you can't get them back, that's okay!!! Don't pull your hair out over it!!!
Threads can be reborn from their ashes *nods*

From Chels from a post re. the other missing post.

I am sure that my post count has not dropped and it appears that Kims post count hasnt dropped either. Both of us were significant posters in the other missing thread - maybe the prolific rainbow room posters could check?

It was RYL's first birthday yesterday and I wonder if that is part of the reason the threads have vamoosed?

I have no idea what my post count was before RYL went down :| otherwise I'd check mine.

But I was quite high up on the first page, and I doubt my RR posts would be enough to make me go off it.

rainbow room! :(.

James your a star...i have no idea how RYL would function without you...infact...it wouldnt!

Thanks James. I would just like to say that i logged on to Ryl at about 1.10 pm and i clicked onto serious discussion and it comes up with database error, the same goes for the creative corner, and moving forward. Does everyone have this problem?

Thanks for the heads-up Avalanche - I was aware and was working on the problem already. Still trying to deal with fallout from yesterday.

Sorry for the additional downtime over the last 15mins everyone - I have to take the database server down while I fix database damage which means RYL has to come offline.

james do u know wats causing these power serges and is chat on a different server as it was the only thing tht has just servived a massive crash of ryl which everything went down apart from chat

*looks up feels silly* fair enough i know you have to take it offline to sort out how it going you found the cause of the serges and cheers for letting us know

We don't have information from our colocation company yet because their suppliers (who own the datacenter) have not given them any reasons for the outages either. We are all massively frustrated as you can imagine and, worse still, we have no guarantee that the original electrical fault has been definitively fixed yet.

Re missing threads:

I'm working to see if we can recover the Rainbow Room, but I regret to say things are not looking all that great at the moment (someone was probably actively working with the thread when the server went down yesterday).

This is a long shot but does anyone have a bookmark or web favourite into the Rainbow Room? This will give me the thread ID which would be enormously helpful for potential recovery of the thread.

I know that I don't, but I think Aidey said that she had the URL to the other missing thread in the staff forum.

Was that in the Common Room Chels? Her URL would be very useful! I'll go try find the thread so I can post in there and grab her attention.

UPDATE: I've put the word out on that thread too now.

she said somethng about it haveing its own page because she uses firefox.

Fingers crossed she still has that URL! Anyone have similar for the Rainbow Room?

Heads up? Thanks James your working really hard. :thumbup:

UPDATE: We now appear have the ID for the large staff thread that has gone missing so, any staff reading this, don't worry about that one.

We're on the hunt for the Rainbow Room ID though. Anyone have RR in their web history maybe?

RYL went off for a while not long ago, is that to do with this issue James?

No i dont have it in my history. What does heads up mean though? I do hope that your able to recover the rainbow room. *Fingers crossed*