Why not Join Amazon Kindle Unlimited 30-Day Free Trial?

Author Topic: December 2015 System Outage and issues  (Read 1885 times)  

Online Betsy the Quilter

  • To paraphrase Bill Murray in Groundhog Day: "I'm a goddess; not THE Goddess. I don't think."
  • Administrator
  • Status: Shakespeare
  • *****
  • Posts: 62969
  • Gender: Female
  • Alexandria, VA
  • Love all, moderate all, to quote my friend Harvey
    • View Profile
    • Betsy True Designs
December 2015 System Outage and issues
« on: December 04, 2015, 03:00:53 AM »
We had a severe database crash on 12/2.  The partial restore to our database did not repair the issues, so our Tech Admin took the forum down last night and did a full restore from our last complete backup on November 30.  I apologize for the loss of data.

If you experience any wonkiness or error messages, please post here to include the exact error message that you receive. 

Thanks,

Betsy
KB Admin
Goodreads Goal -- 75 books


Joined 10/27/2008  | Miss you, Harvey, Jeff & Dona!
Harvey Chute (1962-2015), KB Founder

Proud Owner: Oasis/Voyage/Touch/Basic/K1/Fire HDX
Betsy True Designs Facebook | My Store

Offline Anma Natsu

  • Status: Arthur C Clarke
  • *****
  • Posts: 2576
  • Gender: Female
  • Texas
    • View Profile
    • Anma Natsu
Re: December 2015 System Outage and issues
« Reply #1 on: December 04, 2015, 07:24:21 AM »
I'm glad they were able to restore from a relatively recent back up (and hopefully they are going to do daily ones from now on?).  While it sucks to lose a few days of posts, the full restore is generally the better way to go.

Offline Eva Chase

  • Status: Jane Austen
  • ***
  • Posts: 295
    • View Profile
    • Eva Chase Author Website
Re: December 2015 System Outage and issues
« Reply #2 on: December 04, 2015, 07:34:23 AM »
Ah! I'm sorry to hear that happened, but also a little relieved--I noticed a thread I started was completely gone and was worried I'd broken some major rule or something. Thanks for keeping us up to date!

Eva Chase | Website | Tumblr | Twitter | Facebook | GoodReads

Online Betsy the Quilter

  • To paraphrase Bill Murray in Groundhog Day: "I'm a goddess; not THE Goddess. I don't think."
  • Administrator
  • Status: Shakespeare
  • *****
  • Posts: 62969
  • Gender: Female
  • Alexandria, VA
  • Love all, moderate all, to quote my friend Harvey
    • View Profile
    • Betsy True Designs
Re: December 2015 System Outage and issues
« Reply #3 on: December 04, 2015, 08:20:21 AM »
I'm glad they were able to restore from a relatively recent back up (and hopefully they are going to do daily ones from now on?).  While it sucks to lose a few days of posts, the full restore is generally the better way to go.

Thanks, Anma!  Carrie, the site owner, and our tech admin have been working on a plan to make sure going forward that we have more stability, and it includes changes to our back up procedures!  More when they finalize plans.

Betsy
Goodreads Goal -- 75 books


Joined 10/27/2008  | Miss you, Harvey, Jeff & Dona!
Harvey Chute (1962-2015), KB Founder

Proud Owner: Oasis/Voyage/Touch/Basic/K1/Fire HDX
Betsy True Designs Facebook | My Store

Offline Anma Natsu

  • Status: Arthur C Clarke
  • *****
  • Posts: 2576
  • Gender: Female
  • Texas
    • View Profile
    • Anma Natsu
Re: December 2015 System Outage and issues
« Reply #4 on: December 04, 2015, 11:05:13 AM »
Cool.  and *hugs* to the team, I'm sure it was stressful to deal with.

I had something similar happen to one of my sites once, which is when I learned my hosting company's nightly backups weren't really nightly.  Talk about unhappy!  Lost like two months of data :-(

Offline Annalog

  • Status: Edgar Allan Poe
  • *******
  • Posts: 6721
  • Gender: Female
  • Arizona, USA
  • Goodbye, Mr. Milquetoast
    • View Profile
    • Origami Turtle's photostream
Re: December 2015 System Outage and issues
« Reply #5 on: December 04, 2015, 11:32:58 AM »
Thanks for the update. I wondered what was going on when the Good Morning thread was telling me that no one hadposted for 120 days and asked if I wanted to start a new thread. Now I know why my puzzled post is missing.

Offline Ann in Arlington

  • Global Moderator
  • Status: Shakespeare
  • *****
  • Posts: 64601
  • Gender: Female
  • Arlington, VA
  • Time for Pi!
    • View Profile
Re: December 2015 System Outage and issues
« Reply #6 on: December 04, 2015, 11:55:12 AM »
I do think the 'daily' back ups were actually 'daily' . . . . but apparently some data got corrupted somewhere along the way. :(

My Kindles
Nautilus (Voyage)
Oasis
Hermoine (Fire HD8)
Hogwarts (KDX)
Ed's (Basic)
S7Edge (KApp for Android)
Galaxy (KApp for Android)
(Make your own reading bar) | Ann Von Hagel | Arlington, VA | kboards  MODERATOR

Online Betsy the Quilter

  • To paraphrase Bill Murray in Groundhog Day: "I'm a goddess; not THE Goddess. I don't think."
  • Administrator
  • Status: Shakespeare
  • *****
  • Posts: 62969
  • Gender: Female
  • Alexandria, VA
  • Love all, moderate all, to quote my friend Harvey
    • View Profile
    • Betsy True Designs
Re: December 2015 System Outage and issues
« Reply #7 on: December 04, 2015, 01:02:06 PM »
I do think the 'daily' back ups were actually 'daily' . . . . but apparently some data got corrupted somewhere along the way. :(

Yes, my understanding is that we've always had daily backups; but that some of them were incremental backups--only what had changed in the database since the last backup, and the full database backups were less frequent. The last full backup was done November 30, which was why we had to restore to that date.

I believe it was done that way partly to reduce load on the forum during the backups which was making it difficult for people to log in while the backup was taking place in the wee hours of the morning here.  But there is a trade-off, as we've found out (though not to the extent that Anma did).

The plan Carrie and our Tech Admin are working on should mitigate the problem.

Betsy
Goodreads Goal -- 75 books


Joined 10/27/2008  | Miss you, Harvey, Jeff & Dona!
Harvey Chute (1962-2015), KB Founder

Proud Owner: Oasis/Voyage/Touch/Basic/K1/Fire HDX
Betsy True Designs Facebook | My Store

Offline The Hooded Claw

  • Status: Emily Dickinson
  • *******
  • Posts: 9445
  • Gender: Male
  • Oklahoma
  • Card-carrying melodrama villain
    • View Profile
    • All about me....<hee hee>
Re: December 2015 System Outage and issues
« Reply #8 on: December 04, 2015, 06:37:28 PM »
Here's your business opportunity--Marketing t-shirts reading: "I survived the great KBoards Crash of 2015."

They'd sell like hot cakes....

(Make your own reading bar)
"Once you can accept the universe as matter expanding into nothing that is something, wearing stripes with plaid comes easy."  - Albert Einstein

Online Betsy the Quilter

  • To paraphrase Bill Murray in Groundhog Day: "I'm a goddess; not THE Goddess. I don't think."
  • Administrator
  • Status: Shakespeare
  • *****
  • Posts: 62969
  • Gender: Female
  • Alexandria, VA
  • Love all, moderate all, to quote my friend Harvey
    • View Profile
    • Betsy True Designs
Re: December 2015 System Outage and issues
« Reply #9 on: December 04, 2015, 08:16:38 PM »
I'm thinking of margarita mix instead....
Goodreads Goal -- 75 books


Joined 10/27/2008  | Miss you, Harvey, Jeff & Dona!
Harvey Chute (1962-2015), KB Founder

Proud Owner: Oasis/Voyage/Touch/Basic/K1/Fire HDX
Betsy True Designs Facebook | My Store

Offline Linjeakel

  • Status: Edgar Allan Poe
  • *******
  • Posts: 5461
  • Gender: Female
  • United Kingdom
  • Erm .....
    • View Profile
Re: December 2015 System Outage and issues
« Reply #10 on: December 05, 2015, 04:11:11 AM »
Thanks for keeping us up to date, Betsy.

After the first restore, I re-posted my missing topic, so when I logged in again yesterday and it had disappeared again I began to feel like I was stuck in my very own Groundhog Day??? It's Carrie I really feel for - she's been thrown in at the deep end, hasn't she?

The people who are responsible for these denial of service attacks should be hung up by their f$%*ing whatsits.

"A person who publishes books, willfully appears before the populace with his pants down." Edna St. Vincent Millay


Member since 03/17/2010 | (make your own reading bar) |
My Kindles: Kindle Voyage  | Kindle Paperwhite  |  Fire HD 10  | Kindle DX  | Kindle Touch

Offline Ann in Arlington

  • Global Moderator
  • Status: Shakespeare
  • *****
  • Posts: 64601
  • Gender: Female
  • Arlington, VA
  • Time for Pi!
    • View Profile
Re: December 2015 System Outage and issues
« Reply #11 on: December 05, 2015, 06:31:20 AM »
Thanks for keeping us up to date, Betsy.

After the first restore, I re-posted my missing topic, so when I logged in again yesterday and it had disappeared again I began to feel like I was stuck in my very own Groundhog Day??? It's Carrie I really feel for - she's been thrown in at the deep end, hasn't she?

The people who are responsible for these denial of service attacks should be hung up by their f$%*ing whatsits.



Well, to be fair, the second time was on purpose.

Though I agree with your suggestion for appropriate punishement for DoS organizers.

I gather -- Betsy or BigFresh can correct me if I'm wrong -- that the initial restore wasn't working properly so it was deemed smarter to take the board down and do a PROPER restore. Unfortunately the data corruption meant that nothing from after early a.m. on the 30th was 'clean'. :-(

I am simply referring to that time as 'The Empty Days'. ;D

My Kindles
Nautilus (Voyage)
Oasis
Hermoine (Fire HD8)
Hogwarts (KDX)
Ed's (Basic)
S7Edge (KApp for Android)
Galaxy (KApp for Android)
(Make your own reading bar) | Ann Von Hagel | Arlington, VA | kboards  MODERATOR