Great Lakes Bass Fishing Forum

Okay, this one was not my fault... but what a pain in the rear end!! Sorry the forum was down most of the day (and night too!!)!

Apparently an automatic update of the Linux ksplice process took a dump around 1am this morning (while I was asleep as I should be). If you don't know or care what that is that's okay. I didn't really care much about it either until this happened!! I was up later than I should have been working on updating new ad application codes to GreatLakesBass.com web pages still...

So I wasn't fully awake and alert yet when a client called me this morning before I'd even got on the computer to tell me that her email wasn't connecting. I check and I get no response from this whole server... not good. It was working fine when I shut my eyes last night!

After some quick, normal checks I got on the computer to my network people at the hosting company. The server needed to be rebooted after a 'kernel panic' shutdown (because of the bad ksplice update, remember? Sure you do! ;D )

For now, they shut off the ksplice process so it won't update anymore until they can find a fix to this new problem. It's way better to have ksplice working and on with a Linux server because it allows things like security updates to be applied to the Linux kernel without requiring a server reboot every time! We'll wait on that I guess hoping they find a solution soon that really works...

I thought we were back in business around 10:32am this morning! Things appeared to be working... until I tried this forum. As some of you probably saw, the forum suddenly developed an inability to connect to the MySQL database that runs this show!! The error usually means bad user name and/or password (or one other more technical problem with hashing passwords that I recently learned and fixed!!)... Because many of my other website pages use services from the forum software, they were down too like even the home page!! @$#%@#$!

Only I hadn't changed anything!?! So off I go on a hacker search. After some time of nothing much I found no evidence of changed forum files or intruded upon database. But I found I could not download the database, run a second successful database repair (the first one I ran as the root user actually said it worked?!?), or even make a copy of the database to another new database!! Very not good!!

I tried many things and the problem actually just seemed to get worse. I finally narrowed the issue down to something very wrong with an important table that handles user visit sessions. Without that table apparently nothing will work. I did not get a clear message what was wrong with it either - probably because I made it worse actually by trying so many different things to find a fix.

To make matters worse, since the server went down at 1am I did not get a system backup last night either so if I tried for my first time to restore a database from that huge backup process I would have lost at least a day of forum updates... aaargh! Fun for me, I tried to download the last daily backup - it was even bigger than I thought! Even compressed it was at 8GB when I finally canceled it since many programs can't even handle files that big!

I probably should have got  back online with my network people and hoped for a quick, clear solution but that's spilled milk now... I may have got hold of the right Linux administrator and had a 1 hour fix? Or it may have taken the 6 hours anyway that it took me to finally fix it myself... you never know when contacting support how it will go since you can't just get on the phone to the smartest administrator just like that (unfortunately)!

I finally gave up my continually failing efforts and did a graceful server reboot. It didn't go super graceful but it worked and came back up after a few minutes. Then, I got a clear message from MySQL that the sessions table was marked as crashed! Duh... I figured something like that but how do I fix it!?! Much to my shock, I ran a fix just on that table and this time it didn't time out - IT WORKED!!! I was able to download a backup of the database, optimize it and the forum started working again without any updates or changes to settings or files!!

No apparent loss of already posted data though we all lost the opportunity to visit for 6 or 7 hours today (and I'm waaaaayy behind on chores now!!!). I have no idea what happened to this table but I imagine it got corrupted during the crash or first reboot somehow. I don't know if I had just rebooted again if we would have been back in business right away hours ago?? The error messages were very misleading taking me down some worthless paths, that's for sure...

I have lots of databases but none (other than the adserver, which seems to be okay) are as active as this forum's database.

Sorry if I caused any frustration, issues, hair-pulling or disappointment with any of you while the forum (and half the website) was down! I don't have a great way other than social media that you aren't all on to provide notices. I'll have to think of something in case something like this happens again - say when I actually make the switch to the newer version of PHP in the next few days... still have that to look forward to!

Now, back to your regularly scheduled forum please, and thank you!

If I can't fix it with CTRL ALT delete I'm screwed... ;D

That's pretty common... I wish that would ALWAYS fix things...

Yep   Turn the power off and then back on.  If that don't fix it.   It's broke.

Sometimes... someone has to go beyond turning the power off and back on... sometimes... that someone is me...

Oh well. Someone has to do it. Might as well be me...