Category: Uncategorized

That was no fun at all!

The entire server was offline due to a failed RAID SAS controller card!

The issue is resolved and we have recovered as best we could from the data corrupt.

In our “autopsy” of what went wrong, it was discovered that multiple issues were at fault. Some was hardware and lack of having spares on hand and some of backup related – not having the proper disaster recovery plan.

That is almost laughable because that is what I do for a living (I, being John M. Hoyt)… I simply did not put enough emphasis on my own project and that caused backups of the database for Palmetto Showcase to be for versions other than the backups of the plugins and apps.

What have I done to make sure this doesn’t happen again?

Well, first, we fixed the hardware – but we have a spare part coming in, just in case. And another Upstate Live Music member has offered up a spare server to be used as a fail-over.

We have an IMAGE of the full server, stored to an external media.

We have automated dumps of the SQL databases – all of them.

We have automated offsite backup of the Palmetto Showcase databases and plugins.

I *think* we are much better off than we were.

Not saying we cannot suffer another catastrophic loss, just that it would be much easier to recover from it.

Thanks for your patience!

-John

Recovery is progressing well…

We have managed to restore all systems to full functionality, and begun recovery of as much data as possible.

So far – we have recovered all graphics media that was uploaded and we are going to attempt fix a long term issue where it could not be manipulated by “contributors”, forcing them to keep uploading new graphics each time they added an event.

Backup and recovery plan is being modified to better suit our needs.

We now are using Updraft to backup automatically to dropbox – all files weekly, and databases daily.

An image of the entire virtual server will be made weekly to an external drive and that drive will be backed up to the cloud daily.

Our admin, Ray Akey, will create a sql dump of all server databases, which will go to the external drive and be synced to the cloud daily.

I assure you, this problem will not affect us again!

-John

On 2023-07-05, our server suffered from a major hardware failure and subsequently, the databases were corrupted beyond repair.

To make matters worse, backups were NOT tested nor were they current so it was determined we were better off to start fresh.

I’ll be working diligently to resolve the issue as fast as possible.