Kazza the Blank One

Dear Vodien, 

Here's a bowl of petunias:

Tuesday 27.8.24.

I went to update my blog because I realised I'd forgotten to add a photo from last week's entry. Loaded the blog interface but got this:

Got an error: Connection error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

Loaded terminal through cPanel. Which luckily worked (no CageFS errors this time). Ran my test perl script which connects to SQL and got this:

Content-Type: text/html; charset=ISO-8859-1
DBI connect('db:localhost','<user>',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at common.pl line 18.
Cannot connect: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

In phpMyAdmin in cPanel I got:

mysqli_sql_exception: Access denied for user '<user>'@'localhost' (using password: YES)

Fricken great. Just what I need. Feeling like I should be buying a bowl of petunias for Vodien.

Wednesday 28.8.24.

I remembered that I'd had the phpMyAdmin error before, and the fix was to just reset the cPanel password. So I did that. Great, phpMyAdmin loads.

Except.

My MySQL database had been reverted to a point in time from last October!!

!!!!

AGAIN!!

Yeah so the exact same problem as in June where they reverted it back to the same spot in October. In June, after a week or two of yelling at them, they restored it.

So of course I yelled at them.

I didn't send them a photo of a bowl of petunias. I should have.

Now before you ask, yes I do have recent backups of my database, having learnt my lesson from last time, but that's not the damned point. In fact I'd taken a backup the Sunday before, so I was only missing one entry from my blog. But as it turned out, having my own backup was not going to help. But we'll get to that later.

Thursday 29.8.24

Vodien ticket escalated (supposedly) to the sysadmins.

Friday 30.8.24.

Still no response from Vodien about the database restore.

Saturday 31.8.24.

Still no response from Vodien about the database restore.

Sunday 1.9.24.

Still no response from Vodien - three days now. Started a new ticket. Which they promptly closed.

I changed the permissions on mt-comments.cgi so noone could post any comments. They would either fail because the entries don't exist, or break the site with a republish. And I didn't want more mess to clean up.

Monday 2.9.24.

Vodien provided a restore of the database in an .sql file and said here you go. But it was the database at the "bad" restore point from October last year - so it didn't have any recent data in it.

Tuesday 3.9.24.

One week in.

No progress getting Vodien to restore the correct database for me.

Wednesday 4.9.24.

Vodien basically keep telling me I have to restore my database myself. Which I was considering doing, assuming I actually had the right data. I did end up getting a copy that was taken a few days before they broke it, but I decided to keep pushing for the whole thing.

Then I went to my blog interface to check it and got

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at webmaster@<domain> to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

And logging into the terminal in cPanel I got

cagefs_enter: Error entering cagefs jail: Unable to mount /var/cagefs/35/<user>/home2 -> /usr/share/cagefs-skeleton/home: Too many levels of symbolic links.

So either they were trying to fix it or just making it worse.

Thursday 5.9.24.

Slept like crap. Stressing about my website. At least the cagefs file system on my website was working again in the morning. But the database is still bad.

Friday-Sunday.

Long weekend in Port Macquarie.

Sunday 8.9.24.

Checked my mail when we were near home. I shouldn't have, because the lack of useful response was just making me more and more depressed.

Monday 9.9.24.

While not getting to sleep last night I wondered if maybe there were two database servers on the server. It always used to be MySQL, but the restore they did recently (and I think I might have noticed this previously) was from Maria DB. Same only different. My latest theory is there's actually the old MySQL database server still on the host, and for whatever reason the pointers for Perl and phpMyAdmin have been pointed back at it (again). I mentioned this to them, but their only response was "what's the URL of your blog entry manager?". Yeah right like I'm going to let you into that. So I gave them a command line command that they could use to query my address book database, which has also been impacted by this whole drama. That was in the morning. No response by evening.

Tuesday 10.9.24.

Two weeks in.

No response from Vodien. Eventually they responded with something that didn't make sense, so yelled at them again to put this through to a senior sysadmin.

As an aside: if you Google "Vodien is trash" (I literally did this) you'll get links to review pages that have page after page of people ranting about how utterly trash Vodien is. One star reviews all the way.

Wednesday 11.9.24.

Awake from after 3. Vodien sent me a database restore. But restored from 19 August which was before my last backup anyway.

Tried to reload my blog index, but I got:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at webmaster@kazza.ciapics.cia.com.au to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.

Turns out they've blatted all my cgi scripts permissions to 644 instead of 755 or 744. Every single one of them.

WTF?? I mean seriously, WTF???

So now what. Do I take their database, restore it, and hope they don't flip over to the other database at some random point (which I couldn't even find out from them if that's what had happened). Or do I just give up and take my clean backup to a new hosting company. Ok I'd pretty much made up my mind I was going to move. They've broken my site multiple times in the past few years, and it always takes days or weeks of yelling to get it resolved. I'm sick of it.

Thursday 12.9.24.

Apparently they've restored a database again.

Friday 13.9.24.

Tried opening my blog interface and got this:

Got an error: Connection error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)

Yet my Address Book scripts still work. Ok, gave it some time and it came back. But database was still from October 2023.

Saturday 14.9.24.

Resolved myself to spending my entire weekend sorting out this mess.

Step one: find a new hosting company.

Back in 2020 I tried VentraIP. It looked to have everything I wanted and was a good price. I talked to them online before I joined and said I had a Perl/MySQL setup and they said that was fine. But when I went to get it going I found they didn't have the DBD::MySQL perl module and my blog wouldn't work. So that was a complete waste of time. So I wasn't going to try them again.

Had a look on https://www.productreview.com.au/ and someone recommended CloudLoop and they had excellent reviews. So had a look. $10/month for a basic plan which would cover what I needed. They didn't have an online chat, so I emailed them asking about Perl/MySQL. This was fairly early on a Saturday morning. They responded in MINUTES! I was impressed. So I joined up. Pro-rata til the end of the month it was only around $5, so even if it didn't work out I wouldn't lose much.

After the account was activated a short time later, I logged into cPanel only to find they don't have Terminal enabled. Fricken great. So set myself up an SSH private key to try and use ssh. But Internode blocks ssh doesn't it. Fricken fricken. So tried to log into Internode, but their MFA is trash and I had to log off/on a few times before I could finally get in. Also, Stu's phone number is out of date, but we can't update it because:

Your contact details cannot be changed at this time
Unfortunately, your contact details cannot be viewed or changed online at this time. Please contact us to update your details. We apologise for any inconvenience.

Thanks Internode, sorry iinet, sorry TPG.

Anyways, eventually got in and turned off the firewall, now to wait and see if it will work.

In the meantime I emailed support about Terminal in cPanel. Again they responded in minutes. They said it was not available for security reasons, but they enabled ssh access and I had to whitelist my IP address in the firewall. Which I couldn't actually find.
But.
Terminal now showed up in cPanel! Great! That's really all I need anyway. Let's do this thing!
(I think they may have mixed up ssh/cpanel terminal - so ssh is disabled which is fair enough, but the terminal in cpanel had been enabled which is all I needed anyway).

So first step is to make sure Perl will work with MySQL. I imported my address book database (it's only tiny) and tried out one of my scripts.

Can't locate CGI.pm in @INC (you may need to install the CGI module) (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at quickie.cgi line 3.
BEGIN failed--compilation aborted at quickie.cgi line 3.

Turns out CGI.pm hasn't been around since Perl 5.22. Vodien has 5.16. CloudLoop has 5.26.

So I get CGI.pm from CPAN. Add a line to my script to use lib and the path of the file. Works!

Next up my blog script (mt-check to start with). It wanted Util.pm so copied that up as well.

But then it wanted FCGI.pm. Got that too, but this time I got this error:

Can't locate loadable object for module FCGI in @INC

Which I think means it needs to be properly installed/compiled for binaries, which I can't do. hmmmmm

Ok so then ran mt-check on the web. It actually ran! It complained about a bunch of modules, but then it complains on the old site as well. It loaded enough that I might be able to work with it.

So next I need to get my database imported. I still didn't have the most current database export from Vodien (their exports are a different format to mine), so I used my own backup. But I was running into all sorts of problems. phpMyAdmin wouldn't import the file, giving sql errors when importing the "blob" data. And the file is 50MB so trying to edit it and cut it down to just do individual tables was proving problematic. Now I didn't have these problems when I tried VentraIP a few years back, so I'm guessing there must be some versioning problems or something?? But I was getting super frustrated with it. I did have a go at using Vodien's sql file and that one did import, albeit with old data.

Sunday 15.9.24.

So with some actual blog data in the database I next had to get an eighteen year old release of Movable Type working on a modern server. Yes yes I know I should update to something else. But newer versions of Movable Type are $499USD. PER YEAR. And I've seen the dramas people have moving to WordPress. Stu has threatened to spend his retirement moving it to a more modern platform heh. But in the meantime, I just want my blog to work.

So first up I try running mt.cgi on the command line to see what sort of errors I'd get.

[user@server MT-5.2.2]$ perl mt.cgi
Possible precedence issue with control flow operator at lib/MT/App.pm line 1821.
Content-Type: text/plain; charset=utf-8
Got an error: Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/{{ <-- HERE support}}/?/

Yeah so now you need to escape {} characters in regex strings. Think I fixed that.

So then did a test post and that actually worked!! (using a hosts file entry on my computer so I could test it out on CloudLoop before changing DNS anywhere)

Well that was good enough for me to commit to moving the site across.

Tarballed my directory up on Vodien. ~2.5GB. !! Downloaded it to my computer, extracted it, saved into a number of smaller zip files, uploaded them one at a time, extracted them. Uploading large files must be a bit upsetting for Internode, and I kept getting dumped out of FTP because our IP address kept changing. Have I ever mentioned how trash Internode has become since they got bought out by iinet then TPG? Also I have a 5GB size limit on CloudLoop. Anyways, got there in the end.

Now all that's left is to get a proper copy of my database from Vodien. They did have a restore file taken on the morning of 26 August - the morning after my last blog post. But the files were owned by root and I had no read access. After a few chats/emails back and forth I realised there were actually two copies of the files, one in the root and one in my website. So got them. And they were perfect! Well, they had my latest data in them anyway.

Right, so tried to use phpMyAdmin to import it, but got errors again (didn't record what). But then I found this command

mysql -u database_username -p database_name < file.sql
from this site https://support.hostinger.com/en/articles/4536306-how-to-import-a-database-over-ssh
And that worked perfectly.

But it was at this point I realised that the export sql file was full of funky characters. Somewhere along the line "special characters" from foreign languages lost encoding and were broken. Tried doing search/replaces in vi, but it didn't want to play. So put that aside for now.

Tested posting some blog entries and they worked, so hopefully the database is working well enough.

But I was getting SSL certificate errors - saying it was self signed. I changed the DNS A record in Vodien and waited for it to change over, but I was still getting errors on https://www.sslshopper.com/ssl-checker.html#hostname=kazza.id.au

Emailed CloudLoop support again. Once again I got a response in minutes. And then sslhopper was picking the certificate up properly. Not sure if they did anything or not, but I was blown away by the responsiveness and usefulness of CloudLoop support. It's like a breath of fresh air after dealing with the faceless Vodien.

Monday 16.9.24.

Before all these dramas started I'd already been meaning to fix up my blogs properly. I'd fixed the style sheet on my main blog to be hardcoded to https, but I hadn't published it to everywhere, comments were still broken on https, and I wanted to get all my holidays blogs using a consistent design. I also wanted to replace the defunct Google Analytics scripts with the new ones. So after dinner I spent ages working on my holiday blog templates - fixing everything so it all works in https, fixing style sheet locations etc. I picked the Americas 2023 blog (the last one I did) and made sure everything worked properly, then made a Theme from it.

Tuesday 17.9.24.

Working through all my holiday blogs using the updated theme/design. The special characters in my Turkey blog were proving problematic :( Some of the characters I can do a search/replace in the blog interface and it'll go through and change them all. While this works for some characters (eg umlauts) it wouldn't work for some of the characters in the Turkish language :( I also found and fixed a time zone setting in my blog that has been annoying me for twenty years haha.

Wednesday/Thursday/Friday

Work on fixing holiday blogs.

Saturday 21.9.24.

Finished fixing up my holiday blogs (including the Australian Holidays blog which has a slightly different config to it, although have since found a couple of problems I still need to fix).

So then it was onto my behemoth of a main blog. There's over five thousand entries in it. The last time I tried to publish after design update it took over a day. Look I even took a screen shot! 

So I was dreading the pain it was going to cause. I'd taken notes during the fixing of my holiday blogs design, so it was pretty quick to repeat the changes in the main blog. But before I committed, I wanted to test the design with a test post to make sure it picked up everything ok.

Took a while and then BAM..

500
Internal Server Error
An internal server error has occurred.

The good news is the entry did actually post. Although not properly, and it was intermittent. Like it'd mostly update the entry page, but then not the archives page etc. So thinking there might be resource limits. I didn't see any real spikes/errors in the cPanel Resource Usage thingie though.

I tried clearing out the entire activity log. This cleared nearly 9MB out of the database. But it didn't help. I tried a few other Movable Type config options as well with no luck.

So I emailed CloudLoop support again. Once again I got a super quick response. And a stack trace. And they looked while trying to reproduce the problem themselves. This was all well and truly above and beyond the level of support I would have expected. I was astounded. But a definite possibility was the Assets. Since Movable Type 4, whenever you upload an image it would save it to the database as an asset. This annoyed me when it first started because of all the extra clutter, but I did like it simply because it'd read the image and put height/width info into the html. But after sixteen years, I had 14000 assets. !! So I deleted them all!! (well, firstly I tested what would happen if you did. I found it would delete the asset and the image itself, but it left the html alone. So all I needed to do was restore the images once I'd deleted the assets, and all would be well).

Then I did a test post.

It published.

In SECONDS.

!!!!

After a bit more testing of posting and commenting I was ready to release the hounds. I republished the entire site. I did it in chunks, but it was all done in under ten minutes. And search results are returned super quickly too now. My blog hasn't been this fast in YEARS!!!

The only real remaining issue now is the foreign characters. I download the mt-entry table (16MB) and did some search/replaces and was able to fix a lot of it. But there's some characters that I just can't publish because it won't talk to the database. There might be a way to fix it. Something about telling perl to talk to MySQL with the right version of utf-8. But I'm too dumb for that sort of thing. Replaced a good chunk of the bad characters. Almost accidentally blatted the .bak table because I forgot to change the table name in the insert into lines (yay for it not importing duplicate entries). Had to redo the whole thing again because I didn't escape ' and " characters. Then republished again. Nine minutes, fifty seconds. I should probably republish all my holiday blogs too.

Sunday 22.9.24.

Spent most of the afternoon blogging entries from the past month.

Monday 23.9.24.

Wrote up this post so far.

I still haven't changed my mail or DNS configs over yet. Or the websites for my subdomains. I'll do that first before I post this.

I haven't decided what I'll do with johnson.id.au or trainman.id.au. I'd actually prefer to keep trainman where it is, because then I'd be able to email David again (currently using my account as an SMTP server, which intercepts mail for trainman.id.au and delivers it to me instead of David).

I also need to figure out how to backup my database properly. I'll have to do some testing of exporting/importing the data on the same version. Otherwise I'm going to need to figure out a better way to backup the blob data in the database. If I hadn't been able to get the Maria DB dump from Vodien I would have been a bit stuck as I wasn't able to properly import my own database backup, which is quite upsetting.

Tuesday 24.9.24.

Started going through and documenting everything in my Vodien cPanel setup. Found a place for mail forwarding, so tried setting trainman.id.au to an external MX record to see what would happen if I sent mail there from my account. It hasn't bounced back to me, so maybe it did deliver to David. This is good to know if I end up moving trainman.id.au to CloudLoop. Also found another backup section which lets you download MySQL databases. This has the same format as what Vodien sent to me. Interestingly the Vodien export is full of trash characters, while the CloudLoop export is fine. This is good news. Dropped my TTLs on mail to 300. Setup my mail accounts on CloudLoop. Tried them out in Eudora. TLS errors to start with, but easily fixed by setting TLS to Required, Alternate Port in Eudora. So changed the DNS for mail.kazza.id.au to point at CloudLoop. So inbound mail all working. Had a look at migrating ciapics over. But it's half a gig. And I only have five on CloudLoop. Considering leaving it there, but I really want to take all of kazza.id.au off Vodien. Next up - SMTP. Got it to auth and send ok, but now to battle with the SPF record. Adding the IP address of my host didn't help. Or adding relay.mailchannels.net. hmm. On a lighter note, cPanel and DNS are actually integrated on CloudLoop which is nice.

Wednesday 25.9.24.

Tested mail again and my SPF record has propagated around so that's nice - can send to gmail now. Although this is all likely to be moot once I switch the nameservers across and use CloudLoops DNS which already has all this stuff. While clearing stuff out of Vodien I discovered that at some point they'd actually fixed the ownership of the db restores. Lolz.

Friday 27.9.24.

Setup johnson.id.au as a domain on CloudLoop.  Copied over the content (there's like, one page).  Then changed the nameservers from Vodien to CloudLoop.  Tested email once DNS propagated.

Saturday 2.11.24.

Been on CloudLoop for like six or seven weeks now. No issues. Today I changed the nameservers for kazza.id.au to point at CloudLoop. Pointed a few A names of my photo subdomains back at Vodien. They have a lot more storage so keeping my photos sites there.

Sunday 3.11.24.

Cleaned out most of kazza.id.au off Vodien, including my blog database. Left an entry in my addressbook database and make sure the scripts were working to access it. So I'll be able to tell when Vodien break it again.

In conclusion.

I knew as soon as Vodien broke my database AGAIN for the second time in three months that I was going to have to finally move my blog. This was reinforced over the next few weeks by being impossible to get anyone who actually cared or had the capability to look at the issue and fix it. No matter how much pleading and yelling I did I just couldn't get anywhere. It made me seriously depressed for weeks. I was also upset that the backups of the database I'd been dutifully taking weren't necessarily going to work properly.

On the flip side, the support and responsiveness of CloudLoop (Karthick) was utterly outstanding. They definitely deserve their five star reviews.

In the end my blog is a lot happier. The holiday sites are consistent in design and everything works properly in HTTPS. The culling of the assets out of the database means my entries publish in seconds. Every new year I'll clear out the previous year's assets to keep it trimmed down.

Let's hope I don't have to write another one of these posts any time soon...