Automatic Database Updates

Today (technically), I implemented a set of scripts that will be called at regular intervals in order to keep the MySQL backend to the “interactive alch table” up-to-date!

Originally, the plan was to write a pure C program to do this, and automate it through cron. Unfortunately, preliminary research showed it would in fact have to be written in C++, due to the fact the official MySQL C++ driver is written in, well, C++ (indeed, as is MySQL itself). Difficulty compiling C/C++ JSON parsing libraries – or even finding ones that would compile – simply added to the frustration.

So, instead, I wrote two PHP scripts with limited dependencies and purely ephemeral resources, which will reduce read/write time dramatically. These are automated through a 4-line shell script executed every day at 2000 EST, or 0000 GMT, which is when the master database from which we draw is actually updated!

Additionally, it would appear that (embarrassingly) php5-curl was not installed on my server…! What? How did I miss that? How did anyone miss that?

Remember: debug output is your friend!

Next: updating the VSC landing page to reflect these changes, and updating the table interface to be more, well, interactive. 😉

Back Online, Almost


First, I’d like to thank all of you who brought to my attention (over the last month or so) the problems experienced by the site. First it was the forums, and then it was the databases, and then all of a sudden the entire system was inaccessible.

I talked with an Amazon CS rep briefly; he pointed at the consistently maxed-out CPU usage as the culprit, which makes sense enough, so as of now the entirety of is running on a single 2,66GHz Xeon E5430 with 1,5GB of RAM. This is not very much computing power compared to what’s available, but consider that 90% of the site’s content is handled on a per-request basis (that is, there is no overhead) and of that amount, about half of it is data-driven and dynamically generated (read: PHP scripts, most of which rely entirely on live MySQL queries). So, we’re fine.

Note: The mail server ( is currently broken. Not offline, but broken! I messed up several crucial steps of configuration (that seems to be the case, anyway) and now have no access to any mailboxes. The webmail is completely messed up and unusable, but local mail is still functional. To the best of my knowledge, PostFix is still sending/receiving mail as normal, but it is NOT likely to be accepted by the recipient since it will probably appear to be coming from “root@localhost,” which is pretty sketchy looking.

The forums do work, but nobody posts on them for some reason. I rolled back the MySQL backend to about 19 March, which deleted 850+ users and over 4000 posts (all spam, if you didn’t already guess). User registration e-mails now get dumped to an e-mail account I will rarely check. Your best bet is to contact me and I will pre-register (or completely register) an account FOR you.

User registration and profiles also work. This has actually been the case for awhile, and the only updates I have made this past week were cosmetic re-touches or functionality tweaks. Currently, there is no way to fill out your own profile. Working on it. Sorry.

The interactive alch table has been drastically overhauled, but again, mostly cosmetically. More information is now displayed, and there is now (finally) the option to view adjusted profits should you be buying nature runes or pure essence. In the top left corner, you will see the database version in use. This is accurate. It is not, however, always actually the most recent database. This is part of a new automation overhaul I’m working on with Ivan and Nik…theoretically. There will be more posts on this in the future.

I will possibly be moving the webmail to another server altogether, as I can simply have the domain “” direct to another IP (currently, I have two allocated, but am using only one). The MX record exists for “,” not the IP itself. That should be pretty self-explanatory. Why pay to maintain records associated with resources that are inherently ephemeral?

One last thing  – currently, our TCP I/O is pretty normal, and we transfer less than the limit imposed. I have no objections to the minimal fees associated with additional GB transfers…but I do have a problem with storage! Just yesterday, I deleted 52 GB of EBS volumes that were either unused, unnamed, or both! This costs me money, and it’s stupid to keep them around – especially when you consider that the majority of our old volumes are vestigial installs and outdated BOINC deployments!

I’m currently working on pushing regular (hourly) database backups to S3 instead of Dropbox, since our automatic database backup repo there is growing at the rate of about 75MB/day…which will fill up Dropbox much more easily than it will S3; S3 can in turn be fairly easily migrated to Glacier, which I have plans to begin utilizing anyway.

All this will be discussed in greater detail down the road.

Just wanted to let you know that we are back (and have been for about a week, sorry). If you notice any glaring (or minor) errors or bugs anywhere on the site, please e-mail nullcoding at live dot com with somewhere in the subject line. I have yet to set up (or, indeed, fully design or begin developing) a bug-reporting system, which will come in handy in the near future.

Til next time. 🙂