Thursday, February 20, 2014

Fixing "MySQL server has gone away" Errors in C

I ran across an old question on Stack Overflow the other day in which a user was having issues maintaining his connection to MySQL from C. I left a brief answer there for anyone else who might stumble across the same problem in the future, but I felt it was worth expanding on a bit more.

The error "MySQL server has gone away" means the client's connection to the MySQL server was lost. This could be because of many reasons; perhaps MySQL isn't running, perhaps there's network problems, or perhaps there was no activity after a certain amount of time and the server closed the connection. Detailed information on the error is available in the MySQL documentation.

It's possible for the client to attempt to re-connect to the server when it's "gone away" although it won't try to by default. To enable the reconnecting behavior, you need to set the MYSQL_OPT_RECONNECT option to 1 using the mysql_options() function. It should be set after mysql_init() is called and before calling mysql_real_connect(). This should solve the problem if the connection was closed by the server because of a time-out.

The MySQL documentation that discusses the reconnect behavior points out that only one re-connect attempt will be made, which means the query can still fail if the server is stopped or inaccessible. I ran across this problem myself while writing a daemon in C that would periodically pull data from MySQL. The daemon was polling at set intervals far less than the time-out period, so any such errors were the result of an unreachable or stopped server. I simply jumped execution to just prior to my work loop's sleep() call and the daemon would periodically try to re-connect until the server came back up.

#define DBHOSTNAME localhost
#define DBHOSTNAME dbuser

MYSQL *db = mysql_init(NULL);
if (db == NULL) {
    fprintf(stderr, "Insufficient memory to allocate MYSQL object.");

/* enable re-connect behavior */
my_bool reconnect = 1;
int success = mysql_options(db, MYSQL_OPT_RECONNECT, &reconnect);
assert(success == 0);

    0, NULL, 0) == NULL) {
    fprintf(stderr, "Connection attempt failed: %s\n", mysql_error(db));

for (;;) {
    success = mysql_query(db, "<MYSQL QUERY HERE>");
    if (success != 0) {
        /* The error is most likely "gone away" since the query is
         * hard-coded, doesn't return much data, and the result is
         * managed properly. */
        fprintf(stderr, "Unable to query: %s\n", mysql_error(db));
        goto SLEEP;

    /* call mysql_use_result() and do something with data */


Thursday, February 13, 2014

Generating C Code and Compiling from STDIN

Lately I've been exploring some syslog configurations and needed to generate some log messages to verify they were routed correctly. Of course doing so programmatically would provide an easy and repeatable method to generate a batch of fresh log messages whenever I needed, but because of the number of facilities and priorities defined by the syslog protocol, it made sense to write a code generator to iterate the different permutations.

The following Lua script generates boilerplate C code for each of the 64 messages needed to test LOG_LOCAL 0-7 with all priorities. I chose generating the code in this manner over writing a nested facilities/priorities loop directly in C so I could easily include a textual representation of the facility and priority constants in the log message (this seemed like a cleaner solution to me than having to maintain a mapping of constants to char* strings as well). And why Lua? Well, it seemed a better idea than M4. :)

#! /usr/bin/env lua

local facilities = {

local priorities = {

#include <stdlib.h>
#include <syslog.h>
#include <libgen.h>

int main(int argc, char *argv[])
    char *appName = basename(argv[0]);

for _, facility in pairs(facilities) do 
    for _, priority in pairs(priorities) do
    openlog(appName, LOG_CONS|LOG_NDELAY|LOG_PID, %s);
    syslog(%s, "Test %s.%s message.\n");
            facility, priority, facility, priority

    return EXIT_SUCCESS;

Running the script will output the desired C code, which looks like this:

#include <stdlib.h>
#include <syslog.h>
#include <libgen.h>

int main(int argc, char *argv[])
    char *appName = basename(argv[0]);

    openlog(appName, LOG_CONS|LOG_NDELAY|LOG_PID, LOG_LOCAL0);
    syslog(LOG_DEBUG, "Test LOG_DEBUG message.\n");

    openlog(appName, LOG_CONS|LOG_NDELAY|LOG_PID, LOG_LOCAL0);
    syslog(LOG_INFO, "Test LOG_INFO message.\n");

    openlog(appName, LOG_CONS|LOG_NDELAY|LOG_PID, LOG_LOCAL0);
    syslog(LOG_NOTICE, "Test LOG_NOTICE message.\n");

If I wanted to inspect or tweak the generated code, I could pipe the script's output to a file before compiling it:

./gen-syslog-tests.lua > syslog-tests.c
gcc -o syslog-tests syslog-tests.c

But if I just wanted the compiled binary and had no need to modify the code, it seems inelegant to write things out to a file. Here's where I learned it's possible for gcc to compile code piped in on STDIN.

./gen-syslog-tests.lua | gcc -o syslog-tests -xc -

The two things of note are: gcc can't deduce the programming language from the file extension (since there is no file) so the -x flag is necessary to identify the language, and - is used as the file name (a convention commonly used to indicate reading from STDIN as a file).

Monday, December 16, 2013

Esperanto Accented Characters in Windows

It's not as easy to set up as clicking a checkbox like Ubuntu/Gnome, but it is possible to type proper Esperanto characters in Windows using Right Alt as a modifier key. You need to create and install an alternate keyboard layout and then set the new layout active.

The program Keyboard Layout Creator is used to create the layout, and is available for free from Microsoft. Once it's downloaded and installed, start the program. Navigate File > Load Existing Keyboard and then select your primary keyboard layout (standard US layout in my case). You'll use this as a base and augment it with the Esperanto characters.

For each key that will should an accented character, right-click its position on the virtual keyboard and click "Properties for VK_? in all shift states". A dialog will appear in which the necessary Unicode code points can be entered.

The code points for the accented Esperanto letters are shown below, as well as for the Euro and Spesmilo just for fun:

If you don't want to enter the Unicode values yourself, feel free to use a copy of my keyboard definition file.

When you're finished setting the code points for each letter, navigate Project > Test Keyboard Layout to test them. Then, navigate Project > Properties to provide the necessary name and other descriptive information for the new layout. The name cannot be longer than eight characters, so I simply named mine "EO".

Once you're satisfied with the layout, navigate Project > Build DLL and Setup Package. The keyboard layout will be compiled to a binary format usable by Windows and be saved to your hard drive. Run the setup.exe installer that was written to disk install the layout. The installer will detect your system's architecture and launch the appropriate sub-installer.

Restart your computer once the installer is finished. You'll then be able to toggle between your original layout and the Esperanto layout using the Language Bar.

I set the augmented layout as my default keyboard layout (although I don't recommend this unless you're computer savvy). To do this on Windows 7, go to the Start menu, type "language" in the search bar, and select "Change keyboard and input methods". Click the "Change keyboards" button and you'll see the Text Services and Input Languages dialog. Under the General tab, set the new layout as the default input language and remove the entry for your original layout in the installed services tree.

On Windows 8, start typing "language" on the Start screen and then select "Change input methods" from the Settings group.

The Windows 8 Language panel more or less provides the same functionality as its Windows 7 counterpart but with a less user-friendly manner. The Input method is accessible through the options link.

Friday, November 29, 2013

Password Woes

Happy belated International Change-Your-Password Week! Earlier this month, thanks to the generous sponsorship by the great folks at Adobe, people all around the world were changing their passwords and tech blogs were parroting guidelines for choosing a strong password. But let’s be honest – passwords are a hassle. And, as Adobe was so kind to remind us, even the strongest unique password can be an open door if the company storing it isn’t doing so competently.

As someone who is a programmer, I’m aware of several technical solutions to our password woes. As someone who suffers from cynical realism, I believe the barrier to adopting these solutions to be red-tape and human nature (ego and laziness). There’s no reason for every website to require their own login credentials when OpenID and OAuth exist. Perhaps we should increase liability for password storers and provide incentives to the crackers who hack them. A smart company would migrate to an SSO-provider to mitigate their responsibility and the provider would be diligent in protecting the hashes.

But as much as anyone would like to mitigate responsibility, the fact remains that it’s the individual who’s most affected by password breeches, not corporations. Are there secure ways to ease the burden of password management?

I’ve been trying out KeePass this past week and my overall impression of the program is fair to middling. I’m storing the encrypted password database to Dropbox for the computers I use the most, and keep a duplicate copy of the database on a thumbdrive with a portable version of KeePass for when I need to use someone else’s computer. Although the premise seems secure, and I trust their implementation to be solid, some of the program’s incidentals frustrate me.

KeePass is fine on Windows but almost unusable on Linux. Unfortunately in this case, a good 90% of my day is spent using Linux. I've also noticed that the Auto-Fill feature toggles back to the most recently used window, so if an IM dialog pops up while I'm toggling to KeePass, the password is leaked. I could spend some time scripting in the advanced sections to safe guard against this, but that seems like a hassle.

I’ve also pondered the idea, so long as it contained accented characters, whether I might be able to get away with using the same password for everything. If the website is using proper encryption practices (Blowfish with scalable cost – i.e. Bcrypt – and random salt) then a rainbow table attack is going to be useless. Those sites that aren't have already proven their incompetence, so they probably don't know how to handle UTF-8 correctly either. The password value would be corrupted, truncated, or filtered, and most likely result in differing hashes between different sites... almost like using the site’s algorithm as your own salt! And brute-force crackers probably aren’t using Esperanto dictionaries; “@D0B3.fuŝ1s!” seems secure, doesn’t it?

Ultimately, programs like KeePass only serve as a bandage and don’t address the core problem, and ubiquitous use of SSO-providers is still a pipe-dream. While we’re all stuck in Password Hell, waiting for the next password-change holiday, the best we can do is keep Clifford Stoll’s advice in mind: “Treat your password like your toothbrush. Don't let anybody else use it, and get a new one every six months.”

Tuesday, September 10, 2013

Urba Semajnfino: Sirakuso a Success

The following is an English translation of an article I wrote for La Ondo de Esperanto to share the Urban Weekend: Syracuse event. Thank you to everyone who attended and helped make the event a success.

Urban Weekend: Syracuse, the third Urban Weekend event to happen in the United States, took place during the weekend of August 31 in Syracuse, New York. Esperantists came from near and far to meet new friends and explore the city. As the main organizer, I was a bit nervous. I had never organized an Esperanto event before. Would the weather hold out? Would anyone come? Would they enjoy their time together? But indeed the weather was beautiful, and people came from Rochester NY, Virginia, and even Brazil. Everyone had fun and Urban Weekend: Syracuse was a success!

A little before noon on Saturday, four of us met the city's central park and then walked to a nearby restaurant for lunch. The restaurant is popular for its beer, brewed on-site, and also for its support of Central New York agriculture by using locally-grown ingredients.

After lunch we walked about in the city for a bit and made our way to two museums. The first, the Erie Canal Museum, remembers the Erie Canal which connected Lake Erie to the Hudson River, and there we met two more esperantists. The canal no longer exists in its current form, but it has historical significance to both the region and the United States because it opened the Great Lakes to the Atlantic Ocean and enabled westward migration. Everyone enjoyed learning how the canal helped shape the country and seeing how life was like for those who travelled it almost 200 years ago.

The second museum, the Everson Museum of Art, is an art gallery known for its ceramics, pottery, and film exhibits. The collection may not be as impressive as the ones found in larger museums, but it has its several pieces worth enjoying. And perhaps even more special, the museum building was designed by the internationally acclaimed architect IM Pei who also designed the Pyramide du Louvre in Paris.

After exploring some of the art and history of Syracuse, we were hungry and were ready to eat. The six of us went to a Mexican restaurant occupying a former church building. Even this building had significance; the church was a station in the Underground Railroad in the 19th century. A secret tunnel under the church was a refuge for slaves running north in search of their freedom.

To finish the first day, we socialized and watched a film - House of Ghosts, a comical horror film dubbed with Esperanto voice and subtitles.

Most of the day Sunday was spent visiting the zoo, home to over 700 animals. A family of five esperantists who couldn't attend the first day joined us. The children in the group loved looking at the elephants, penguins, and lions. It was also a good opportunity for the adults to improve their animal-related vocabulary.

We ate lunch after the zoo in a near-by popular Irish restaurant; the food was great, and there were some local musicians playing in the pub that we enjoyed. The neighborhood where the restaurant is located was settled by Irish immigrants who came to work on the Erie Canal, and near the restaurant is the famous “green on top” traffic light. As the story goes, the settlers wouldn't allow red (the color of the British) to sit above green, and they threw tones at the light in protest anytime the city tried to hang the light correctly.

Weekend events similar to Urban Weekend are good for busy esperantists who are not able to attend the longer major events, and like all Esperanto gatherings, is a good opportunity to meet new friends, explore new places, and take part in Esperantujo. If one is held near you, I highly recommend that you participate. If not, why not organize your own? It's easier than you might think (I speak from experience!). The Manlibro pri Urba Semajnfino is a good place to start.

Friday, June 21, 2013

Building an Array with array_reduce?

Whether it’s twisting a function or taking advantage of side effects and flexible language constructs, it’s no secret I occasionally take joy in writing bastard PHP code (of course I’m a responsible developer and such snippets don’t make it into production). The other day I was in an “evil” mood and did something that might strike most of you as utterly silly; I used array_reduce() to construct an array.

If you’re not familiar with array_reduce(), it’s a function that iteratively reduces a given array to a single value using a callback function. For example, let’s suppose the function array_sum() didn’t exist. We could achieve the desired functionality using array_reduce() like so:

$nums = [1, 2, 3, 4, 5];
$sum = array_reduce(
    function ($acc, $val) { return $acc + $val; },

array_reduce() executes the callback function for each element in the array, passing to it an accumulator and the current array member. The returned value is used as the accumulator value for the next iteration. This is roughly the functional equivalent of this iterative approach:

$acc = 0;
foreach ($nums as $val) {
    $acc = $acc + $val;

All and all this is pretty straightforward. array_reduce() is nothing more than a mechanism that iterates over a list with an available accumulator. But what happens when you realize that nothing mandates the single result value must be a scalar? For functional programmers, this is obvious. For most PHP programmers with a procedural or OO background, this is potentially a jaw-dropping realization.

Suppose we need to build an array using data from the Unicode Consortium’s supplemental windowsZones.xml file. To make things interesting, the keys need to be the values of the mapZones’ type properties, and the members need to be the value in the leading comments. That is, the resulting array must look like this:

Array (
    ["Etc/GMT+12"] => "(UTC-12:00) International Date Line West",
    ["Etc/GMT+11"] => "(UTC-11:00) Coordinated Universal Time-11",
    ["Pacific/Pago_Pago"] => "(UTC-11:00) Coordinated Universal Time-11",
    ["Pacific/Niue"] => "(UTC-11:00) Coordinated Universal Time-11",
    ["Pacific/Midway"] => "(UTC-11:00) Coordinated Universal Time-11",
    ["Pacific/Honolulu"] => "(UTC-10:00) Hawaii",

The standard XML-processing strategies become cumbersome because of the requirement pertaining to the comment values, and the next best solution is to script a rudimentary stateful parser. We can iterate each line and extract the textual value if it’s a comment, or extract the attribute value if it’s a mapZone element, and assign to the array when we have both pieces of information available.

$comValue = '';
$zones = array_reduce(
    function ($acc, $line) use (&$comValue) {
        $line = trim($line);
        if (strpos($line, '<!-- (') !== false) {
            $comValue = trim($line, '<!-> ');
        elseif ($pos = strpos($line, 'type="')) {
            $typeValues = substr($line, $pos + 6, -3);
            foreach (explode(' ', $typeValues) as $value) {
                $acc[$value] = $comValue;
        return $acc;

Don't bother benchmarking it; this approach is going to be an order of magnitude slower than an iterative foreach loop that builds up the $zones array directly because the PHP run-time just isn't optimized for abuse like this. We’d gain speed with foreach, but then we’d miss an opportunity to explore how things work, blend different concepts together, and just have fun. Ultimately it's little excursions like this that help one grow and become a better programmer.

Monday, May 27, 2013

Composing Music with PHP

I'm not an expert on probability theory, artificial intelligence and machine learning, and even my Music 201 class from years ago has been long forgotten. But if you'll indulge me for the next 10 minutes, I think you'll find that even just a little knowledge can yield impressive results if creatively woven together into an application. I'd like to share with you how PHP can be taught to compose music.

Here's an example:

generated melody

You're looking at a melody generated by PHP. Sure it's not the most memorable melody, but it's not unpleasant either. And surprisingly, the code to generate the sequence of notes is relatively brief.

So what's going on? In short, musical data is analyzed by the script to "learn" which intervals make up pleasing melodies, and then the script creates a new composition by selecting pitches based on what it knows. Speaking technically, the script calculates a probability map of melodic intervals and applies a Markov process to generate a new sequence.

Standing on Shoulders

Music composition does not happen in a vacuum. Bach was fond of Buxtehude and Vivaldi; Chopin influenced Lizt and Wagner; Mozart and Hayden instructed Beethoven. Even the same melodic phrase can be found in different pieces of work. Orfeo ed Euridice by Gluck and the hymn tune Non Dignus for example both share a common phrase.

similar melody in Orfeo ed Euridice by Gluck and hymn tune Non Dignus

If you ask PHP to compose blindly, the results aren't pretty. To prove this point, here's a melody generated by mapping random values returned by successive calls to rand() to notes on a staff.

random melody

Unless you're keen on twelve-tone, it's better to draw from earlier compositions for inspiration and understanding.

To do just this, I first transcribed the melody of several pieces of music using Scientific Pitch Notation. I didn't concern myself with note durations, instead focusing on the pitches themselves. A middle C on paper was entered as 'C4' (C is the note name, 4 is the octave), a semitone above that was 'C#4', the next semitone 'D4', and so on, until a melody (the first 8 measures of Tantum Ergo by Bottazzo shown here) was encoded like:

first 8 measures of Tantum Ergo by Bottazzo

A4 C5 G4 A4 G4 A#4 D5 A4 B4 A4 C5 D5 C5 G4 B4 B4 C5

I now had a sequence that was easily parsable and on which I could perform some basic analysis. For example, given an instance of A4, what is the next probable note to follow?

A4 C5 G4 A4 G4 A#4 D5 A4 B4 A4 C5 D5 C5 G4 B4 B4 C5


C5 G4 B4 C5

There is a 50% chance that the next note would be C5, a 25% chance it will be C4, and a 25% chance it will be B4.

This process translated warm flowing music into something the computer, understanding things only within the context of cold, unfeeling mathematics, could comprehend and reason about.

Paging Doctor Markov

We're familiar with deterministic systems – systems where the same input will always generate the same output. Addition is a deterministic function, with inputs 2 and 4 always yielding 6. Stochastic systems on the other hand behave with some level of randomness. Identical inputs can result in wildly different outputs, such as with the function array_rand(). There is an element of randomness in composition, or else all compositions starting on F4 would end up the same, making generations of composers irrelevant and filling the coffers of the RIAA. But the randomness is tempered, even if at a subconscious level, by the composer with the knowledge of what combinations are pleasing.

A prime example of a stochastic system, one which is also relevant to the composition script, is a Markov process (named after the mathematician Andrey Markov who not only studied them but also had an amazing beard). As Nick Didkovsky explains:

Markov analysis looks at a sequence of events, and analyzes the tendency of one event to be followed by another. Using this analysis, you can generate a new sequence of random but related events, which will look similar to the original. A Markov process is useful for analyzing dependent random events - that is, events whose likelihood depends on what happened last.

The traditional example used to illustrate the concept is a weather predicting graph. Suppose the day following a sunny day has a 90% chance of also being sunny, and the one following a rainy day has a 50% chance of being rainy. The graph would look like this:

Markov Sunny/Rainy graph

Depending on the mechanism used to resolve probabilities, walking the graph for 5 iterations we might find ourselves transitioning with Sunny the first day, Rainy the next, Sunny after that, then Sunny, and Sunny, or we might find ourselves transitioning Sunny, Sunny, Sunny, Rainy, Rainy another.

Hopefully it's obvious where I'm going with all of this; it's possible to constrain the random process of "next note selection" using the weighted probabilities learned by analyzing melodies for a better sounding result.

A4 graph

This simple process allows us to generate passable melodies in infinitely less time than it would take for a monkey hitting random keys on an organ to play the complete works of Messiaen.

Robot Composers (the singularity is near)

At this point you hopefully have a cursory understanding of the two key concepts I employed to generate music. Even if you don't, you've at least survived my humor and deserve to be rewarded by seeing some code.

namespace Zaemis;

class Composer
    private $pitchProb;

    public function __construct() {
        $this->pitchProb = [];

    public function train($noteData) {
       $numNotes = count($noteData);
       for ($i = 0; $i < $numNotes - 1; $i++) {
           $current = $noteData[$i];
           $next = $noteData[$i + 1];
           $this->pitchProb[$current][] = $next;

   public function compose($note, $numNotes) {
       $melody = [$note];
       while (--$numNotes) {
           $i = array_rand($this->pitchProb[$note], 1);
           $note = $this->pitchProb[$note][$i];
           $melody[] = $note;
       return $melody;

require_once '../include/Zaemis/Composer.php';
use Zaemis\Composer;

$noteData = trim(file_get_contents('../data.txt'));
$noteData = explode(' ', $noteData);

$c = new Composer();
$melody = $c->compose($_GET['note'], $_GET['count']);

echo '<img src="img/notes/clef.png" alt="Treble Clef">';
foreach ($melody as $note) {
    echo '<img src="img/notes/' . urlencode($note) . '.png" alt="' .
        $note . '">';

The learning process takes place in the train() method which accepts an array of training notes (the encoded melody string split on its spaces). This code is simple, quick, and dirty; the notes are pushed to a 2-dimensional array with their probabilities indirectly implied by the quantity of elements themselves.

When populated, the array would look similar to:

array(9) {
  ["A4"]=> array(13) {
    [0]=> string(2) "C5"
    [1]=> string(2) "G4"
    [2]=> string(3) "A#4"
    [3]=> string(2) "C5"
    [4]=> string(2) "B4"
    [5]=> string(3) "A#4"
    [6]=> string(2) "G4"
    [7]=> string(2) "A4"
    [8]=> string(2) "D5"
    [9]=> string(2) "G4"
    [10]=> string(2) "C5"
    [11]=> string(2) "C5"
    [12]=> string(2) "G4"
  ["C5"]=> array(11) {

Looking at the data, a randomly selected note to follow A4 has approximately a 31% chance of being C5 since 4 out of the 13 members of the list hold that value. (Yes, I know maintaining probabilities instead of a memory-exhausting list of all of the notes in a composition is the "right way", but I wanted to keep things simple for illustrative purposes.)

The compose() method encapsulates the logic to generate the melodic sequence. A starting note the desired length is given, and the method randomly selects a value for the following note from the array until the desired number of notes has been retrieved.

Of course we humans would rather see the result notated on a staff as opposed to a list of note values, so I created a set of note images to accompany the script. Each image displays a note on the appropriate position on a staff, and the files are named according to the note name. Looping through the melody to emit some IMG elements was an effective rendering method for my needs.

Harder, Better, Faster, Stronger

It is impressive that such simple concepts can be used to create a script capable of emulating a composer. Of course, there is infinitely more that can be done to build and improve. Consider this your first exploration into musical intelligence. David Cope, who has been exploring computer composition since 1981, has this to say:

Simply breaking a musical work into smaller parts and randomly combining them into new orders almost certainly produces gibberish. Effective recombination requires extensive musical analysis and very careful recombination to be effective at even an elemental level.

Beyond the obvious changes, such as changing the pitch matrix to maintain probabilities, how would you improve things? Maybe replace this naive approach with a completely different mechanism for analyzing music? Parse input from MIDI files? What would be needed to identify harmonies? How about chord progressions? Note durations? Composer "signatures"? Could your script learn from itself by analyzing and feeding pleasing melodies it produced back into its knowledge base? In what ways could you recombine samples to form new works?

I look forward to hearing about your own experiments in AI-driven composition in the comments below.

Update 6/10/13: I've tossed some code up on GitHub if anyone's interested.