The Order of the Elephant is a Danish order awarded mostly to foreign heads of state and royals. Its elephant-themed insignia is appropriate for this latest release of goiardi, version 0.6.0.

Along with some bug fixes and smaller improvements, this release introduces Postgres as a supported database backend. The schema is not compatible with erchef’s schema, but is close to the goiardi MySQL schema (with some differences of course). This is the first project I’ve done with Postgres, so comments on how goiardi is using it are welcome.

Full CHANGELOG:

• Postgres support.
• Fix rebuilding indexes with an SQL backend.
• Fix a bug where in MySQL mode events were being logged twice.
• Fix an annoying chef-pedant error with data bags.
• Event logging methods that are not allowed now return Method Not Allowed rather than Bad Request.
• Switch the logger to a fork that can be built and used with Windows that excludes syslog when building on Windows.
• Add basic syslog support.
• Authentication protocol version 1.2 now supported.
• Add a ‘status’ param to reporting, so a list of reports return by ‘knife runs’ can be narrowed by the status of the chef run (started, success, and failure).
• Fix an action at a distance problem with in-memory mode objects. If this behavior is still desirable (it seems to be slightly faster than the new way), it can be turned back on with the —use-unsafe-mem-store flag. This change DEFINITELY breaks in-mem data file compatibility. If upgrading, export your data, upgrade goiardi, and reload your data.
• Add several new searchable parameters for logged events.
• Add organization_id to all MySQL tables that might need it someday. Orgs are not used at all, so only the default value of 1 currently makes it to the database.
• Finally ran ‘go fmt’ on goiardi. It didn’t even mess up the long comment blocks, which was what I was afraid it would do. I also ran golint against goiardi and took its recommendations where it made sense, which was most areas except for some involving generated parser code, comments on GobEncode/Decode, commenting a bunch of identical functions on an interface in search, and a couple of cases involving make and slices. All in all, though, the reformatting, linting, and light refactoring has done it good.

To reiterate, this update breaks saved data file compatibility. You’ll need to export and re-import your data if you’re using the persistent in-memory data store.

A selection of precompiled binaries are provided on the release page, including Windows and Solaris builds. There are also knife plugins for goiardi’s reporting and event logging capabilities at knife-goiardi-reporting and knife-goiardi-event-log. The reporting plugin was forked from the official Chef knife-reporting plugin to add support for the “status” parameter. Both plugins are also available on rubygems.

Finally, after someone asked how goiardi would handle running ~500 nodes, I decided to take a look and see. I whipped up a little test script to create a thousand nodes and clients to see how goiardi (and more importantly the search) would handle it. I wasn’t particularly worried about the non-search portions of goiardi (1000 rows on a table isn’t very much at all, after all), but I hadn’t tested search extensively with large amounts of data. After creating those 1,000 nodes and clients, goiardi was using about 1.40 GB of RAM. An earlier test with 5,500 nodes (but no clients) used 6.45GB RAM. The numbers show that there’s definitely room for improvement with the search, in that the capability for indexing nodes and data bags is limited by RAM, but performance was good even at 5,500 nodes. More rigorous testing is needed, however. Search is an area I intend to revisit shortly to improve these problems.

The next issues to work on with goiardi are currently improving search to work even better with large amounts of data and use raft (or something similar) to allow multiple goiardi instances to keep their search indexes consistent, reworking the permissions system, being able to upload files to s3 or similar storage providers, and the serf based pushy type thing.

With the recent release of Go 1.3, Go now has experimental support for Solaris and its various derivatives. This means that goiardi will build and run on Solaris, although in an experimental way.

Unfortunately to get Go on Solaris/illumos/etc. you have to build it yourself from source, and at least when I was trying it out with OmniOS in vagrant it did not come with gcc by default. Getting go up so you can try out goiardi (if you don’t want to get the binary is pretty straightforward.

• First, install gcc. These instructions set you through installing gcc. It wasn’t clear to me at first, but you do have to run all the pkg install commands mentioned. First install the appropriate version of gcc, then install those other packages.

Once that’s done, make sure your PATH is set per the instructions on that page.

• Install git with pkg install developer/versioning/git

• Next, download go 1.3: wget http://golang.org/dl/go1.3.src.tar.gz and extract it somewhere. This will be your GOROOT. Set the environment variable with export GOROOT=<go's location> (assuming you’re using bash or similar).

The next steps come from *Installing Go from source:

• In this new GOROOT, cd src/, then run ./all.bash. It will build go for you. If it complains about being unable to find unistd.h, you were like me and forgot the second step on that Installing GCC page.

• You want to actually be able to use go, so add it to your path: export PATH=$PATH:$GOROOT/bin.

• Now you need a workspace. In your home directory, run:

1
2
3
4
5
6

mkdir gocode
mkdir gocode/src
mkdir gocode/bin
mkdir gocode/pkg
export GOPATH=$HOME/gocode
export PATH=$PATH:$GOPATH/bin

• Almost there. All that’s left is running

1
2

go get -t github.com/ctdk/goiardi
go install github.com/ctdk/goiardi

Assuming all went well, you can run goiardi at your prompt with whatever options you want, and it will happily run. After this point using goiardi is like using it on any other platform.

Between the previous goiardi release and this one, my kids discovered Minecraft. This means that I started playing Minecraft, which ended up slowing goiardi development down for a while.

This blocky adversary has been overcome, however, and goiardi version 0.5.2 is out.

Originally the next goiardi release was going to add Postgres support. This was about halfway done, and will be resumed shortly, but I decided that it was more important to add the ability to import and export data before any of the goiardi structs changed.

Changes in this release, from the CHANGELOG:

• Add import/export of goiardi data through a JSON dump.
• Add configuration options to specify the max sizes for objects uploaded to the filestore and for JSON requests from the client.

Importing and exporting is explained in the README and in the godocs. Basically you add the -x/--export flag to your usual goiardi command to export the data to a JSON file, and run the -m/--import flags with your new desired goiardi set of flags to import the data. The administrator is responsible for removing any existing data (on-disk or SQL) and index files. This can help with backups or moving between the different storage engines goiardi supports, keeping data between upgrades, and should make testing easier in the future.

Also, per the README, there are new options to specify the maximum size of an object that can be uploaded to the filestore, and the maximum size of a JSON request from a client. The -Q/--obj-max-size flag gets the filestore object max size, while -j/--json-req-max-size sets the JSON request max size limit.

Completely unrelated to any of my efforts, but while I was preparing this release Go 1.3 was released. Goiardi and its dependencies build just fine with Go 1.3 with no changes. This also means that goiardi should be able run on some of the stranger platforms out there, like Solaris, the illumos based distributions, plan9, and apparently Dragonfly BSD. So far goiardi has only been build on OmniOS r151008j, but it built fine.

Finally, as an added convenience: on the goiardi 0.5.2 release page, precompiled binaries are being provided for download, if you don’t feel like setting up go and compiling them yourself. As of this writing binaries are provided for Linux (built on Debian wheezy), MacOS X (built on 10.9 Mavericks), FreeBSD (built on FreeBSD 9.2), and illumos (built on OmniOS r151008j). The Linux, MacOS X, and FreeBSD binaries are x86_64, the illumos binary i386 update: I misinterpreted the output of psrinfo – it was compiled for 64 bit. It appears that there’s no reason that goiardi can’t easily be cross compiled though, so a more sensible set of precompiled binaries will be coming soon. If you don’t see binaries for your platform, try grabbing a binary that’s built on a similar platform to yours, or compile it yourself.

Coming down the pipeline is resuming and re-merging the Postgres work, making some data structures safer for in-memory mode, some extra features for event logging and reporting, improving tests and moving away from depending on chef-pedant, and an idea I’ve been kicking around for a serf-based pushy type thing. For those curious about the Postgres goiardi schema, you can see it in the ‘postgres’ tree in goiardi-schema. It’s firmed up, but still subject to change before getting merged back in.

There’s now a knife plugin to use with goiardi’s new event logging capabilities: knife-goiardi-event-log 0.1.1.

Usage info from the README:

There are four goiardi event log commands: knife goiardi gel list, knife goiardi gel show, knife goiardi gel delete and knife goiardi gel purge.

knife goiardi gel list returns a list of logged events. A limit may be specified with --limit; the default limit is 15. An offset may also be specified; the default offset is 0.

knife goiardi gel show [id] gives a detailed listing of a particular event log item.

knife goiardi gel delete [id] deletes the given event, and prints out a detailed display of the deleted event.

knife goiardi gel purge [id] purges all event log items with an id less than the given id. After purging the events, it displays the number of events that were purged.

Being able to list events of a certain kind of action, type of object, time range, etc. would be nice and are on the roadmap for this plugin. Adding that capability depends on changes to goairdi before that can happen though. This plugin makes use of all the current event log functionality however.

The sky darkens. Blotting out the sun, Rodan appears to destroy beloved landmarks and stomp on small models of tanks. Goiardi version 0.5.1 has been released.

This is an incremental release, made up of bug fixes and relatively minor new features. The most interesting new features are event logging and reporting, although the log level changes are also nice.

From the CHANGELOG:

• Add log levels (from debug to critical). This makes -V/—verbose useful.
• Add an easier option in the config file to specify log levels by name.
• ipv6 already worked, but accidentally. Now it works in a more deliberate fashion, preventing mishaps with addresses, colons, and port numbers.
• Authentication protocol version 1.1 now supported.
• Remove a sort on run lists that was there for some reason. I have no idea what it was put there for, but it was wrong.
• Add an event log to log changes to objects like nodes, clients, etc. See the README or godocs for details.
• Add support for reporting (http://docs.opscode.com/reporting.html)

This release should not break old save files, if you’re using that. If you’re using MySQL, you’ll need to deploy the sqitch changes.

The next big things to start working on are the long promised Postgres support, and etcd/serf/consul etc. integration. Groupcache has also been asked for and looks like a worthwhile thing to pick up. As always, the issues page is a good place to look at what’s being worked on.

Goiardi is also starting to grow beyond using chef-pedant for most of its testing; it’s about time to start more native golang tests for goiardi and not use chef-pedant for the main tests. It should always work with chef-pedant, of course, but goiardi’s starting to get more features that chef-pedant doesn’t check.

Stormily agitated, a new goiardi release appears! Goiardi version 0.5.0 has been released.

Aside from a few bug fixes, this release brings MySQL support. Now, in addition to running entirely in memory, goiardi can use MySQL for storing data. The README and godocs explain how to set that up.

From the CHANGELOG:

• MySQL support added
• No longer redirect /environments/NAME/roles/NAME to /roles/NAME/environments/NAME
• Update documentation, reformat godocs
• Split actors apart into separate user and client types, made new Actor interface that encompasses both users and clients.

Note: This release contains one breaking change. Saved data files from previous goiardi releases will not work with 0.5.0, so if you’re in that situation either don’t upgrade or remove your saved index and data files before starting up. This will regenerate your admin, validator, and webui keys, and of course nuke anything you had in place before.

Postgres support has already been requested by a few people, so that should be coming fairly soon. The issues page has been filled in with some enhancements that would be nice for goiardi to have. Feel free to add your own if there’s something you’d like to see.

It seems like only yesterday that there was a goiardi release, but really it was a few days ago. Goiardi version 0.4.2 is hot off the presses and ready for your perusal.

This is a bugfix release. The reason this is the “Spinal Tap” release was because I noticed these problems while I took the day off after getting a spinal tap and was bored, so it seemed fitting. The bug fixes, from the CHANGELOG, are:

• Perm tweak for nodes updating themselves.
• Small change with validating role descriptions when creating or updating from JSON.
• Fix issue with saving complicated indexed objects to disk where improperly flattened indexable objects were making the gob encoder puke all over itself when encoding the tries in the index docs.
• Fixed a possible regression with synchronizing cookbooks that did not show up in testing, but only in real use.
• An absolutely bonkers fix for listing cookbook files with webui. Webui wants all of the cookbook top level attributes sent over with a request to /cookbooks/<name>/<version>, but this is the exact opposite of the behavior chef-pedant wants, where empty definitions, attributes, etc. are not sent over. Knife also seems quite content with this, so the fix for now, since the two cases are mutually exclusionary, is to only send the empty hashes for those top level attributes with a GET if the request is coming from the webui. Bizarre, but it seems to be what’s necessary.

These were all problems that evaded notice during testing, but did pop up when I was using goiardi to set up a vagrant VM to start working on letting goiardi run a real database (optionally, of course) instead of the in-memory data store system it’s using now. The in-memory mode is great and all for testing, and being able to freeze the data to disk for persistence is very handy, but being able to use a real database would be nice for some workloads. Supporting both the current in-memory storage and a database backend is a top priority.

Which database to use is still a bit up in the air, but I’m leaning towards throwing caution to the wind and providing support for both MySQL and Postgres, with SQLite as a distinct possibility down the line. MySQL is likely to come first, but that is not set in stone.

Hot on the heels of the 0.4.0 release comes goiardi version 0.4.1. This release adds a —disable-webui option to disable using chef-webui with goiardi, fixes a bug with parsing configuration file options and rearranges how some of the config struct items are set, fixes a typo in the sample config file, and makes some small documentation tweaks.

Other than that, all the changes announced in the 0.4.0 release announcement and previous releases remain relevant.

A new version of goiardi, version 0.4.0, as been released.

This version fixes a bug with pessimistic matching, adds more tests, improves the documentation with an expanded README and more godocs, and introduces client/user authentication and authorization and SSL for connections as optional settings. It will also run and authenticate properly with the chef webui, if you’re so inclined. The webui is a separate install, however, and does not ship with goiardi.

There is also a fork of chef-pedant customized for goiardi now, to handle some tests with authentication where goiardi and the chef-server make slightly different decisions.

Goiardi still works fine in no-auth mode. The authentication and authorization parts are completely optional, but they’re there if you want or need authentication. It’s recommended that if you do use authentication that you should set the —index-file and —data-file options for persistent data and index storage, but it’s not strictly necessary. See the README for more on how to set up authentication.

SSL is pretty straightforward, and also explained in the README. You’ll need to generate a certificate/key pair for it to work, but that isn’t too hard. This does assume that you’d use a self-signed certificate for this; if you’re going to use a real cert I assume you know what you’re doing (and am a little surprised, to be honest).

That’s it for this release. The next major goiardi task coming up, barring any unexpected changes, is allowing it to use a database for the backend. This will be optional, of course; I plan on supporting the in-memory, no-auth mode a la chef-zero for goiardi indefinitely. Further out, there may be a smoother installation process, and someday even perhaps goiardi-specific features.

While there is still some work remaining with finishing some needed authorization checks, I’m happy to report that in the goiardi auth branch, authentication for chef clients and webui is working and passing the chef-pedant tests.

Once those authorization issues are ironed out, I’ll start work on updating the documentation and making a new formal release. If, in the meantime, anyone felt like testing it, it’s pretty easy. Start goiardi with the -A or –use-auth flags, and set –conf-root to whereever you’d like the chef-validator, chef-webui, and admin certificates to be stored. You’d probably also want to specify index and data files for persistence, but you don’t have to. Goiardi works with chef-webui fine as well, but at the moment you’ll have to set a password for the admin user with knife.

As of this writing goiardi authentication is still a work in progress, but it’s far enough along to start banging on to look for problems. The chef-pedant tests are pretty happy, but that doesn’t always catch every real world issue.