It’s true, goiardi now has a new website to call home. This Github Pages site will stay up indefinitely, since it’s not hurting anything by being here, but future goiardi updates will be found at http://goiardi.gl. It also features a new (but probably temporary) logo for goiardi.

I switched the goiardi site from using Octopress to Hugo, a static site generator written in Go. It’s pretty nice to use, and it’s fast.

I’m going to spend some time sprucing the new site up some and adding content where appropriate, but it’s at a good place to roll out. Thanks, and take a look at it if you feel the yen.

This is a minor release fixing a couple of small issues that popped up that seemed important enough that they probably shouldn’t wait until 1.0.0 comes out.

Goiardi version 0.8.2 fixes a couple of typos, one with checking errors when checking for the existence of a client in SQL mode, and one for the option to use postgres in the sample config file. The use-postgresql option was documented correctly elsewhere, but was listed incorrectly as use-postgres in the sample config file. This release also adds some checks to make sure the directory given for the local filestore exists, and is a directory.

Binaries are up on the release page, as usual.

In ancient Mesopotamia, diseases were often referred to as being the “hand” of some god. The hand of Nergal was a plague of some kind, while the hand of Ishtar was some kind of STD. Ishtar is generally identified with the Greek Aphrodite and Roman Venus as a goddess of love (although she was by no means identical to Aphrodite and Venus), so the name’s quite appropriate. The word “venereal” ultimately derives from “Venus”, after all.

These are two minor releases. Goiardi version 0.8.1 moves the bulk of the documentation from the README to readthedocs, where it should be much easier to read and work on, fixes a small issue with the postgres SQL dump file, and disables SSLv3. Schob 0.1.1 adds a fix to allow building schob on 32 bit platforms and on Windows, and updates links to the goiardi shovey documentation.

In honor of the upcoming Chef Summit, there are new releases from goiardi and friends. This officially brings out shovey, the goiardi push jobs facility, to the world.

Because of the relative complexity of shovey, there are actually three releases here:

• goiardi v0.8.0 – Hojotoho! Hojotoho!
• schob v0.1.0 – Heiaha! Heiaha!
• knife-shove v0.6.0 – Helmwige! Hier!

Astute readers may recognize the releases’ codenames as the opening lines of Act III of Die Walküre.

Shovey’s included with goiardi. Schob is the shovey client that gets installed on the nodes that will run jobs, and knife-shove is a knife plugin forked from knife-push to manage shovey jobs. The goiardi shovey docs, the schob README, and the knife-shove README have a bunch of documentation on how to use shovey.

Nice, functional cookbooks for shovey and goiardi are also coming. There is a shovey-jobs cookbook already, but it only supports Debian. Other platforms are being added, and will be ready soon.

Also coming reasonably soon are organizations and RBAC (it’s finally time), and revamping the documentation. The README has gotten too long and unweildy. It will be split apart, reorganized, and expanded where needed, while the godocs for goiardi will stop being a mirror of the README and focus more on development and internals.

See you at the Summit!

Goiardi 0.8.0 is in the final stages of release engineering before its official release. This, of course, is the release that introduces shovey, the goiardi facility for running jobs on client nodes without having to do a full chef-client run.

0.8.0-pre3 should be the last prerelease version of goiardi 0.8.0. It goes hand in hand with schob 0.1.0pre2, now with updated documentation, and knife-shove 0.6.0-pre1. The shovey-jobs cookbook for setting up shovey on a node is important too, although it currently only works on Debian systems.

Documentation for all the pieces of shovey are available. The shovey docs in goiardi give an overview, and there’s some shovey API documentation as well. There’s knife-shove documentation and schob (the shovey client) documentation available in their respective repositories as well.

Barring any complications, a formal release should be coming in the next couple of days.

Shovey still isn’t finished, but it’s come a long ways. To make it easier to play with and find potential issues, I’ve released a shovey-jobs cookbook to install and configure shovey on a node.

It’s a preliminary cookbook, but since shovey’s pretty preliminary still I’m OK with that. To use the shovey-jobs cookbook, you’ll need to set goiardi and the knife-shove plugin up as explained in this previous post on shovey first. Goiardi still only works with shovey in in-memory mode; the SQL support for shovey hasn’t been finished yet. After goiardi and serf on the goiardi server are set up, spin up a node and run the shovey-jobs cookbook on it, following the instructions in the shovey-jobs README. This cookbook has only been tested on Debian, but it may work with Ubuntu as well. At the moment it’s unlikely to work with RHEL and its derivatives or other operating systems, mostly because the init script is Debian specific.

This cookbook depends on having the serf and golang cookbooks uploaded and in the node’s run list. Serf may need some extra configuration to work correctly; I had to add this to the test node’s attributes for it to work right.

1
2
3
4
5
6
7

"serf" => {
    "agent" => { 
      "node_name" => "goiardi-test.local",
      "start_join" => [ "10.250.55.108" ],
      "advertise" => "10.34.10.15"
    }
  }

The main serf points are that the serf node’s name needs to be the same as the chef client name on the node, it needs to join the same serf cluster as goiardi’s serf agent is running on, and it needs to advertise the correct address. The last one came up in testing for me because goiardi was running on my dev box, while the shovey node’s serf was advertising the internal vagrant address rather than the shared network address. That should be enough to get you going running commands with shovey. See the knife-shove docs for possible commands there.

On another note, there’s some documentation now for the shovey API. The new thing there is how the shovey client now streams job output back to the server, and the server can in turn stream it to a client. Unfortunately the knife-shove plugin does not yet allow you to watch the job output stream by, but it’s on the list of things to do before the formal release.

The long promised “shovey” jobs for goiardi, a serf-based way to push commands out to nodes, is finally ready for a preview. There’s still some loose ends to pull together, like saving job status on the client so if it’s interrupted it can report that fact back to the server and making the goiardi end work with MySQL and Postgres, but it’s ready to play with.

To try playing with it, you’ll need schob, the shovey client, and knife-shove, the knife plugin to manage shovey jobs.

As explained in the schob README:

1. Checkout the “serfing” branch from goiardi
2. Create an RSA keypair for signing shovey requests
3. Install serf and run it as “serf agent”
4. Run goiardi in in-memory mode (at this writing, MySQL and Postgres in goiardi don’t support shovey yet) with these extra options: --use-serf --use-shovey --sign-priv-key=/path/to/private.key --sign-pub-key=/path/to/public.key
5. Install knife-shove from https://github.com/ctdk/knife-shove
6. Set up the node in goiardi you want to test shovey on. I’ve been using the computer I do my usual goiardi dev work for most of it.
7. Install schob the usual go way
8. Note where the testing whitelist file is (probably somewhere like ~/go/src/github.com/ctdk/schob/test/whitelist.json).
9. Run schob like so: schob -VVVV -e http://chef-server.local:4545 -n node-name.local -k /path/to/node.key -w /path/to/schob/test/whitelist.json -p /path/to/public.key --serf-addr=127.0.0.1:7373

Run schob with -h to see all current options.

Cookbooks for setting this all up and better documentation are coming, including documenting the HTTP and serf APIs for communicating with the server and client. The HTTP API is broadly similar to the Chef Pushy API, but not identical. Everything about shovey is still subject to change until it’s officially released, but it’s ready to play with now. Suggestions and comments are, of course, welcome.

Goiardi version 0.7.2 is a minor yak shaving release that there isn’t a lot to say about. The CHANGELOG pretty much sums it up:

• Remove a newline in a debug statement, courtesy of @spheromak.
• Also per @spheromak’s suggestion, fixed some possible race conditions revealed by building goiardi with the -race flag and running chef-pedant against it.
• Edit doc.go slightly to make godocs more attractive.

Elsewhere, shovey work continues and comes closer to completion.

The media refuses to report on this scourge afflicting the Florida coast, which is why you never hear about it.

“Shovey” is chugging along on a fast pace, but in the meantime a couple of issues came up that I thought deserved attention. Goiardi version 0.7.1 is a minor release, addressing two relatively small issues.

First, this release adds options for configuring the db pool size and max number of connections. It was happening automagically before, but I was always a bit uneasy about that. Until last night, though, I kept missing how to set that in the documentation for database/sql.

Second, for cookbook files stored locally (which is currently all of them), goiardi will now set the URL of the resource to use the current configured hostname. This was brought to my attention because someone had uploaded a bunch of cookbooks, changed the hostname and port and turned on SSL, and had the URLs break. It takes a specific chain of events to see the problem, which is why it had gone unnoticed up to this point.

From the CHANGELOG:

• Add —db-pool-size and —max-connections options for configuring the number of idle db connections kept around and the maximum number of db connections to make to the server. It isn’t particularly useful if you’re not using one of the SQL backends.
• For locally stored cookbook files (which is currently all of them), goiardi now generates the URL to the resource from the currently configured hostname. This fixes an issue where if you uploaded a cookbook and then changed the goairdi server’s hostname, the URLs to download cookbooks would break.

Other than that, there’s nothing else new in this release.

In case you haven’t read it, Orphans of the Sky is a novel by Robert A. Heinlein about a generation ship floating through space where the people on board have forgotten that they are on a space ship. It’s pretty good, but more importantly the name of the first of the two novellas that make up the novel is titled “Universe”. The title of the second part, “Common Sense”, became relevant after I really dug into making this release for reasons that will soon become clear.

While it ended up being a bigger deal than I expected it to be, I am pleased to announce the release of goiardi version 0.7.0.

I took what I thought would be a brief detour from my current work on the serf integration and “shovey” stuff for goiardi to implement Chef RFC 014 real quick like, which I had been considering doing even before the RFC came along. I had been planning on making this a small release numbered 0.6.1, but then after I proposed a mechanism for caching the universe endpoint it was suggested in the RFC discussion that I see how goiardi performed with the full load of cookbooks in the supermarket before jumping the gun and adding caching before it was necessarily needed.

It turned out that will over 6,000 cookbooks the universe endpoint ran like a dog. Ultimately the problem boiled down to using gob to encode the complex data structures in cookbooks, nodes, etc. in the database. Generally gob encoding in golang is faster than using JSON, but in this particular case JSON is actually faster. The “common sense” thing to do was to change how those complex structures are stored in the database, even though it’s a bit of a brutal change. Sometimes, though, you have to bite the bullet and do the thing that sucks. It’s ended up working nicely, however; the postgres implementation is now able to make use of the json type and the json functions built into Postgres. Before switching from gob to json, serving /universe with over 6200 cookbooks took over 3 seconds with Postgres or MySQL, but afterwards Postgres can serve it in roughly 325 milliseconds. MySQL isn’t able to take advantage of the Postgres-specific json functionality, but it still went down to 1 second, so that’s a marked improvement. Using the in-memory (with or without file backing) takes about 1.2 seconds. I’m willing to take this; if you have that many cookbooks and can’t wait a second for /universe, you may want to use Postgres for your data store.

From the CHANGELOG:

• Add /universe API endpoint, per https://github.com/opscode/chef-rfc/blob/master/rfc014-universe-endpoint.md.
• Make file uploading a little more forgiving.
• Make validating some cookbook metadata more forgiving, to bring goiardi’s validations in line with erchef.
• Added some functions to make listing all cookbooks and recipes on the server faster and move the logic into the cookbook package.

• Breaking DB change: with both MySQL and Postgres, the way data structures for cookbooks, nodes, etc. has changed from gob encoding to using JSON. This obviously breaks existing items in the database, so the following steps must be followed by users using either SQL backend for data storage:

  • Export their goiardi server’s data with the -x flag.
  • Either revert all changes to the db with sqitch, then redeploy, or drop the database manually and recreate it from either the sqitch patches or the full table dump of the release (provided starting with 0.7.0)
  • Reload the goiardi data with the -m flag. See the README or the godocs for more information.

More documentation on the universe endpoint in goiardi is available in the README. Specific information on upgrading is also in the README in the UPGRADING section.

Back to the serf/shovey stuff, then.