github.com/ctdk/goiardi

This announcement actually includes the info from both the 0.11.10 release and the 0.11.9 release as well. For a small variety of reasons, including the fact that these past two releases were both pretty small and because I've been cranking on the Chef Server 12 support lately, I never got around to drafting formal releases and binaries. At last, here they are. There's a fix to remove a dependency that was being removed from Debian, and two small changes to support newer versions of the chef tools.

From the CHANGELOG:

0.11.10
-------
* It was brought to my attention by @joerg that 'create_key' is now a valid JSON
  hash key for when creating clients with newer chef tools. Added it to the
  whitelist of valid elements for creating clients. Thanks again for bringing
  it to my attention.

0.11.9
------
* Follow chef RFC041. Newer chef-clients and knifes (knives?) were breaking
  goiardi wasn't following this RFC by reporting the supported api versions.
  (Thanks julian7 for the PR and bringing it to my attention.)
* Remove dependency on golang.org/x/exp/utf8string - it's failing some tests on
  the Debian build servers, and goiardi's liable to get kicked out of testing
  shortly if it isn't addressed. Since goiardi wasn't actually using utf8string
  for anything real complicated, it was easy enough to tear that out and whip
  up a replacement with built-in functions. (Thanks jordi for bringing this
  situation to my attention as well.)

cut a tiny 0.11.10 release

• It was brought to my attention by @joerg that 'create_key' is now a valid JSON hash key for when creating clients with newer chef tools. Added it to the whitelist of valid elements for creating clients. Thanks again for bringing it to my attention.

And commit the CHANGELOG update from last night - these commits were …

• Follow chef RFC041. Newer chef-clients and knifes (knives?) were breaking goiardi wasn't following this RFC by reporting the supported api versions. (Thanks julian7 for the PR and bringing it to my attention.)
• Remove dependency on golang.org/x/exp/utf8string - it's failing some tests on the Debian build servers, and goiardi's liable to get kicked out of testing shortly if it isn't addressed. Since goiardi wasn't actually using utf8string for anything real complicated, it was easy enough to tear that out and whip up a replacement with built-in functions. (Thanks jordi for bringing this situation to my attention as well.)

While there are other sticks in the fire, both goiardi related and otherwise, here's another release with tweaks and improvements to brighten the days of anyone who happens to use goiardi.

From the CHANGELOG:

* Made some small tweaks and updates to the depsolver to make that a little
  better. NB: Down the road, there may be some further changes to the depsolver,
  especially where 'most constrained' cookbooks are concerned.
* Update the circleci config to use the version 2.0 syntax.
* Add purging old sandboxes that have been hanging around for too long.

• Made some small tweaks and updates to the depsolver to make that a little better. NB: Down the road, there may be some further changes to the depsolver, especially where 'most constrained' cookbooks are concerned.
• Update the circleci config to use the version 2.0 syntax.
• Add purging old sandboxes that have been hanging around for too long.

Add purge-sandboxes-after option to sample config file, update help a…

• Allow access to /debug/pprof with a whitelist of IP addresses
• Properly index arrays of hashes, arrays of arrays, etc. in object attributes.
• Pretty serious memory usage improvements with search (both the in-memory and postgres searches).
• Fix reconnecting to serf if the connection is somehow interrupted.
• Fix negated range queries (it turns out they do have a use after all), and refactor how NOT queries are handled generally.
• Add options to purge old reports and node statuses.
• Add option to skip logging extended object information in the event log.
• A handful of other bugfixes.
• Bump up to using golang 1.9.3 for builds.
• Minor changes to the documentation.

0.11.6 (cancelled)

• Skipped because of a miscommunication snafu involving Debian packaging and a pre-release tag for 0.11.6.

• Several search fixes:
  • With postgres search:
    • Fixed reindexing after it broke with the previous update that eliminated a lot of unneeded extra rows in the database.
    • Fixed basic queries with NOT statements.
    • Separately, fixed using NOT with subqueries. On a somewhat complicated note, but in a way that appears to match standard Solr behavior, when doing a query like "name:chef* AND NOT (admin:true OR admin:bleh)" it works as is, but when a negated subquery is followed by another basic query statement, it needs to have extra parentheses around the NOT + subquery, like "name:chef* AND (NOT (admin:true OR admin:bleh)) AND public_key:*". A convoluted and unlikely scenario, but it could happen.
  • With in-memory search:
    • NOT + subqueries was also broken with the in-mem search. The fixes for the pg-search partially fixed it for in-mem in that it no longer made the server panic, but it was returning incorrect results. Additional work ended up being needed for in-mem search.

• Implement Chef authentication version 1.3.
• Move the custom goiardi error type out of util and into its own module. Wrappers around the new module are in util still for convenience, and because the functions and interface are used all over the place.
• Many endpoints now handle HEAD requests where appropriate. With some endpoints this is not especially useful, but with others it's a lightweight way to see what resources exist and so forth. Implements Chef RFC 090.
• Start using contexts with requests. This does mean that goiardi will require at least go 1.7. (As of 0.11.3 goiardi only supported go 1.7+, but it was likely to build with somewhat older versions anyway.)
• Minor bugfixes - deal with a possible race condition with the in-mem search index, change some logging statements from Info to Debug that didn't need to be Info level and removed a test log statement that was no longer necessary, updated copyright dates.
• Add the Chef API version header to responses.
• Change behavior if the data file and use-(mysql|postgresql) are specified together; formerly it was a fatal error, but now it'll just emit a warning in the error log and ignore the data file setting.

• Add an option to trim values in search indexes. Currently not enabled by default, but will be in the next minor goiardi release (so, either 0.12.0 or 1.0.0, depending on which ends up being next). Existing indexes ought to be reindexed upon upgrading, but they should still work if this is skipped.
• Fix a bug where duplicated items in slices in objects being indexed with the in-memory trie based index would cause goiardi to crash. For good measure, even though it isn't necessary to prevent a crash remove those same duplicate items from objects being indexed with the postgres index.
• Mark --use-unsafe-mem-store as deprecated. In the unlikely event someone's using that option, a warning will print in the log. This option may be removed at any time.
• Allow setting configuration options via environment variables. (See the documentation for the details.)
• Finally allow configuring MySQL or PostgreSQL connection options with command line flags (or, now, environment variables).
• Fixed format issues and wording in a few places in the documentation, along with updating the docs for the current version.
• Add a hidden flag to generate a simple man page.
• Add that simple man page, along with the html docs, to the packagecloud.io packages.
• Add a Dockerfile to allow running the local goiardi source in docker.
• Add Debian "stretch" and Ubuntu "yakkety yak" to the distro versions we have in the package repository.

• Fix a bug with escaped characters in certain searches (thanks ickymettle). Does require rebuilding the search index.
• Allow using 'novault' as a build tag to avoid having to have the vault api present when building goiardi. Not relevant to most people.

• Allow storing secrets (client & user public keys, shovey signing private keys, and user password hashes) in an external service. Currently only vault is supported.
• Rework reindexing to break it into smaller chunks and ensure that only one reindexing job can run at a time.
• Package goiardi for RHEL 7 and Debian jessie for s390x. Rather experimental, of course.

• Ability to upload cookbooks to S3.
• Add script to upload local files to S3 to migrate.
• Change how items are indexed with the postgres indexer, to reduce the number of rows in the search_items table substantially (at the cost of possible differences in search results in a few weird corner cases).
• Search parser no longer chokes on Unicode. Unfortunately Postgres' ltree module does not accept all Unicode alphanumeric characters as valid still.
• Use vendoring.
• Rejigger the package building process a bit - changing how the different packages are built and how version numbers are determined.
• Fix a long-standing annoyance where the log file would get truncated when goiardi started or restarted.
• Allow passing environment variables to goiardi through the config file.
• Fix in-memory indexer to work with go 1.7.
• Add packages for CentOS 6 and 7. Also use a gox fork pulling in someone's PR with better ARM support until that gets merged upstream eventually.
• Change the postgres columns using the 'json' data type to use 'jsonb' instead. This is generally better, but does mean that goiardi now requires PostgreSQL 9.4 or later.

• Export pprof info over HTTP, but only accept connections from localhost for that information.
• Add statsd metrics for things like chef-client run timings (requires reporting) and started/succeeded/failed, number of nodes, API endpoint timings, various pieces of runtime info like GC pauses, RAM used, and number of resources updated & total resources for client runs.
• Fix JSON decoding issue where very large numbers would suddenly turn into floats.

• Handle someone trying to use syslog on Windows ourselves, rather than letting the logging library do it (it was causing trouble with gox).

• Fix up packaging and deploy scripts a bit
• Add sql schemas to the deb
• Fixed a logic error when configuring the address to listen on where the value specified in the config file was always ignored, and only an address specified on the command line worked. (Thanks to jordi and DQEbert here for bringing this to my attention.)
• Added options to specify proxy hostname and port different than what goairdi itself is listening on. (Thanks to jordi and DQEbert here as well.)
• CoC
• Added Debian wheezy to the list of distros we generate packages for.
• The logging library goiardi used moved. It had been forked, but since the dependencies of said fork also moved, goiardi switched to the new version of that library. Happily the logger library had added logging to syslog as an option, so we just went back to using upstream at the new location. (Thanks to theckman for providing a fix for this.)
• In concert with the above, add a "fatal" log level.
• Terraform removed the depgraph module, so that's been vendored into goiardi along with its digraph dependency.

• Fix some tests
• Scripts, configuration files for more efficient packaging
• circleci integration
• Bomb on importing data if public keys don't validate. (thanks jordi and DQEbert for bringing this to my attention.)
• Validate older PKCS#1 keys -- golang's stdlib pukes on them without some massaging. (thanks jordi and DQEbert for bringing this to my attention.)
• Fix reindexing - databags were not being reindexed with the postgres search, and the SaveItem calls were moved to goroutines; otherwise, the request from knife would time out and knife would restart the reindex.
• Allow '.' in cookbook names; despite what an error chef-pedant is looking for, those are allowed. (thanks jordi and DQEbert for bringing this to my attention.)
• Make the authentication lib more general (thanks theckman)
• Output the version of golang used to build a particular goiardi binary (again, thanks theckman)
• The changed hostname in URLs to download bug didn't get fixed in 0.7.1 quite all the way after all. It is now. (Thanks to oker1 for bringing that to my attention.)
• Fixed search tests to pass when run using more than one processor. (Brought to my attention by theckman.)
• Fixed a deadlock that could happen when saving an in-mem index to disk at the exact moment an object was being indexed. Seems to be specific to go1.5.1 (or at least it never happened before that I saw), but needs fixed anyway. (Also brought to my attention by theckman.)
• Fixed broken pipe errors with too large requests when running chef-pedant against goiardi built with go 1.5.1.
• Update some docs.

• Search architecture changed so different search backends can be used (thanks oker1 for your work on that).
• Postgres search is here at last! If you're using Postgres, instead of using the ersatz solr search, you can instead use Postgres to power your searches.
• Add a mutex for the original goiardi search - multiple simple queries executing simultaneously are not a problem, but multiple complex queries can eat up all the RAM on the machine and cause goiardi to crash. This mitigates that situation.
• Be a little more forgiving with reporting protocol versions - allow specifying the protocol version as a query param instead of only as a header. This is to make showing reports with the webui a little easier.
• Bump the Chef Server version we claim to be from 11.1.6 to 11.1.7.

• Fix broken import/export function with reports - bringing goiardi's variable naming inline with golang conventions a while back inadvertently renamed a reporting JSON field. The field was renamed, and the import code will now handle both correct and incorrect names for the node reporting.

• Fix error where requests for zero byte cookbook files would crash.
• Authentication docs improvments (thanks oker1!)
• Rewritten and more robust cookbook depsolver.
• Fix for client creation with cheffish (thanks whiteley!)
• Fix for search where searching for something like "foo:bar AND NOT foo:bar" was returning incorrect results. (brought to my attention and test provided by brimstone, thanks!)
• Fixed a bug where clients could be created with the same name as a user (or vice versa) in in-memory mode.

• Validate IP address supplied on the command line or in the config file.
• Compress index docs to reduce memory usage with the search index.
• Ordering searches works now.
• Index and datastore files now only write to disk if there have been changes since the last time they were saved.
• In tandem with the previous change, freeze interval default has been changed from 300 seconds to 10 seconds.
• Bump Chef Server version we claim to be from 11.1.3 to 11.1.6.