Session Title: Cookbooks: Enhance and/or ReplaceConvener: Carl Perry Participants: Summary of discussions:The primary topic of discussion was limitations within the data bag system. The basis was there are several things that chef does not and should not support (eg: a full graph database system for tracking dependencies and interconnects for networking). External systems and application are much better suited for some specific needs, but they need a way to put that information into chef. An idea was proposed to have an "link URL" in which the necessary information could be periodically fetched from an external system, cached for certain period of time, and then merged into the existing JSON structure. However, it was decided that a much simpler approach would be implement a form of deltas and opportunistic locking for all JSON transactions: data bags, roles, and node attributes. Opportunistic LockingCurrently, all JSON documents stored are given a unique identifier by CouchDB. With the move to a SQL based backend, that ID field will need to be preserved. We discussed just using an incremental integer, but some discussions after made it sound like a last modified timestamp field would be better. Once this last modified timestamp is in place, whenever a save is done to a JSON document (node attributes, role attributes, or data bag) the last modified timestamp in the PUT must match the last modified timestamp in the currently stored document. If there is a mismatch, then another PUT has occurred and the new contents are invalid. This will result in a failure. If they match, then the server will update the last modified timestamp in the PUT JSON document and then save to the data store. It was suggested that this change can be implemented now, with little impact to existing tools and will provide a much greater level of data integrity. JSON DeltasIn some cases, data written by external tools to chef JSON documents is meant as read-only for the chef client. In order to better support these, the idea of JSON deltas was proposed. The idea is to have a new API endpoint where only parts of the JSON document could be submitted. These would be in a dict containing three keys:
Under normal conditions a failure of the above statements, such as attempting to update or delete a key which does not already exist, or creating a key which does already exist, would result in a failure. This would be the standard method for a save call within a recipe. However, the save at the end of a chef-client run would have the force flag enabled on the call. This would cause most errors to continue (deletion of a non-existent key is ignored, creation of a key which already exists acts as an update, updating a key which already exists acts as a create). The idea is that the delta API could ignore the last modified timestamp used for opportunistic locking. Encrypted Data Bag EnhancementsThere was also discussion about making encrypted data bags more friendly. The idea is to use the the existing chef client keys for the encrypted data bags as well. In order for this to be as secure as possible, the validation.pem method of creating a client needs to be changed slightly. The client key should be generated on the client and never sent to the server. Instead, a CSR should be sent, and the server should return the signed certificate. This also means fixing the node creation method where a client can submit their own public key to the server (may work in the platform, does not work on open source server). Mathieu Sauve-Frankel will detail the server side public key escrow system here What will we do now? What needs to happen next?
|
Labels:
None