Summary

There are cases in Chef where the configuration of one or more resources depends on the system state or data collected by another resource which is not available until some part of the system's configuration has converged. However, Chef's two phase (compile/converge) execution model, combined with the pass-by-value semantics of Node attributes, makes it unwieldy to dynamically determine configuration values based on data that is not available until the converge phase.

Given that both Chef itself and the Chef community strongly prefer to bring a node to full convergence in a single run, we need to review the use cases for this kind of behavior, as well as the existing patterns/solutions/hacks/workarounds used to achieve it. After a review of this information, we will be in a position to determine if modifications should be made to Chef to enhance support for this behavior, and what those enhancements will look like.

Use Cases Needed If you need this behavior, please edit this page to describe your use case in the section below

Use Cases

(1) Calling packages in ruby_block above the package resource

Having the capability to have resources cycle thru an arbitrary run-time list is very useful. For example: in a later section of a script, ownership is to bve fixed across a bunch of files, trivial to do using legacy bash, but cant be done easily in Chef (the ability to retrieve the list and manipulate it is also extremely easy in Ruby, but isnt translating to Chef).

Code in a ruby_block is evaluated with other resources during configuration/convergence (2nd phase), but packages are only added to the collection during compilation (1st phase). During compilation (1st Phase) the files are not available from the resource/package evaulation in the (2nd phase). So when calling package (or rpm_package) in the ruby_block above the package resource effectively does not exist, and if it did it would never get executed.

Hacky solution:

(2) JIT evaluation

There are times when passing parameters to resources it would be nice if that parameter was not evaluated at compile time but at runtime. This is clearly and most easily in play with not_if/only_if, where the lambda/string that is passed is evaluated just before runtime.

An easy implementation of this would be to allow the passage of lambdas as parameter arguments, such that the evaluation of the lambda is done at runtime. I'm not sure what downside this brings other than complexity, but it can be very valuable in certain situations.

The problem here isn't in triggering one action off of another, but fulfilling the values of that action. For example, what if I had a user creation resource, then a template that needed to fill in a value based on the created UID of that resource?

Obviously this won't work because the lookup is done at compile time, before the new user is ever created.

However, if I can tuck that into a lambda:

Then I can defer that the lookup isn't evaluated until the value is actually needed, JIT.

It may be valuable to do this in more than template variables, but that's where my example is best seen I think.

(3) Service Discovery

Ganglia wants to know where all its agents are. Its agents want to know where their master is.

Zabbix wants to monitor flume. Flume wants to transport zabbix's logs to the HDFS.

Even when there is no Ouroboros present, this is a persistent source of race conditions. Service A wants to consume Service B, which could be on the same or a different node (staging vs production, master is also a worker, lightweight redis for rails app vs bigass replicated redis for primary data store). Stop/starting a machine on EC2 yields a new IP address, so it's difficult to distinguish a remote resource from an outdated shadow of your former self.

Anyway, the best time for this discovery pass is when the machine has a full sense of its purpose in life and what it has to offer.

It would also mitigate cross-cluster race conditions. Simultaneously launching 30 machines that all want to cross-discover is an interesting affair; a little breathing room between announce and discover would greatly smooth the path to idempotent runs.

(4) Components that might not be present

An hbase node might have one some or all of the hbase master, namenode, datanode, and eight other services. Some of those depend on the same file (in this case, /etc/hadoop/conf/core-site.xml and a few others). When that file is modified, I'd like to restart the daemons. If the file is prepared at the end of convergence, I can know what came before it.

However, the hadoop apt packages start the daemons on install – which means you'd like to have the template file in place before anything else runs.

We use a workaround – we always declare a :run_state attribute for daemons, so other components will assume that the service exists if and only if the service is in the 'start' run_state. But that's hacky and wrong.

(4) Competing for Utilization

Going still with HBase/Hadoop – if I'm a typical hadoop compute machine, with only a tasktracker and datanode, I can be agressive in memory usage, largely towards the tasktracker – serving data is boring, it's compute jobs that need ram. If I'm a machine that has an hbase regionserver on it, I want one little wussy tasktracker (one which will be off/asleep for most of its life), a healthy datanode, a highly aggressive regionserver and probably 50+% of ram left over for system caches.

Now you're saying to yourself "self, that sounds like a case for machine-specific tuning". But I'm not talking about an infinite multiplicity of use cases: it's "be a reasonably aggressive compute node when there is no regionserver" or "be a reasonably aggressive database node when there is" or "do it your damn self".

In that case, you have two fairly close-bound components, so it might be fair (if distasteful) to have some coupled logic.

However, consider a machine that specifies both say tokyotyrant and elasticsearch (or their like). Each uses only a single volume, but it uses the hell out of that volume. There's no built-in way to have them peacefully choose distinct volumes on a machine with more than one disk.

More broadly, the machine has a certain set of resources – ram, volumes, ports, uids, etc – that these components would like to discover, claim or use their proportionate share of. Until they know what's out there, they can't be good neighbors and ambitious public servants.