Second release of "Learning CFEngine 3"

I am happy to announce that as of today, the second release of "Learning CFEngine 3" is available. If you have purchased the e-book version of the book you can get the new release from the "Your Products" page at oreilly.com (pro tip: set up Dropbox synchronization, and you will always have the latest version with you!). And if you haven't purchased it, now it's an excellent time to do it! You get all the fresh content, and all future releases for free.

These are the main updates in this release:
  • The book has been updated to reflect changes introduced up to CFEngine 3.3.9 (the latest stable release as of this writing), including many features that were previously only available in CFEngine Enterprise, such as guest_environments promises, databases promises, and more.
  • The installation instructions now reflect the availability of binary package repositories for many Linux distributions, as well as the “Free 25” Enterprise packages that allow you to try CFEngine Enterprise for free, for up to 25 machines.
  • Vim fans rejoice! A new Appendix, contributed by Neil H. Watson, provides details on how to use Vim to edit CFEngine policy files.
  • This release fixes all known errata, including many typos, omissions, and other numerous miscellaneous things.
I hope you enjoy this update! And if you have any feedback, please feel free to contact me.

CFEngine tip #004: How to bootstrap a CFEngine client

My apologies for the long delay since the last tip! Today we come back with a simple tip: how to bootstrap a CFEngine client.

First, some background

CFEngine is designed to operate in a fully distributed fashion – each CFEngine client (i.e. a machine running cf-agent) can operate fully autonomously, using only the policy files stored locally. For regular operation, cf-agent does not require any type of network connectivity.

In real deployments, of course, maintaining each machine’s policy independently would be impractical, so CFEngine has the concept of a policy hub. In CFEngine Community, the policy hub is simply a file server, a machine from which others can download policy files. This allows having a single point of distribution, so that changes made there can be distributed to a large number of clients. In CFEngine Enterprise, the hub has a more complex role, acting also as an aggregator of data from the clients, for the purposes of reporting and analysis.

All CFEngine clients will copy to their local /var/cfengine/inputs/ directory the contents of /var/cfengine/masterfiles/ in the policy hub, so that is where you should make any changes that you want distributed to all the machines.

Bootstrapping

When a new client is installed, it needs to be told which machine is the policy hub to which it should connect. This is done using the following command:

# cf-agent --bootstrap --policy-server=10.0.2.15

You should, of course, replace 10.0.2.15 with the actual IP address of the hub to which you want to bootstrap. The first machine you install will be the policy hub itself, and it should bootstrap to itself. In this case, you have to use its own external IP address, and not “localhost” or 127.0.0.1.

When you issue this command on the policy hub, you will see a message like this, that indicates the host recognizes itself as a policy hub:

# cf-agent --bootstrap --policy-server=10.0.2.15
** CFEngine BOOTSTRAP probe initiated

   @@@     
   @@@      CFEngine

 @ @@@ @    CFEngine Core 3.3.5
 @ @@@ @   
 @ @@@ @   
 @     @   
   @@@     
   @ @     
   @ @     
   @ @     

Copyright (C) CFEngine AS 2008-2012
See Licensing at http://cfengine.com/3rdpartylicenses

 -> This host is: precise32
 -> Operating System Type is linux
 -> Operating System Release is 3.2.0-23-generic-pae
 -> Architecture = i686
 -> Internal soft-class is linux
 -> No previous policy has been cached on this host
 -> Assuming the policy distribution point at: 10.0.2.15:/var/cfengine/masterfiles
 -> Attempting to initiate promised autonomous services...

 ** This host recognizes itself as a CFEngine Policy Hub, with policy distribution and knowledge base.
 -> The system is now converging. Full initialisation and self-analysis could take up to 30 minutes

R: This host assumes the role of policy distribution host
R:  -> Updated local policy from policy server
R:  -> Started the server
R:  -> Started the scheduler
-> Bootstrap to 10.0.2.15 completed successfully

If you issue it on a client, you will see a similar message, but indicating that it is bootstrapping from a different machine.

Regardless of what you use, after bootstrap you should see /var/cfengine/inputs/ populated with the default set of policy files, and both cf-execd and cf-serverd should be running:

# ps axw | grep [c]f-
16099 ?        Ss     0:00 /var/cfengine/bin/cf-execd
16102 ?        Ss     0:00 /var/cfengine/bin/cf-serverd

Once this has been done, CFEngine will start running every 5 minutes, updating its policy files from the hub, and executing them afterwards.

CFEngine tip #003: The distinction between bodies and bundles

This is an extract from Chapter 3 of "Learning CFEngine 3", where you can also find a much more comprehensive description of bodies and bundles in the CFEngine 3 policy language.

The distinction between bundles and bodies can be confusing at first. Remembering these points may help:

  • Bodies are named groups of attributes, whereas bundles are collections of promises. Promises are the units that actually do something in CFEngine (for example, run a command or add a line to a file), whereas attributes specify characteristics of how things are done (for example, whether to run the command in a shell, or where in the file to add the line).

  • The value of an attribute can be a basic data type (string, integer, list, etc.), it can be the name of a body, or it can be the name of a bundle.

  • The type of an attribute’s value is fixed, and determined by the attribute itself (for example, the value of the depth_search attribute in a files: promise is always a body, and the value of an edit_line attribute is always a bundle).

  • For bodies and bundles, their type is always the name of the attribute to which they correspond. For example, bodies to be used with the depth_search attribute are always declared as “body depth_search xyz”, where xyz is an arbitrary name of your choosing. The same goes for bundles: bundles to be used with the edit_line attribute are always declared as “bundle edit_line xyz".

    There are only four types of “top level” bundles that are not used as arguments to attributes: agent, server, knowledge and monitor.

  • The promise types (sections) that can appear in a bundle are determined by the bundle type. For example, commands: promises can only appear in bundles of type agent. 

CFEngine tip #002: How to pass arguments to bundles using arrays

(This tip is based on a section from Chapter 5 of Learning CFEngine 3.)

Many system configuration tasks require groups of name-value pairs as arguments. For example:

  • Editing configuration files in which parameters and their values need to be stored (ssh configuration files, Windows-style INI files, etc.)
  • Setting user parameters. In this case, sets of name-value pairs (home directory, full name, shell, etc.) are associated with a single user, identified by name.

Having sets of related values in a single array has a number of advantages, since they can be manipulated by a single set of promises just by varying the indices used to access them. To make use of this array, you have to pass it as an argument to a bundle. One of the most useful functions in this technique is getindices(), which returns a list containing the indices of the given array, and can be used to produce an enumeration of the elements over which to iterate. The complementary function to get just the values is getvalues(). For example, consider this bundle:

To pass arrays as arguments we must pass a string with the name of the array, and then dereference it inside the function. The argument we are passing to set_config_values() is “configfiles.sshd”, which refers to the sshd array defined in the configfiles() bundle. The dereferencing happens in the set_config_values() bundle:

This bundle receives the name of the array as the “v” parameter, so we dereference the array and its values by using $(v) wherever we would normally use the array name. For example, to loop over the array elements using the indices stored in the $(index) list, we use $($(v)[$(index)]) instead of $(configfiles.sshd[$(index)]).

To group name/value sets into named groups, we can use two-dimensional arrays, as in this example:

In this case the dereferencing can get a little more complicated. For example, let us look at some of the code inside the create_users() bundle:

This bundle is being called from the methods: section of the manage_users() bundle, with the string "manage_users.users" as the value of $(info). We use getindices() directly on this value to get a list of the first-level indices of the array (the user names), which we store in @(user). Then we use implicit looping over @(user) to cycle through all those values, and we use the following construction to access individual elements of each user’s data: $($(info)[$(user)][field]). This expands to $(manage_users.users[$(user)][field]), on which implicit looping is applied through the $(user) variable. Remember that parenthesis (or curly braces, they mean the same) are required around the whole expression, so that CFEngine recognizes it properly as a variable reference.

While the syntax looks complicated, this data structure allows great flexibility in passing around and using data structures to be used in configuration operations.

Get a free copy of "Learning CFEngine 3" in exchange for a review

Are you a technical blogger? O'Reilly's "Blogger Review Program" allows bloggers to get free ebooks and videos, in exchange for writing reviews of them.

I'd love to see more reviews of "Learning CFEngine 3". If you have a technical blog, and would be interested in reviewing the book, I encourage you to visit the program's website to find out more.

Plus, I'm sure you will enjoy the book, and I would love to read your review!

Day Against DRM at O'Reilly: 50% discount on "Learning CFEngine 3" and all other ebooks

Today, May 4th, 2012, O'Reilly Media is participating in support of the FSF's Day Against DRM. All O'Reilly ebooks and videos are DRM-free, and to celebrate this, you can get 50% off on any purchase by using the code DRMFREE.

I'm proud to participate. Learning CFEngine 3 is also available as a DRM-free ebook, and you can use this code to get it at half price.

Here's the message from O'Reilly. Find out more at http://shop.oreilly.com/category/deals/day-against-drm.do

In Celebration of *Day Against DRM*

Save 50% on ALL Ebooks & Videos – Use Code: DRMFREE

Having the ability to download files at your convenience, store them on all your devices, or share them with a friend or colleague as you would a print book, is liberating, and is how it should be. If you haven't tried a DRM-free ebook or video, we encourage you to do so now. And if you're already a fan, take advantage of our sale and add to your library. For one day only, you can save 50% on all O'ReillyNo Starch, and Rocky Nook ebooks and videos. Learn more about Day Against DRM

Ebooks from oreilly.com are DRM-free. You get free lifetime access, multiple file formats, free updates. 
Deal expires May 4, 2012 at 11:59pm PT and cannot be combined with other offers.

CFEngine talks at PICC'12 conference

There will be several CFEngine-related classes and talks at the PICC'12 conference (Professional IT Community Conference) on May 11-12th, in New Jersey. I will be talking about using CFEngine as part of your security infrastructure. My colleague Joe Netzel will be talking about migration from CFEngine 2 to CFEngine 3, so if you are using CF2 and have been considering (or dreading) the migration, make sure to come by! Aleksey Tsalolikhin will give an introductory class to CFEngine. And Mark Burgess will give a class not related to CFEngine, but to the perils and skills needed to survive as a technical person in the "business world".

If you are coming to the conference, we look forward to seeing you there!

A full shelf

I got this photo of a bookshelf at the CFEngine office in Palo Alto. Looks awesome, doesn't it? :)

Photo

CFEngine tip #001: Modularize complex policies using methods: calls

Welcome to the new "CFEngine tip" series. In this series of posts I will explore different tips, tricks and techniques for making better use of CFEngine. For now I will post weekly, but the frequency may change as time goes by. If you have any suggestions for topics that I should cover in this series, please let me know!

For our first installment, we will talk about methods: promises, and how they can be used to make policies both clearer and more extensible.

As your CFEngine policies grow in complexity, they become harder to read if all your promises are in a single bundle, or if you split them among bundles, to have all those bundles listed in the bundlesequence declaration.

Enter the methods: promise type. Promises of this type allow you to call other bundles in sequence, passing arbitrary parameters. For example, consider the following example (from Chapter 5 of "Learning CFEngine 3"):

The methods: section makes it very easy to see the sequence of actions that will take place: backup some files, configure sysctl, sshd, inittab and users in sequence. This helps both in clarity and in code reusability: each bundle can perform logically different functionality, and they could potentially be reused in different settings and with different parameters. The promiser string in a methods: promise is an arbitrary string. The CFEngine documentation uses "any", but you can also use it, as in this example, to indicate the purpose of the promise.

Furthermore, you can use CFEngine's implicit looping to create generic promises that will call as many other bundles as necessary. Consider this revised code:

Now the list of bundles to call (edit_sysctl, edit_sshd and edit_inittab) is being built on the fly from the contents of the files array, and used through implicit looping to call the bundles in the methods: promises. Note how you can even pass arguments to these calls. This makes it very easy to extend the functionality by simply adding new elements to the files array (and of course, defining the appropriate edit_* bundle).

Until next time!

New "reviews" page

I have started collecting some of the nice things I have read people say about the book, and put them in the Reviews page. Check it out!