diff --git a/documentation/documentation/howitworks.page b/documentation/documentation/howitworks.page index 548db225d..a686c4c90 100644 --- a/documentation/documentation/howitworks.page +++ b/documentation/documentation/howitworks.page @@ -1,285 +1,285 @@ --- inMenu: true title: How It Works orderInfo: 9 --- # Introduction The goal of this document is to describe how a manifest you write in Puppet gets converted to work being done on the system. This process is relatively complex, but you seldom need to know many of the details; this document only exists for those who are pushing the boundaries of what Puppet can do or who don't understand why they are seeing a particular error. It can also help those who are hoping to extend Puppet beyond its current abilities. # High Level When looked at coarsely, Puppet has three main phases of execution -- compiling, instantiation, and configuration. ## Compiling Here is where we convert from a text-based manifest into the actual code we'll be executing. Any code not meant for the host in question is ignored, and any code that is meant for that host is fully interpolated, meaning that variables are expanded and all of the results are literal strings. The only connection between the compiling phase and the library of Puppet elements is that all resulting elements are verified that the referenced type is valid and that all specified attributes are valid for that type. There is no value validation at this point. In a networked setup, this phase happens entirely on the server. The output of this phase is a collection of very simplistic elements that closely resemble basic hashes and arrays. ## Instantiation This phase converts the simple hashes and arrays into Puppet library objects. Because this phase requires so much information about the client in order to work correctly (e.g., what type of packaging is used, what type of services, etc.), this phase happens entirely on the client. The conversion from the simpler format into literal Puppet objects allows those objects to do greater validation on the inputs, and this is where most of the input validation takes place. If you specified a valid attribute but an invalid value, this is where you will find it out, meaning that you will find it out when the config is instantiated on the client, not (unfortunately) on the server. The output of this phase is the machine's entire configuration in memory and in a form capable of modifying the local system. ## Configuration This is where the Puppet library elements actually modify the system. Each of them compares their specified state to the state on the machine and make any modifications that are necessary. If the machine exactly matches the specified configuration, then no work is done. The output of this phase is a correctly configured machine, in one pass. # Lower Level These three high level phases can each be broken down into more steps. ## Compile Phase 1: Parsing * *Inputs* Manifests written in the Puppet language * *Outputs* Parse trees (instances of [AST][ast] objects) * *Entry* [Puppet::Parser::Parser#parse][parse] At this point, all Puppet manifests start out as text documents, and it's the parser's job to understand those documents. The parser (defined in ``parser/grammar.ra`` and ``parser/lexer.rb``) does very little work -- it converts from text to a format that maps directly back to the text, building parse trees that are essentially equivalent to the text itself. The only validation that takes place here is syntactic. This phase takes place immediately for all uses of Puppet. Whether you are using nodes or no nodes, whether you are using the standalone puppet interpreter or the client/server system, parsing happens as soon as Puppet starts. ## Compile Phase 2: Interpreting * *Inputs* Parse trees (instances of [AST][] objects) and client information (collection of facts output by [Facter][]) * *Outputs* Trees of [TransObject][] and [TransBucket][] instances (from transportable.rb) * *Entry* [Puppet::Parser::AST#evaluate][ast_evaluate] * *Exit* [Puppet::Parser::Scope#to_trans][] Most configurations will rely on client information to make decisions. When the Puppet client starts, it loads the [Facter][] library, collects all of the facts that it can, and passes those facts to the interpreter. When you use Puppet over a network, these facts are passed over the network to the server and the server uses them to compile the client's configuration. This step of passing information to the server enables the server to make decisions about the client based on things like operating system and hardware architecture, and it also enables the server to insert information about the client into the configuration, information like IP address and MAC address. The [interpreter][] combines the parse trees and the client information into a tree of simple [transportable][] objects which maps roughly to the configuration as defined in the manifests -- it is still a tree, but it is a tree of classes and the elements contained in those classes. ### Nodes vs. No Nodes When you use Puppet, you have the option of using [node elements][] or not. If you do not use node elements, then the entire configuration is interpreted every time a client connects, from the top of the parse tree down. In this case, you must have some kind of explicit selection mechanism for specifying which code goes with which node. If you do use nodes, though, the interpreter precompiles everything except the node-specific code. When a node connects, the interpreter looks for the code associated with that node name (retrieved from the Facter facts) and compiles just that bit on demand. ## Configuration Transport * *Inputs* [Transportable][] objects * *Outputs* [Transportable][] objects * *Entry* [Puppet::Server::Master#getconfig][] * *Exit* [Puppet::Client::MasterClient#getconfig][] If you are using the stand-alone puppet executable, there is no configuration transport because the client and server are in the same process. If you are using the networked puppetd client and puppetmasterd server, though, the configuration must be sent to the client once it is entirely compiled. Puppet currently converts the Transportable objects to [YAML][], which it then CGI-escapes and sends over the wire using XMLRPC over HTTPS. The client receives the configuration, unescapes it, caches it to disk in case the server is not available on the next run, and then uses YAML to convert it back to normal Ruby Transportable objects. ## Instantiation Phase * *Inputs* [Transportable][] objects * *Outputs* [Puppet::Type][] instances * *Entry* [Puppet::Client::MasterClient#getconfig][] * *Exit* [Puppet::Type#finalize][] To create Puppet library objects (all of which are instances of [Puppet::Type][] subclasses), ``to_trans`` is called on the top-level transportable object. All container objects get converted to [Puppet::Type::Component][] instances, and all normal objects get converted into the appropriate Puppet type instance. This is where all input validation takes place and often where values get converted into more usable forms. For instance, filesystems always return user IDs, not user names, so Puppet objects convert them appropriately. (Incidentally, sometimes Puppet is creating the user that it's chowning a file to, so whenever possible it ignores validation errors until the last minute.) -The last phase of instantiation is the *finalization* phase. One of the goals -of the Puppet language is to make file order matter as little as possible; -this means that a Puppet object needs to be able to require other objects -listed later in the manifest, which means that the required object will be -instantiated after the requiring object. So, the finalization phase is used +The last stage of the instantiation process finalizes the object dependencies. +One of the goals of the Puppet language is to make file order matter as little +as possible; this means that a Puppet object needs to be able to require other +objects listed later in the manifest, which means that the required object will +be instantiated after the requiring object. So, this finalization stage is used to actually handle all of these requirements -- Puppet objects use their references to objects and verify that the objects actually exist. ## Configuration Phase 1: Comparison * *Inputs* [Puppet::Type][] instances * *Outputs* [Puppet::StateChange][] objects collected in a [Puppet::Transaction][] instance * *Entry* [Puppet::Client::MasterClient#apply][] * *Exit* [Puppet::Type::Component#evaluate][component_evaluate] Before Puppet does any work at all, it compares its entire configuration to the state on disk (or in memory, or whatever). To do this, it recursively iterates across the tree of [Puppet::Type][] instances (which, again, still roughly maps to the class structure defined in the manifest) and calls ``evaluate``. Things are a bit messier than this in real life, but the summary is that ``evaluate`` retrieves the state of each object, compares that state to the desired state, and creates a Puppet::StateChange object for every individual bit that's out of sync (e.g., if a file has the wrong owner and wrong mode, then each of those are in separate StateChange instances). The end result of evaluating the whole tree is a collection of StateChange objects for every bit that's out of sync, all sorted in order of dependencies so that objects are always fixed before the objects that depend on them. The top-level component (which is also responsible for this sorting) creates a Puppet::Transaction instance and inserts these changes into it. ### Notes About Recursion Recursion muddies this phase considerably. While it's tempting to merely handle recursion in the instantiation phase, the state on disk can (and will) change between runs, so the configured state and the on-disk state must be compared on every run (and it is assumed that ``puppetd`` will be a long-running process that only does instantiation once but does configuration many times). This means that there might still be objects that don't exist at the end of instantiation but do exist at the end of comparison. In particular, when doing recursive file copies from a remote machine, Puppet creates an object in memory to map to every remote file, and that recursive object creation would not make sense at instantiation time, only at comparison time. This might introduce some strangenesses, though, and it is expected that this could cause interesting-in-a-not-particularly-good-way edge cases. ## Configuration Phase 2: Syncing * *Inputs* [Puppet::Transaction][] instance containing [Puppet::StateChange][] instances * *Outputs* Completely configured operating system * *Entry* [Puppet::Type::Component#evaluate][component_evaluate] * *Exit* [Puppet::Transaction#evaluate][] The transaction's job is just to execute each change. The changes themselves are responsible for logging everything that happens (one of the reasons that all work is done by StateChange objects rather than just letting the objects do it is to guarantee that every modification is logged). This execution is done by calling ``go`` on each change in turn, and if the change does any work then it produces an event of some kind. These events are collected until all changes have been executed. Once the transaction is complete, all of the events are checked to see if there are any callbacks associated with them. Puppet currently only supports one type of callback and one way of specifying them: Calling ``refresh`` on objects based on that object subscribing to another object. For instance, take the following snippet: file { "/etc/ssh/sshd.conf": source => "puppet://puppet/config/sshd.conf" } service { sshd: running => true, subscribe => file["/etc/ssh/sshd.conf"] } If the local file is out of sync with the remote file, then a StateChange instance is created reflecting this. When that change is executed, it creates a ``file_changed`` event. Because of the above subscription, the callback associated with this event is to call ``refresh`` on the ``sshd`` service; for services, ``refresh`` is equivalent to restarting, so sshd is restarted. In this way, Puppet elements can react to changes that it makes to the system. While transactions are fully capable of moving both forward and backward (e.g., if a transaction encountered an error, it could back out all of its changes), there are currently no hooks within Puppet itself to specify when and why that would happen. If this is a critical feature for you or you have a brilliant way to go about creating it, I would love to hear it, but it is currently a back-burner goal. # Conclusion That's the entire flow of how a Puppet manifest becomes a complete configuration. There is more to the Puppet system, such as FileBuckets, but those are more support staff rather than the main attraction. [facter]: /projects/facter [parse]: /downloads/puppet/apidocs/classes/Puppet/Parser/Parser.html [AST]: /downloads/puppet/apidocs/classes/Puppet/Parser/AST.html [node elements]: /projects/puppet/documentation/structures.html#nodes [yaml]: http://www.yaml.org/ [Puppet::Parser::Parser#parse]: /downloads/puppet/apidocs/classes/Puppet/Parser/Parser.html [ast_evaluate]: /downloads/puppet/apidocs/classes/Puppet/Parser/AST.html [Puppet::Parser::Scope#to_trans]: /downloads/puppet/apidocs/classes/Puppet/Parser/Scope.html [TransObject]: /downloads/puppet/apidocs/classes/Puppet/TransObject.html [TransBucket]: /downloads/puppet/apidocs/classes/Puppet/TransBucket.html [Puppet::Server::Master#getconfig]: /downloads/puppet/apidocs/classes/Puppet/Server/Master.html [Puppet::Client::MasterClient#getconfig]: /downloads/puppet/apidocs/classes/Puppet/Client/MasterClient.html [Transportable]: /downloads/puppet/apidocs/classes/Puppet/TransBucket.html [Puppet::StateChange]: /downloads/puppet/apidocs/classes/Puppet/StateChange.html [Puppet::Transaction]: /downloads/puppet/apidocs/classes/Puppet/Transaction.html [Puppet::Client::MasterClient#apply]: /downloads/puppet/apidocs/classes/Puppet/Client/MasterClient.html [component_evaluate]: /downloads/puppet/apidocs/classes/Puppet/Type/Component.html [Puppet::Type::Component]: /downloads/puppet/apidocs/classes/Puppet/Type/Component.html [Puppet::Transaction#evaluate]: /downloads/puppet/apidocs/classes/Puppet/Transaction.html [interpreter]: /downloads/puppet/apidocs/classes/Puppet/Parser/Interpreter.html [Puppet::Type]: /downloads/puppet/apidocs/classes/Puppet/Type.html [Puppet::Type#finalize]: /downloads/puppet/apidocs/classes/Puppet/Type.html *$Id$*