diff --git a/CHANGELOG b/CHANGELOG index 659ae28b7..657bac6d6 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -1,1068 +1,1077 @@ + Found all instances of methods where split() is used without + any local variables and added a local variable -- see + http://snurl.com/21zf8. My own testing showed that this + caused memory growth to level off at a reasonable level. + Note that the link above says the problem is only with class + methods, but my own testing showed that it's any method that + meets these criteria. This is not a functional change, but + should hopefully be the last nail in the coffin of #1131. + Found an array that leaked pretty quickly between reparsing files, thanks to work by Adam Jacob and Arjuna Christenson (the finding, not the leak). I'm going to act like this fixes #1131, at least for now, but I doubt it does, since that shows general memory growth over time, whereas the leak here should go away as soon as files are reparsed (because the parser is holding the reference to the leaking array). Fixed #1147: Cached nodes are correctly considered out of date if the node facts have been updated (thus causing node facts to again be available in manifests, for those cases where they were not). Fixed #1137: The certificate name is correctly being added to the facts hash. Fixed #1136: Verbose and Debug no longer clobber each other. Hopefully *finally* fixed the "already being managed" problem (#1036). The problem only cropped up if there was a failure when trying to manage the system -- this would cause the setting-based resources not to get cleaned up. 0.24.3 Modified the ldap node terminus to also use the facts version as the version for a node, which should similarly encourage the use of the yaml cache. (Related to #1130) Caching node information in yaml (I figured caching in memory will cause ever-larger memory growth), and changing the external node terminus to use the version of the facts as their version. This will usually result in the cached node information being used, instead of always hitting the external node app during file serving. Note that if the facts aren't changed by the client, then this will result in the cached node being used, but at this point, the client always updates its facts. (#1130) Fixing #1132 -- host names can now have dashes anywhere. (Patch by freiheit.) Fixing #1118 -- downloading plugins and facts now ignores noop. Note that this changes the behaviour a bit -- the resource's noop setting always beats the global setting (previously, whichever was true would win). The change in checksums from 'timestamp' to 'mtime' no longer result in updates on every run (#1116). Aliases again work in relationships (#1094). The CA serial file will no longer ever be owned by root (#1041). Fixing the rest of #1113: External node commands can specify an environment and Puppet will now use it. Partially fixing #1113: LDAP nodes now support environments, and the schema has been updated accordingly. Always duplicating resource defaults in the parser, so that stacked metaparameter values do not result in all resources that receive a given default also getting those stacked values. 0.24.2 Fixing #1062 by moving the yamldir setting to its own yaml section. This should keep the yamldir from being created on clients. Fixed #1047 -- Puppet's parser no longer changes the order in which statements are evaluated, which means that case statements can now set variables that are used by other variables. Fixed #1063 -- the master correctly logs syntax errors when reparsing during a single run. Removed the loglevels from the valid values for `logoutput` in the Exec resource type -- the log levels are specified using the `loglevel` parameter, not `logoutput`. This never worked, or at least hasn`t for ages, and now the docs are just correct. Somewhat refactored fileserving so that it no longer caches any objects, nor does it use Puppet's RAL resources. In the process, I fixed #894 (you can now copy links) and refactored other classes as necessary. Mostly it was fixing tests. Hopefully partially fixed #1010 -- clients should now fail to install files whose checksums do not match the checksum from the server. Fixed #1018 -- resources now have their namevars added as aliases in the resource catalog, just like they were added in the resource classes. Fixed #1037 -- remote unreadable files no longer have the permission denied exceptions caught, thus forbidding them from being replaced with 'nil'. Fixed #1043 -- autoloading now searches the plugins directory in each module, in addition to the lib directory. The 'lib' directory is also deprecated, but supported for now to give people a chance to convert. Fixed #1003 -- Applying DavidS's patch to fix searching for tags in sql. Fixed #992 -- Puppet is now compatible with gems 1.0.1. Fixed #968 again, this time with tests -- parseonly works, including not compiling the configurations, and also storeconfigs is no longer required during parse-testing. Fixed #1021 -- the problem was that my method of determining the in-degree sometimes resulted in a lower number than the number of in-edges. Fixed #997 -- virtual defined types are no longer evaluated. NOTE: This introduces a behaviour change, in that you previously could realize a resource within a virtual defined resource, and now you must realize the entire defined resource, rather than just the contained resource. Fixed #1030 - class and definition evaluation has been significantly refactored, fixing this problem and making the whole interplay between the classes, definitions, and nodes, and the Compile class much cleaner. Exec resources must now have unique names, although the commands can still be duplicated. This is easily accomplished by just specifying a unique name with whatever (unique or otherwise) command you need. Fixed #989 -- missing CRL files are correctly ignored, and the value should be set to 'false' to explicitly not look for these files. Fixed #1017 -- environment-specific modulepath is no longer ignored. Fixing #794 -- consolidating the gentoo configuration files. Fixing #976 -- both the full name of qualified classes and the class parts are now added as tags. I've also created a Tagging module that we should push throughout the rest of the system that uses tags. Fixing #995 -- puppetd no longer dies at startup if the server is not running. Fixing #977 -- the rundir is again set to 1777. Fixed #971 -- classes can once again be included multiple times. Added builtin support for Nagios types using Naginator to parse and generate the files. 0.24.1 Updated vim filetype detection. (#900 and #963) Default resources like schedules no longer conflict with managed resources. (#965) Removing the ability to disable http keep-alive, since it didn't really work anyway and it should no longer be necessary. Refactored http keep-alive so it actually works again. This should be sufficient enough that we no longer need the ability to disable keep-alive. There is now a central module responsible for managing HTTP instances, along with all certificates in those instances. Fixed a backward compatibility issue when running 0.23.x clients against 0.24.0 servers -- relationships would consistently not work. (#967) Closing existing http connections when opening a new one, and closing all connections after each run. (#961) Removed warning about deprecated explicit plugins mounts. 0.24.0 (misspiggy) Modifying the behaviour of the certdnsnames setting. It now defaults to an empty string, and will only be used if it is set to something else. If it is set, then the host's FQDN will also be added as an alias. The default behaviour is now to add 'puppet' and 'puppet.$domain' as DNS aliases when the name for the cert being signed is equal to the signing machine's name, which will only be the case for CA servers. This should result in servers always having the alias set up and no one else, but you can still override the aliases if you want. External node support now requires that you set the 'node_terminus' setting to 'exec'. See the IndirectionReference on the wiki for more information. http_enable_post_connection_check added as a configuration option for puppetd. This defaults to true, which validates the server SSL certificate against the requested host name in new versions of ruby. See #896 for more information. Mounts no longer remount swap filesystems. Slightly modifying how services manage their list of paths (and adding documention for it). Services now default to the paths specified by the provider classes. Removed 'type' as a valid attribute for services, since it's been deprecated since the creation of providers. Removed 'running' as a valid attribute for services, since it's been deprecated since February 2006. Added modified patch by Matt Palmer which adds a 'plugins' mount, fixing #891. See PluginsInModules on the wiki for information on usage. Empty dbserver and dbpassword settings will now be ignored when initializing Rails connections (patch by womble). Configuration settings can now be blank (patch by womble). Added calls to endpwent/endgrent when searching for user and group IDs, which fixes #791. Obviated 'target' in interfaces, as all file paths were automatically calculated anyway. The parameter is still there, but it's not used and just generates a warning. Fixing some of the problems with interface management on Red Hat. Puppet now uses the :netmask property and does not try to set the bootproto (#762). You now must specify an environment and you are required to specify the valid environments for your site. (#911) Certificates now always specify a subjectAltName, but it defaults to '*', meaning that it doesn't require DNS names to match. You can override that behaviour by specifying a value for 'certdnsnames', which will then require that hostname as a match (#896). Relationship metaparams (:notify, :require, :subscribe, and :before) now stack when they are collecting metaparam values from their containers (#446). For instance, if a resource inside a definition has a value set for 'require', and you call the definition with 'require', the resource gets both requires, where before it would only retain its initial value. Changed the behavior of --debug to include Mongrel client debugging information. Mongrel output will be written to the terminal only, not to the puppet debug log. This should help anyone working with reverse HTTP SSL proxies. (#905) Fixed #800 -- invalid configurations are no longer cached. This was done partially by adding a relationship validation step once the entire configuration is created, but it also required the previously-mentioned changes to how the configuration retrieval process works. Removed some functionality from the Master client, since the local functionality has been replaced with the Indirector already, and rearranging how configuration retrieval is done to fix ordering and caching bugs. The node scope is now above all other scopes besides the 'main' scope, which should help make its variables visible to other classes, assuming those classes were not included in the node's parent. Replaced GRATR::Digraph with Puppet::SimpleGraph as the base class for Puppet's graphing. Functionality should be equivalent but with dramatically better performance. The --use-nodes and --no-nodes options are now obsolete. Puppet automatically detects when nodes are defined, and if they are defined it will require that a node be found, else it will not look for a node nor will it fail if it fails to find one. Fixed #832. Added the '--no-daemonize' option to puppetd and puppetmasterd. NOTE: The default behavior of 'verbose' and 'debug' no longer cause puppetd and puppetmasterd to not daemonize. Added k5login type. (#759) Fixed CA race condition. (#693) Added shortname support to config.rb and refactored addargs 0.23.2 Fixed the problem in cron jobs where environment settings tended to multiple. (#749) Collection of resources now correctly only collects exported resources again. This was broken in 0.23.0. (#731) 'gen_config' now generates a configuration with all parameters under a heading that matches the process name, rather than keeping section headings. Refactored how the parser and interpreter relate, so parsing is now effectively an atomic process (thus fixing #314 and #729). This makes the interpreter less prone to error and less prone to show the error to the clients. Note that this means that if a configuration fails to parse, then the previous, parseable configuration will be used instead, so the client will not know that the configuration failed to parse. Added support for managing interfaces, thanks to work by Paul Rose. Fixed #652, thanks to a patch by emerose; --fqdn again works with puppetd. Added an extra check to the Mongrel support so that Apache can be used with optional cert checking, instead of mandatory, thus allowing Mongrel to function as the CA. This is thanks to work done by Marcin Owsiany. 0.23.1 (beaker) You can now specify relationships to classes, which work exactly like relationships to defined types: require => Class[myclass] This works with qualified classes, too. You can now do simple queries in a collection of exported resources. You still cannot do multi-condition queries, though. (#703) puppetca now exits with a non-zero code if it cannot find any host certificates to clean. (Patch by Dean Wilson.) Fully-qualified resources can now have defaults. (#589) Resource references can now be fully-qualified names, meaning you can list definitions with a namespace as dependencies. (#468) Files modified using a FileType instance, as ParsedFile does, will now automatically get backed up to the filebucket named "puppet". Added a 'maillist' type for managing mailing lists. Added a 'mailalias' type for managing mail aliases. Added patch by Valentin Vidic that adds the '+>' syntax to resources, so parameter values can be added to. The configuration client now pulls libraries down to $libdir, and all autoloading is done from there with full support for any reloadable file, such as types and providers. (#621) Note that this is not backward compatible -- if you're using pluginsync right now, you'll need to disable it on your clients until you can upgrade them. The Rails log level can now be set via (shockingly!) the 'rails_loglevel' parameter (#710). Note that this isn't exactly the feature asked for, but I could not find a way to directly copy ActiveRecord's concept of an environment. External node sources can now return undefined classes (#687). Puppet clients now have http proxy support (#701). The parser now throws an error when a resource reference is created for an unknown type. Also, resource references look up defined types and translate their type accordingly. (#706) Hostnames can now be double quoted. Adding module autoloading (#596) -- you can now 'include' classes from modules without ever needing to specifically load them. Class names and node names now conflict (#620). 0.23.0 Modified the fileserver to cache file information, so that each file isn't being read on every connection. Also, added londo's patch from #678 to avoid reading entire files into memory. Fixed environment handling in the crontab provider (#669). Added patch by trombik in #572, supporting old-style freebsd init scripts with '.sh' endings. Added fink package provider (#642), as provided by 'do'. Marked the dpkg package provider as versionable (#647). Applied patches by trombik to fix FreeBSD ports (#624 and #628). Fixed the CA server so that it refuses to send back a certificate whose public key doesn't match the CSR. Instead, it tells the user to run 'puppetca --clean'. Invalid certificates are no longer written to disk (#578). Added a package provider (appdmg) able to install .app packages on .dmg files on OS X (#641). Applied the patch from #667 to hopefully kill the client hanging problems (permanently, this time). Fixed functions so that they accept most other rvalues as valid values (#548). COMPATIBILITY ALERT: Significantly reworked external node support, in a way that's NOT backward-compatible: Only ONE node source can be used -- you can use LDAP, code, or an external node program, but not more than one. LDAP node support has two changes: First, the "ldapattrs" attribute is now used for setting the attributes to retrieve from the server (in addition to required attriutes), and second, all retrieved attributes are set as variables in the top scope. This means you can set attributes on your LDAP nodes and they will automatically appear as variables in your configurations. External node support has been completely rewritten. These programs must now generate a YAML dump of a hash, with "classes" and "parameters" keys. The classes should be an array, and the parameters should be a hash. The external node program has no support for parent nodes -- the script must handle that on its own. Reworked the database schema used to store configurations with the storeconfigs option. Replaced the obsolete RRD ruby library with the maintained RubyRRDtool library (which requires rrdtool2) (#659). The Portage package provider now calls eix-update automatically when eix's database is absent or out of sync (#666). Mounts now correctly handle existing fstabs with no pass or dump values (#550). Mounts now default to 0 for pass and dump (#112). Added urpmi support (#592). Finishing up the type => provider interface work. Basically, package providers now return lists of provider instances. In the proces, I rewrote the interface between package types and providers, and also enabled prefetching on all packages. This should significantly speed up most package operations. Hopefully fixing the file descriptor/open port problems, with patches from Valentin Vidic. Significantly reworked the type => provider interface with respect to listing existing provider instances. The class method on both class heirarchies has been renamed to 'instances', to start. Providers are now expected to return provider instances, instead of creating resources, and the resource's 'instances' method is expected to find the matching resource, if any, and set the resource's provider appropriately. This *significantly* reduces the reliance on effectively global state (resource references in the resource classes). This global state will go away soon. Along with this change, the 'prefetch' class method on providers now accepts the list of resources for prefetching. This again reduces reliance on global state, and makes the execution path much easier to follow. Fixed #532 -- reparsing config files now longer throws an exception. Added some warnings and logs to the service type so users will be encouraged to specify either "ensure" or "enabled" and added debugging to indicate why restarting is skipped when it is. Changed the location of the classes.txt to the state directory. Added better error reporting on unmatched brackets. Moved puppetd and puppetmasterd to sbin in svn and fixed install.rb to copy them into sbin on the local system appropriately. (#323) Added a splay option (#501). It's disabled when running under --test in puppetd. The value is random but cached. It defaults to the runinterval but can be tuned with --splaylimit Changing the notify type so that it always uses the loglevel. Fixing #568 - nodes can inherit from quoted node names. Tags (and thus definitions and classes) can now be a single character. (#566) Added an 'undef' keyword (#629), which will evaluate to "" within strings but when used as a resource parameter value will cause that parameter to be evaluated as undefined. Changed the topological sort algorithm (#507) so it will always fail on cycles. Added a 'dynamicfacts' configuration option; any facts in that comma-separated list will be ignored when comparing facts to see if they have changed and thus whether a recompile is necessary. Renamed some poorly named internal variables: @models in providers are now either @resource or @resource_type (#605). @children is no longer used except by components (#606). @parent is now @resource within parameters (#607). The old variables are still set for backward compatibility. Significantly reworking configuration parsing. Executables all now look for 'puppet.conf' (#206), although they will parse the old-style configuration files if they are present, although they throw a deprecation warning. Also, file parameters (owner, mode, group) are now set on the same line as the parameter, in brackets. (#422) Added transaction summaries (available with the --summarize option), useful for getting a quick idea of what happened in a transaction. Currently only useful on the client or with the puppet interpreter. Changed the interal workings for retrieve and removed the :is attribute from Property. The retrieve methods now return the current value of the property for the system. Removed acts_as_taggable from the rails models. 0.22.4 Execs now autorequire the user they run as, as long as the user is specified by name. (#430) Files on the local machine but not on the remote server during a source copy are now purged if purge => true. (#594) Providers can now specify that some commands are optional (#585). Also, the 'command' method returns nil on missing commands, rather than throwing an error, so the presence of commands be tested. The 'useradd' provider for Users can now manage passwords. No other providers can, at this point. Parameters can now declare a dependency on specific features, and parameters that require missing features will not be instantiated. This is most useful for properties. FileParsing classes can now use instance_eval to add many methods at once to a record type. Modules no longer return directories in the list of found manifests (#588). The crontab provider now defaults to root when there is no USER set in the environment. Puppetd once again correctly responds to HUP. Added a syntax for referring to variables defined in other classes (e.g., $puppet::server). STDIN, STDOUT, STDERR are now redirected to /dev/null in service providers descending from base. Certificates are now valid starting one day before they are created, to help handle small amounts of clock skew. Files are no longer considered out of sync if some properties are out of sync but they have no properties that can create the file. 0.22.3 Fixed backward compatibility for logs and metrics from older clients. Fixed the location of the authconfig parameters so there aren't loading order issues. Enabling attribute validation on the providers that subclass 'nameservice', so we can verify that an integer is passed to UID and GID. Added a stand-alone filebucket client, named 'filebucket'. Fixed the new nested paths for filebuckets; the entire md5 sum was not being stored. Fixing #553; -M is no longer added when home directories are being managed on Red Hat. 0.22.2 (grover) Users can now manage their home directories, using the managehome parameter, partially using patches provided by Tim Stoop and Matt Palmer. (#432) Added 'ralsh' (formerly x2puppet) to the svn tree. When possible it should be added to the packages. The 'notify' type now defaults to its message being the same as its name. Reopening $stdin to read from /dev/null during execution, in hopes that init scripts will stop hanging. Changed the 'servername' fact set on the server to use the server's fqdn, instead of the short-name. Changing the location of the configuration cache. It now defaults to being in the state directory, rather than in the configuration directory. All parameter instances are stored in a single @parameters instance variable hash within resource type instances. We used to use separate hashes for each parameter type. Added the concept of provider features. Eventually these should be able to express the full range of provider functionality, but for now they can test a provider to see what methods it has set and determine what features it provides as a result. These features are integrated into the doc generation system so that you get feature documentation automatically. Switched apt/aptitide to using "apt-cache policy" instead of "apt-cache showpkg" for determining the latest available version. (#487) FileBuckets now use a deeply nested structure for storing files, so you do not end up with hundreds or thousands of files in the same directory. (#447) Facts are now cached in the state file, and when they change the configuration is always recompiled. (#519) Added 'ignoreimport' setting for use in commit hooks. This causes the parser to ignore import statements so a single file can be parse-checked. (#544) Import statements can now specify multiple comma-separated arguments. Definitions now support both 'name' and 'title', just like any other resource type. (#539) Added a generate() command, which sets values to the result of an external command. (#541) Added a file() command to read in files with no interpolation. The first found file has its content returned. puppetd now exits if no cert is present in onetime mode. (#533) The client configuration cache can be safely removed and the client will correctly realize the client is not in sync. Resources can now be freely deleted, thus fixing many problems introduced when deletion of required resources was forbidden when purging was introduced. Only resources being purged will not be deleted. Facts and plugins now download even in noop mode (#540). Resources in noop mode now log when they would have responded to an event (#542). Refactored cron support entirely. Cron now uses providers, and there is a single 'crontab' provider that handles user crontabs. While this refactor does not include providers for /etc/crontab or cron.d, it should now be straightforward to write those providers. Changed the parameter sorting so that the provider parameter comes right after name, so the provider is available when the other parameters and properties are being created. Redid some of the internals of the ParsedFile provider base class. It now passes a FileRecord around instead of a hash. Fixing a bug related to link recursion that caused link directories to always be considered out of sync. The bind address for puppetmasterd can now be specified with --bindaddress. Added (probably experimental) mongrel support. At this point you're still responsible for starting each individual process, and you have to set up a proxy in front of it. Redesigned the 'network' tree to support multiple web servers, including refactoring most of the structural code so it's much clearer and more reusable now. Set up the CA client to default to ca_server and ca_port, so you can easily run a separate CA. Supporting hosts with no domain name, thanks to a patch from Dennis Jacobfeuerborn. Added an 'ignorecache' option to tell puppetd to force a recompile, thanks to a patch by Chris McEniry. Made up2date the default for RHEL < 4 and yum the default for the rest. The yum provider now supports versions. Case statements correctly match when multiple values are provided, thanks to a patch by David Schmitt. Functions can now be called with no arguments. String escapes parse correctly in all cases now, thanks to a patch by cstorey. Subclasses again search parent classes for defaults. You can now purge apt and dpkg packages. When doing file recursion, 'ensure' only affects the top-level directory. States have been renamed to Properties. 0.22.1 (kermit) -- Mostly a bugfix release Compile times now persist between restarts of puppetd. Timeouts have been added to many parts of Puppet, reducing the likelihood if it hanging forever on broken scripts or servers. All of the documentation and recipes have been moved to the wiki by Peter Abrahamsen and Ben Kite has moved the FAQ to the wiki. Explicit relationships now override automatic relationships, allowing you to manually specify deletion order when removing resources. Resources with dependencies can now be deleted as long as all of their dependencies are also being deleted. Namespaces for both classes and definitions now work much more consistently. You should now be able to specify a class or definition with a namespace everywhere you would normally expect to be able to specify one without. Downcasing of facts can be selectively disabled. Cyclic dependency graphs are now checked for and forbidden. The netinfo mounts provider was commented out, because it really doesn't work at all. Stupid NetInfo stores mount information with the device as the key, which doesn't work with my current NetInfo code. Otherwise, lots and lots of bugfixes. Check the tickets associated with the 'kermit' milestone. 0.22.0 Integrated the GRATR graph library into Puppet, for handling resource relationships. Lots of bug-fixes (see bugs tickets associated with the 'minor' milestone). Added new 'resources' metatype, which currently only includes the ability to purge unmanaged resources. Added better ability to generate new resource objects during transactions (using 'generate' and 'eval_generate' methods). Rewrote all Rails support with a much better database design. Export/collect now works, although the database is incompatible with previous versions. Removed downcasing of facts and made most of the language case-insensitive. Added support for printing the graphs built during transactions. Reworked how paths are built for logging. Switched all providers to directly executing commands instead of going through a subshell, which removes the need to quote or escape arguments. 0.20.1 Mostly a bug-fix release, with the most important fix being the multiple-definition error. Completely rewrote the ParsedFile system; each provider is now much shorter and much more maintainable. However, fundamental problems were found with the 'port' type, so it was disabled. Also, added a NetInfo provider for 'host' and an experimental NetInfo provider for 'mount'. Made the RRDGraph report *much* better and added reference generation for reports and functions. 0.20.0 Significantly refactored the parser. Resource overrides now consistently work anywhere in a class hierarchy. The language was also modified somewhat. The previous export/collect syntax is now used for handling virtual objects, and export/collect (which is still experimental) now uses double sigils (@@ and <<| |>>). Resource references (e.g., File["/etc/passwd"]) now have to be capitalized, in fitting in with capitalizing type operations. As usual, lots of other smaller fixes, but most of the work was in the language. 0.19.3 Fixing a bug in server/master.rb that causes the hostname not to be available in locally-executed manifests. 0.19.2 Fixing a few smaller bugs, notably in the reports system. Refreshed objects now generate an event, which can result in further refreshes of other objects. 0.19.1 Fixing two critical bugs: User management works again and cron jobs are no longer added to all user accounts. 0.19.0 Added provider support. Added support for %h, %H, and %d expansion in fileserver.conf. Added Certificate Revocation support. Made dynamic loading pervasive -- nearly every aspect of Puppet will now automatically load new instances (e.g., types, providers, and reports). Added support for automatic distribution of facts and plugins (custom types). 0.18.4 Another bug-fix release. The most import bug fixed is that cronjobs again work even with initially empty crontabs. 0.18.3 Mostly a bug-fix release; fixed small bugs in the functionality added in 0.18.2. 0.18.2 Added templating support. Added reporting. Added gem and blastwave packaging support. 0.18.1 Added signal handlers for HUP, so both client and server deal correctly with it. Added signal handler for USR1, which triggers a run on the client. As usual, fixed many bugs. Significant fixes to puppetrun -- it should behave much more correctly now. Added "fail" function which throws a syntax error if it's encountered. Added plugin downloading from the central server to the client. It must be enabled with --pluginsync. Added support for FreeBSD's special "@daily" cron schedules. Correctly handling spaces in file sources. Moved documentation into svn tree. 0.18.0 Added support for a "default" node. When multiple nodes are specified, they must now be comma-separated (this introduces a language incompatibility). Failed dependencies cause dependent objects within the same transaction not to run. Many updates to puppetrun Many bug fixes Function names are no longer reserved words. Links can now replace files. 0.17.2 Added "puppetrun" application and associated runner server and client classes. Fixed cron support so it better supports valid values and environment settings. 0.17.1 Fixing a bug requiring rails on all Debian boxes Fixing a couple of other small bugs 0.17.0 Adding ActiveRecord integration on the server Adding export/collect functionality Fixing many bugs 0.16.5 Fixing a critical bug in importing classes from other files Fixing nodename handling to actually allow dashes 0.16.4 Fixing a critical bug in puppetd when acquiring a certificate for the first time 0.16.3 Some significant bug fixes Modified puppetd so that it can now function as an agent independent of a puppetmasterd process, e.g., using the PuppetShow web application. 0.16.2 Modified some of the AST classes so that class names, definition names, and node names are all set within the code being evaluated, so 'tagged(name)' returns true while evaluating 'name', for instance. Added '--clean' argument to puppetca to remove all traces of a given client. 0.16.1 Added 'tagged' and 'defined' functions. Moved all functions to a general framework that makes it very easy to add new functions. 0.16.0 Added 'tag' keyword/function. Added FreeBSD Ports support Added 'pelement' server for sending or receiving Puppet objects, although none of the executables use it yet. 0.15.3 Fixed many bugs in :exec, including adding support for arrays of checks Added autoloading for types and service variants (e.g., you can now just create a new type in the appropriate location and use it in Puppet, without modifying the core Puppet libs). 0.15.2 Added darwinport, Apple .pkg, and freebsd package types Added 'mount type Host facts are now set at the top scope (Bug #103) Added -e (inline exection) flag to 'puppet' executable Many small bug fixes 0.15.1 Fixed 'yum' installs so that they successfully upgrade packages. Fixed puppetmasterd.conf file so group settings take. 0.15.0 Upped the minor release because the File server is incompatible with 0.14, because it now handles links. The 'symlink' type is deprecated (but still present), in favor of using files with the 'target' parameter. Unset variables no longer throw an error, they just return an empty string You can now specify tags to restrict which objects run during a given run. You can also specify to skip running against the cached copy when there's a failure, which is useful for testing new configurations. RPMs and Sun packages can now install, as long as they specify a package location, and they'll automatically upgrade if you point them to a new file with an upgrade. Multiple bug fixes. 0.14.1 Fixed a couple of small logging bugs Fixed a bug with handling group ownership of links 0.14.0 Added some ability to selectively manage symlinks when doing file management Many bug fixes Variables can now be used as the test values in case statements and selectors Bumping a minor release number because 0.13.4 introduced a protocol incompatibility and should have had a minor rev bump 0.13.6 Many, many small bug fixes FreeBSD user/group support has been added The configuration system has been rewritten so that daemons can now generate and repair the files and directories they need. (Fixed bug #68.) Fixed the element override issues; now only subclasses can override values. 0.13.5 Fixed packages so types can be specified Added 'enable' state to services, although it does not work everywhere yet 0.13.4 A few important bug fixes, mostly in the parser. 0.13.3 Changed transactions to be one-stage instead of two Changed all types to use self[:name] instead of self.name, to support the symbolic naming implemented in 0.13.1 0.13.2 Changed package[answerfile] to package[adminfile], and added package[responsefile] Fixed a bunch of internal functions to behave more consistently and usefully 0.13.1 Fixed RPM spec files to create puppet user and group (lutter) Fixed crontab reading and writing (luke) Added symbolic naming in the language (luke) 0.13.0 Added support for configuration files. Even more bug fixes, including the infamous 'frozen object' bug, which was a problem with 'waitforcert'. David Lutterkort got RPM into good shape. 0.12.0 Added Scheduling, and many bug fixes, of course. 0.11.2 Fixed bugs related to specifying arrays of requirements Fixed a key bug in retrieving checksums Fixed lots of usability bugs Added 'fail' methods that automatically add file and line info when possible, and converted many errors to use that method 0.11.1 Fixed bug with recursive copying with 'ignore' set. Added OpenBSD package support. 0.11.0 Added 'ensure' state to many elements. Modified puppetdoc to correctly handle indentation and such. Significantly rewrote much of the builtin documentation to take advantage of the new features in puppetdoc, including many examples. 0.10.2 Added SMF support Added autorequire functionality, with specific support for exec and file Exec elements autorequire any mentioned files, including the scripts, along with their CWDs. Files autorequire any parent directories. Added 'alias' metaparam. Fixed dependencies so they don't depend on file order. 0.10.1 Added Solaris package support and changed puppetmasterd to run as a non-root user. 0.10.0 Significant refactoring of how types, states, and parameters work, including breaking out parameters into a separate class. This refactoring did not introduce much new functionality, but made extension of Puppet significantly easier Also, fixed the bug with 'waitforcert' in puppetd. 0.9.4 Small fix to wrap the StatusServer class in the checks for required classes. 0.9.3 Fixed some significant bugs in cron job management. 0.9.2 Second Public Beta 0.9.0 First Public Beta diff --git a/ext/puppet-test b/ext/puppet-test index 117a5a11a..dc0aeca92 100755 --- a/ext/puppet-test +++ b/ext/puppet-test @@ -1,484 +1,485 @@ #!/usr/bin/env ruby # == Synopsis # # Test individual client performance. Can compile configurations, describe # files, or retrieve files. # # = Usage # # puppet-test [-c|--compile] [-D|--describe ] [-d|--debug] # [--fork ] [-h|--help] [-H|--hostname ] [-l|--list] [-r|--repeat ] # [-R|--retrieve ] [-t|--test ] [-V|--version] [-v|--verbose] # # = Description # # This is a simple script meant for doing performance tests with Puppet. By # default it pulls down a compiled configuration, but it supports multiple # other tests. # # = Options # # Note that any configuration parameter that's valid in the configuration file # is also a valid long argument. For example, 'server' is a valid configuration # parameter, so you can specify '--server ' as an argument. # # See the configuration file documentation at # http://reductivelabs.com/projects/puppet/reference/configref.html for # the full list of acceptable parameters. A commented list of all # configuration $options can also be generated by running puppetd with # '--genconfig'. # # compile:: # Compile the client's configuration. The default. # # debug:: # Enable full debugging. # # describe:: # Describe the file being tested. This is a query to get information about # the file from the server, to determine if it should be copied, and is the # first half of every file transfer. # # fork:: # Fork the specified number of times, thus acting as multiple clients. # # fqdn:: # Set the fully-qualified domain name of the client. This is only used for # certificate purposes, but can be used to override the discovered hostname. # If you need to use this flag, it is generally an indication of a setup problem. # # help:: # Print this help message # # list:: # List all available tests. # # pause:: # Pause before starting test (useful for testing with dtrace). # # repeat:: # How many times to perform the test. # # retrieve:: # Test file retrieval performance. Retrieves the specified file from the # remote system. Note that the server should be specified via --server, # so the argument to this option is just the remote module name and path, # e.g., "/dist/apps/myapp/myfile", where "dist" is the module and # "apps/myapp/myfile" is the path to the file relative to the module. # # test:: # Specify the test to run. You can see the list of tests by running this command with --list. # # verbose:: # Turn on verbose reporting. # # version:: # Print the puppet version number and exit. # # = Example # # puppet-test --retrieve /module/path/to/file # # = Author # # Luke Kanies # # = Copyright # # Copyright (c) 2005, 2006 Reductive Labs, LLC # Licensed under the GNU Public License # Do an initial trap, so that cancels don't get a stack trace. trap(:INT) do $stderr.puts "Cancelling startup" exit(1) end require 'puppet' require 'puppet/network/client' require 'getoptlong' class Suite attr_reader :name, :doc @@suites = {} @@tests = {} def self.[](name) @@suites[name] end # Run a test by first finding the suite then running the appropriate test. def self.run(test) unless suite_name = @@tests[test] raise "Unknown test %s" % test end unless suite = @@suites[suite_name] raise "Unknown test suite %s from test %s" % [suite_name, test] end suite.run(test) end # What suites are available? def self.suites @@suites.keys end def forked? defined? @forking end # Create a new test suite. def initialize(name, doc, &block) @name = name @doc = doc @tests = {} @@suites[name] = self raise "You must pass a block to the Test" unless block_given? instance_eval(&block) end # Define a new type of test on this suite. def newtest(name, doc, &block) @tests[name] = doc if @@tests[name] raise "Test names must be unique; cannot redefine %s" % name end @@tests[name] = @name meta_def(name, &block) end # Run the actual test. def run(test) unless doc = @tests[test] raise "Suite %s only supports tests %s; not %s" % [@name, @tests.keys.collect { |k| k.to_s }.join(","), test] end puts "Running %s %s test" % [@name, test] prepare() if respond_to?(:prepare) if $options[:pause] puts "Hit any key to continue" $stdin.readline puts "Continuing with test" end if $options[:fork] > 0 @forking = true $options[:fork].times { if pid = fork $pids << pid else break end } end $options[:repeat].times do |i| @count = i if forked? msg = doc + " in PID %s" % Process.pid else msg = doc end Puppet::Util.benchmark(:notice, msg) do begin send(test) rescue => detail puts detail.backtrace if Puppet[:trace] Puppet.err "%s failed: %s" % [@name, detail.to_s] end end end end # What tests are available on this suite? def tests @tests.keys end end Suite.new :parser, "Manifest parsing" do newtest :parse, "Parsed files" do @parser = Puppet::Parser::Parser.new(:environment => Puppet[:environment]) @parser.file = Puppet[:manifest] @parser.parse end end Suite.new :local_catalog, "Local catalog handling" do def prepare + @node = Puppet::Node.find(Puppet[:certname]) end newtest :compile, "Compiled catalog" do - Puppet::Node::Catalog.find(Puppet[:certname]) + Puppet::Node::Catalog.find(@node) end end Suite.new :remote_catalog, "Remote catalog handling" do def prepare $args[:cache] = false # Create a config client and pull the config down @client = Puppet::Network::Client.master.new($args) unless @client.read_cert fail "Could not read client certificate" end # Use the facts from the cache, to skip the time it takes # to load them. @client.dostorage @facts = Puppet::Util::Storage.cache(:configuration)[:facts] || {} if @facts.empty? if tmp = Puppet::Node::Facts.find("me") @facts = tmp.values end end if host = $options[:fqdn] @facts["fqdn"] = host @facts["hostname"] = host.sub(/\..+/, '') @facts["domain"] = host.sub(/^[^.]+\./, '') end @facts = YAML.dump(@facts) end newtest :getconfig, "Compiled catalog" do @client.driver.getconfig(@facts, "yaml") end # This test will always force a false answer. newtest :fresh, "Checked freshness" do @client.driver.freshness end end Suite.new :file, "File interactions" do def prepare unless $options[:file] fail "You must specify a file (using --file ) to interact with on the server" end @client = Puppet::Network::Client.file.new($args) unless @client.read_cert fail "Could not read client certificate" end end newtest :describe, "Described file" do @client.describe($options[:file], :ignore) end newtest :retrieve, "Retrieved file" do @client.retrieve($options[:file], :ignore) end end Suite.new :filebucket, "Filebucket interactions" do def prepare require 'tempfile' @client = Puppet::Network::Client.dipper.new($args) end newtest :backup, "Backed up file" do Tempfile.open("bucket_testing") do |f| f.print rand(1024) f.close @client.backup(f.path) end end end # Note that this uses an env variable to determine how many resources per # host to create (with a default of 10). 'repeat' determines how # many hosts to create. You probably will get mad failures if you # use forking and sqlite. # Here's an example run of this test, using sqlite: # RESOURCE_COUNT=50 ext/puppet-test --suite rails --test storage --confdir /tmp/storagetesting --vardir /tmp/storagetesting --repeat 10 Suite.new :rails, "Rails Interactions" do def prepare Puppet::Rails.init @facts = Facter.to_hash if num = ENV["RESOURCECOUNT"] @resources = Integer(num) else @resources = 10 end end Resource = Puppet::Parser::Resource def execute(test, msg) begin send(test) rescue => detail puts detail.backtrace if Puppet[:trace] Puppet.err "%s failed: %s" % [@name, detail.to_s] end end def mkresource(type, title, parameters) source = "fakesource" res = Resource.new(:type => type, :title => title, :source => source, :scope => "fakescope") parameters.each do |param, value| res.set(param, value, source) end res end def ref(type, title) Resource::Reference.new(:type => type, :title => title) end newtest :storage, "Stored resources" do hostname = "host%s" % @count @facts["hostname"] = hostname args = {:facts => @facts, :name => hostname} # Make all of the resources we want. Make them at least slightly complex, # so we model real resources as close as we can. resources = [] args[:resources] = resources @resources.times do |resnum| exec = mkresource("exec", "exec%s" % resnum, :command => "/bin/echo do something %s" % resnum) exec.tags = %w{exec one} << "exec%s" % resnum user = mkresource("user", "user%s" % resnum, :uid => resnum.to_s, :require => ref("exec", "exec%s" % resnum)) user.tags = %w{user one two} << "user%s" % resnum file = mkresource("file", "/tmp/file%s" % resnum, :owner => resnum.to_s, :require => [ref("exec", "exec%s" % resnum), ref("user", "user%s" % resnum)]) file.tags = %w{file one three} << "file%s" % resnum file.exported = true resources << exec << user << file end Puppet::Rails::Host.store(args) end end $cmdargs = [ [ "--compile", "-c", GetoptLong::NO_ARGUMENT ], [ "--describe", GetoptLong::REQUIRED_ARGUMENT ], [ "--retrieve", "-R", GetoptLong::REQUIRED_ARGUMENT ], [ "--fork", GetoptLong::REQUIRED_ARGUMENT ], [ "--fqdn", "-F", GetoptLong::REQUIRED_ARGUMENT ], [ "--suite", "-s", GetoptLong::REQUIRED_ARGUMENT ], [ "--test", "-t", GetoptLong::REQUIRED_ARGUMENT ], [ "--pause", "-p", GetoptLong::NO_ARGUMENT ], [ "--repeat", "-r", GetoptLong::REQUIRED_ARGUMENT ], [ "--debug", "-d", GetoptLong::NO_ARGUMENT ], [ "--help", "-h", GetoptLong::NO_ARGUMENT ], [ "--list", "-l", GetoptLong::NO_ARGUMENT ], [ "--verbose", "-v", GetoptLong::NO_ARGUMENT ], [ "--version", "-V", GetoptLong::NO_ARGUMENT ], ] # Add all of the config parameters as valid $options. Puppet.settings.addargs($cmdargs) Puppet::Util::Log.newdestination(:console) result = GetoptLong.new(*$cmdargs) $args = {} $options = {:repeat => 1, :fork => 0, :pause => false} begin explicit_waitforcert = false result.each { |opt,arg| case opt # First check to see if the argument is a valid configuration parameter; # if so, set it. when "--compile" $options[:suite] = :configuration $options[:test] = :compile when "--retrieve" $options[:suite] = :file $options[:test] = :retrieve $options[:file] = arg when "--fork" begin $options[:fork] = Integer(arg) rescue => detail $stderr.puts "The argument to 'fork' must be an integer" exit(14) end when "--describe" $options[:suite] = :file $options[:test] = :describe $options[:file] = arg when "--fqdn" $options[:fqdn] = arg when "--repeat" $options[:repeat] = Integer(arg) when "--help" if Puppet.features.usage? RDoc::usage && exit else puts "No help available unless you have RDoc::usage installed" exit end when "--version" puts "%s" % Puppet.version exit when "--verbose" Puppet::Util::Log.level = :info Puppet::Util::Log.newdestination(:console) when "--debug" Puppet::Util::Log.level = :debug Puppet::Util::Log.newdestination(:console) when "--suite" $options[:suite] = arg.intern when "--test" $options[:test] = arg.intern when "--file" $options[:file] = arg when "--pause" $options[:pause] = true when "--list" Suite.suites.sort { |a,b| a.to_s <=> b.to_s }.each do |suite_name| suite = Suite[suite_name] tests = suite.tests.sort { |a,b| a.to_s <=> b.to_s }.join(", ") puts "%20s: %s" % [suite_name, tests] end exit(0) else Puppet.settings.handlearg(opt, arg) end } rescue GetoptLong::InvalidOption => detail $stderr.puts detail $stderr.puts "Try '#{$0} --help'" exit(1) end # Now parse the config Puppet.parse_config $args[:Server] = Puppet[:server] unless $options[:test] $options[:suite] = :remote_catalog $options[:test] = :getconfig end unless $options[:test] raise "A suite was specified without a test" end $pids = [] Suite.run($options[:test]) if $options[:fork] > 0 Process.waitall end # $Id$ diff --git a/lib/puppet/file_serving/configuration/parser.rb b/lib/puppet/file_serving/configuration/parser.rb index 707c3f9b1..cda6889d4 100644 --- a/lib/puppet/file_serving/configuration/parser.rb +++ b/lib/puppet/file_serving/configuration/parser.rb @@ -1,124 +1,126 @@ require 'puppet/file_serving/configuration' require 'puppet/util/loadedfile' class Puppet::FileServing::Configuration::Parser < Puppet::Util::LoadedFile Mount = Puppet::FileServing::Mount MODULES = 'modules' # Parse our configuration file. def parse raise("File server configuration %s does not exist" % self.file) unless FileTest.exists?(self.file) raise("Cannot read file server configuration %s" % self.file) unless FileTest.readable?(self.file) @mounts = {} @count = 0 File.open(self.file) { |f| mount = nil f.each { |line| # Have the count increment at the top, in case we throw exceptions. @count += 1 case line when /^\s*#/: next # skip comments when /^\s*$/: next # skip blank lines when /\[([-\w]+)\]/: mount = newmount($1) when /^\s*(\w+)\s+(.+)$/: var = $1 value = $2 raise(ArgumentError, "Fileserver configuration file does not use '=' as a separator") if value =~ /^=/ case var when "path": path(mount, value) when "allow": allow(mount, value) when "deny": deny(mount, value) else raise ArgumentError.new("Invalid argument '%s'" % var, @count, file) end else raise ArgumentError.new("Invalid line '%s'" % line.chomp, @count, file) end } } return @mounts end private # Add the mount for getting files from modules. def add_module_mount unless @mounts[MODULES] mount = Mount.new(MODULES) mount.allow("*") @mounts[MODULES] = mount end end # Allow a given pattern access to a mount. def allow(mount, value) - value.split(/\s*,\s*/).each { |val| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = value.split(/\s*,\s*/).each { |val| begin mount.info "allowing %s access" % val mount.allow(val) rescue AuthStoreError => detail raise ArgumentError.new(detail.to_s, @count, file) end } end # Deny a given pattern access to a mount. def deny(mount, value) - value.split(/\s*,\s*/).each { |val| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = value.split(/\s*,\s*/).each { |val| begin mount.info "denying %s access" % val mount.deny(val) rescue AuthStoreError => detail raise ArgumentError.new(detail.to_s, @count, file) end } end # Create a new mount. def newmount(name) if @mounts.include?(name) raise ArgumentError, "%s is already mounted at %s" % [@mounts[name], name], @count, file end mount = Mount.new(name) @mounts[name] = mount return mount end # Set the path for a mount. def path(mount, value) if mount.name == MODULES Puppet.warning "The '#{MODULES}' module can not have a path. Ignoring attempt to set it" else begin mount.path = value rescue ArgumentError => detail Puppet.err "Removing mount %s: %s" % [mount.name, detail] @mounts.delete(mount.name) end end end # Make sure all of our mounts are valid. We have to do this after the fact # because details are added over time as the file is parsed. def validate @mounts.each { |name, mount| unless mount.valid? raise ArgumentError, "No path specified for mount %s" % name end } end end diff --git a/lib/puppet/indirector/facts/facter.rb b/lib/puppet/indirector/facts/facter.rb index 5b9a7b3c0..a8c47e3bf 100644 --- a/lib/puppet/indirector/facts/facter.rb +++ b/lib/puppet/indirector/facts/facter.rb @@ -1,65 +1,66 @@ require 'puppet/node/facts' require 'puppet/indirector/code' class Puppet::Node::Facts::Facter < Puppet::Indirector::Code desc "Retrieve facts from Facter. This provides a somewhat abstract interface between Puppet and Facter. It's only `somewhat` abstract because it always returns the local host's facts, regardless of what you attempt to find." def self.loaddir(dir, type) return unless FileTest.directory?(dir) Dir.entries(dir).find_all { |e| e =~ /\.rb$/ }.each do |file| fqfile = ::File.join(dir, file) begin Puppet.info "Loading #{type} %s" % ::File.basename(file.sub(".rb",'')) Timeout::timeout(self.timeout) do load fqfile end rescue => detail Puppet.warning "Could not load #{type} %s: %s" % [fqfile, detail] end end end def self.loadfacts - Puppet[:factpath].split(":").each do |dir| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet[:factpath].split(":").each do |dir| loaddir(dir, "fact") end end def self.timeout timeout = Puppet[:configtimeout] case timeout when String: if timeout =~ /^\d+$/ timeout = Integer(timeout) else raise ArgumentError, "Configuration timeout must be an integer" end when Integer: # nothing else raise ArgumentError, "Configuration timeout must be an integer" end return timeout end def initialize(*args) super self.class.loadfacts end def destroy(facts) raise Puppet::DevError, "You cannot destroy facts in the code store; it is only used for getting facts from Facter" end # Look a host's facts up in Facter. def find(key) Puppet::Node::Facts.new(key, Facter.to_hash) end def save(facts) raise Puppet::DevError, "You cannot save facts to the code store; it is only used for getting facts from Facter" end end diff --git a/lib/puppet/indirector/node/ldap.rb b/lib/puppet/indirector/node/ldap.rb index 8537e1cf3..73b5cdd70 100644 --- a/lib/puppet/indirector/node/ldap.rb +++ b/lib/puppet/indirector/node/ldap.rb @@ -1,129 +1,130 @@ require 'puppet/node' require 'puppet/indirector/ldap' class Puppet::Node::Ldap < Puppet::Indirector::Ldap desc "Search in LDAP for node configuration information. See the `LdapNodes`:trac: page for more information." # The attributes that Puppet class information is stored in. def class_attributes - Puppet[:ldapclassattrs].split(/\s*,\s*/) + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet[:ldapclassattrs].split(/\s*,\s*/) end # Look for our node in ldap. def find(name) return nil unless information = super node = Puppet::Node.new(name) parent_info = nil parent = information[:parent] parents = [name] while parent if parents.include?(parent) raise ArgumentError, "Found loop in LDAP node parents; %s appears twice" % parent end parents << parent ldapsearch(parent) { |entry| parent_info = process(parent, entry) } unless parent_info raise Puppet::Error.new("Could not find parent node '%s'" % parent) end information[:classes] += parent_info[:classes] parent_info[:parameters].each do |param, value| # Specifically test for whether it's set, so false values are handled # correctly. information[:parameters][param] = value unless information[:parameters].include?(param) end information[:environment] ||= parent_info[:environment] parent = parent_info[:parent] end node.classes = information[:classes].uniq unless information[:classes].empty? node.parameters = information[:parameters] unless information[:parameters].empty? node.environment = information[:environment] if information[:environment] node.fact_merge return node end # The parent attribute, if we have one. def parent_attribute if pattr = Puppet[:ldapparentattr] and ! pattr.empty? pattr else nil end end # Process the found entry. We assume that we don't just want the # ldap object. def process(name, entry) result = {} if pattr = parent_attribute if values = entry.vals(pattr) if values.length > 1 raise Puppet::Error, "Node %s has more than one parent: %s" % [name, values.inspect] end unless values.empty? result[:parent] = values.shift end end end result[:classes] = [] class_attributes.each { |attr| if values = entry.vals(attr) values.each do |v| result[:classes] << v end end } result[:parameters] = entry.to_hash.inject({}) do |hash, ary| if ary[1].length == 1 hash[ary[0]] = ary[1].shift else hash[ary[0]] = ary[1] end hash end result[:environment] = result[:parameters]["environment"] if result[:parameters]["environment"] return result end # Default to all attributes. def search_attributes ldapattrs = Puppet[:ldapattrs] # results in everything getting returned return nil if ldapattrs == "all" search_attrs = class_attributes + ldapattrs.split(/\s*,\s*/) if pattr = parent_attribute search_attrs << pattr end search_attrs end # The ldap search filter to use. def search_filter(name) filter = Puppet[:ldapstring] if filter.include? "%s" # Don't replace the string in-line, since that would hard-code our node # info. filter = filter.gsub('%s', name) end filter end def version(name) Puppet::Node::Facts.version(name) end end diff --git a/lib/puppet/indirector/report/processor.rb b/lib/puppet/indirector/report/processor.rb index fd1bc413a..fa2b7f36b 100644 --- a/lib/puppet/indirector/report/processor.rb +++ b/lib/puppet/indirector/report/processor.rb @@ -1,50 +1,51 @@ require 'puppet/transaction/report' require 'puppet/indirector/code' require 'puppet/reports' class Puppet::Transaction::Report::Processor < Puppet::Indirector::Code desc "Puppet's report processor. Processes the report with each of the report types listed in the 'reports' setting." def initialize Puppet.settings.use(:main, :reporting, :metrics) end def save(report) process(report) end private # Process the report with each of the configured report types. # LAK:NOTE This isn't necessarily the best design, but it's backward # compatible and that's good enough for now. def process(report) return if Puppet[:reports] == "none" reports().each do |name| if mod = Puppet::Reports.report(name) # We have to use a dup because we're including a module in the # report. newrep = report.dup begin newrep.extend(mod) newrep.process rescue => detail if Puppet[:trace] puts detail.backtrace end Puppet.err "Report %s failed: %s" % [name, detail] end else Puppet.warning "No report named '%s'" % name end end end # Handle the parsing of the reports attribute. def reports - Puppet[:reports].gsub(/(^\s+)|(\s+$)/, '').split(/\s*,\s*/) + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet[:reports].gsub(/(^\s+)|(\s+$)/, '').split(/\s*,\s*/) end end diff --git a/lib/puppet/network/authstore.rb b/lib/puppet/network/authstore.rb index c16bef32f..cb1fdc57d 100755 --- a/lib/puppet/network/authstore.rb +++ b/lib/puppet/network/authstore.rb @@ -1,296 +1,297 @@ # standard module for determining whether a given hostname or IP has access to # the requested resource require 'ipaddr' require 'puppet/util/logging' module Puppet class AuthStoreError < Puppet::Error; end class AuthorizationError < Puppet::Error; end class Network::AuthStore include Puppet::Util::Logging # Mark a given pattern as allowed. def allow(pattern) # a simple way to allow anyone at all to connect if pattern == "*" @globalallow = true else store(:allow, pattern) end return nil end # Is a given combination of name and ip address allowed? If either input # is non-nil, then both inputs must be provided. If neither input # is provided, then the authstore is considered local and defaults to "true". def allowed?(name, ip) if name or ip # This is probably unnecessary, and can cause some weirdnesses in # cases where we're operating over localhost but don't have a real # IP defined. unless name and ip raise Puppet::DevError, "Name and IP must be passed to 'allowed?'" end # else, we're networked and such else # we're local return true end # yay insecure overrides if globalallow? return true end if decl = @declarations.find { |d| d.match?(name, ip) } return decl.result end self.info "defaulting to no access for %s" % name return false end # Deny a given pattern. def deny(pattern) store(:deny, pattern) end # Is global allow enabled? def globalallow? @globalallow end def initialize @globalallow = nil @declarations = [] end def to_s "authstore" end private # Store the results of a pattern into our hash. Basically just # converts the pattern and sticks it into the hash. def store(type, pattern) @declarations << Declaration.new(type, pattern) @declarations.sort! return nil end # A single declaration. Stores the info for a given declaration, # provides the methods for determining whether a declaration matches, # and handles sorting the declarations appropriately. class Declaration include Puppet::Util include Comparable # The type of declaration: either :allow or :deny attr_reader :type # The name: :ip or :domain attr_accessor :name # The pattern we're matching against. Can be an IPAddr instance, # or an array of strings, resulting from reversing a hostname # or domain name. attr_reader :pattern # The length. Only used for iprange and domain. attr_accessor :length # Sort the declarations specially. def <=>(other) # Sort first based on whether the matches are exact. if r = compare(exact?, other.exact?) return r end # Then by type if r = compare(self.ip?, other.ip?) return r end # Next sort based on length unless self.length == other.length # Longer names/ips should go first, because they're more # specific. return other.length <=> self.length end # Then sort deny before allow if r = compare(self.deny?, other.deny?) return r end # We've already sorted by name and length, so all that's left # is the pattern if ip? return self.pattern.to_s <=> other.pattern.to_s else return self.pattern <=> other.pattern end end def deny? self.type == :deny end # Are we an exact match? def exact? self.length.nil? end def initialize(type, pattern) self.type = type self.pattern = pattern end # Are we an IP type? def ip? self.name == :ip end # Does this declaration match the name/ip combo? def match?(name, ip) if self.ip? return pattern.include?(IPAddr.new(ip)) else return matchname?(name) end end # Set the pattern appropriately. Also sets the name and length. def pattern=(pattern) parse(pattern) @orig = pattern end # Mapping a type of statement into a return value. def result case @type when :allow: true else false end end def to_s "%s: %s" % [self.type, self.pattern] end # Set the declaration type. Either :allow or :deny. def type=(type) type = symbolize(type) unless [:allow, :deny].include?(type) raise ArgumentError, "Invalid declaration type %s" % type end @type = type end private # Returns nil if both values are true or both are false, returns # -1 if the first is true, and 1 if the second is true. Used # in the <=> operator. def compare(me, them) unless me and them if me return -1 elsif them return 1 else return false end end return nil end # Does the name match our pattern? def matchname?(name) name = munge_name(name) return true if self.pattern == name # If it's an exact match, then just return false, since the # exact didn't match. if exact? return false end # If every field in the pattern matches, then we consider it # a match. pattern.zip(name) do |p,n| unless p == n return false end end return true end # Convert the name to a common pattern. def munge_name(name) - name.downcase.split(".").reverse + # LAK:NOTE http://snurl.com/21zf8 [groups_google_com] + x = name.downcase.split(".").reverse end # Parse our input pattern and figure out what kind of allowal # statement it is. The output of this is used for later matching. def parse(value) case value when /^(\d+\.){1,3}\*$/: # an ip address with a '*' at the end @name = :ip match = $1 match.sub!(".", '') ary = value.split(".") mask = case ary.index(match) when 0: 8 when 1: 16 when 2: 24 else raise AuthStoreError, "Invalid IP pattern %s" % value end @length = mask ary.pop while ary.length < 4 ary.push("0") end begin @pattern = IPAddr.new(ary.join(".") + "/" + mask.to_s) rescue ArgumentError => detail raise AuthStoreError, "Invalid IP address pattern %s" % value end when /^([a-zA-Z][-\w]*\.)+[-\w]+$/: # a full hostname @name = :domain @pattern = munge_name(value) when /^\*(\.([a-zA-Z][-\w]*)){1,}$/: # *.domain.com @name = :domain @pattern = munge_name(value) @pattern.pop # take off the '*' @length = @pattern.length else # Else, use the IPAddr class to determine if we've got a # valid IP address. if value =~ /\/(\d+)$/ @length = Integer($1) end begin @pattern = IPAddr.new(value) rescue ArgumentError => detail raise AuthStoreError, "Invalid pattern %s" % value end @name = :ip end end end end end diff --git a/lib/puppet/network/client/master.rb b/lib/puppet/network/client/master.rb index e914d5c69..955acbfb5 100644 --- a/lib/puppet/network/client/master.rb +++ b/lib/puppet/network/client/master.rb @@ -1,573 +1,575 @@ # The client for interacting with the puppetmaster config server. require 'sync' require 'timeout' require 'puppet/network/http_pool' class Puppet::Network::Client::Master < Puppet::Network::Client unless defined? @@sync @@sync = Sync.new end attr_accessor :catalog attr_reader :compile_time class << self # Puppetd should only have one instance running, and we need a way # to retrieve it. attr_accessor :instance include Puppet::Util end def self.facts # Retrieve the facts from the central server. if Puppet[:factsync] self.getfacts() end down = Puppet[:downcasefacts] facts = Facter.to_hash.inject({}) do |newhash, array| name, fact = array if down newhash[name] = fact.to_s.downcase else newhash[name] = fact.to_s end newhash end # Add our client version to the list of facts, so people can use it # in their manifests facts["clientversion"] = Puppet.version.to_s # And add our environment as a fact. unless facts.include?("environment") facts["environment"] = Puppet[:environment] end facts end # Return the list of dynamic facts as an array of symbols def self.dynamic_facts - Puppet.settings[:dynamicfacts].split(/\s*,\s*/).collect { |fact| fact.downcase } + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet.settings[:dynamicfacts].split(/\s*,\s*/).collect { |fact| fact.downcase } end # Cache the config def cache(text) Puppet.info "Caching catalog at %s" % self.cachefile confdir = ::File.dirname(Puppet[:localconfig]) ::File.open(self.cachefile + ".tmp", "w", 0660) { |f| f.print text } ::File.rename(self.cachefile + ".tmp", self.cachefile) end def cachefile unless defined? @cachefile @cachefile = Puppet[:localconfig] + ".yaml" end @cachefile end def clear @catalog.clear(true) if @catalog Puppet::Type.allclear @catalog = nil end # Initialize and load storage def dostorage begin Puppet::Util::Storage.load @compile_time ||= Puppet::Util::Storage.cache(:configuration)[:compile_time] rescue => detail if Puppet[:trace] puts detail.backtrace end Puppet.err "Corrupt state file %s: %s" % [Puppet[:statefile], detail] begin ::File.unlink(Puppet[:statefile]) retry rescue => detail raise Puppet::Error.new("Cannot remove %s: %s" % [Puppet[:statefile], detail]) end end end # Check whether our catalog is up to date def fresh?(facts) if Puppet[:ignorecache] Puppet.notice "Ignoring cache" return false end unless self.compile_time Puppet.debug "No cached compile time" return false end if facts_changed?(facts) Puppet.info "Facts have changed; recompiling" unless local? return false end newcompile = @driver.freshness # We're willing to give a 2 second drift if newcompile - @compile_time.to_i < 1 return true else Puppet.debug "Server compile time is %s vs %s" % [newcompile, @compile_time.to_i] return false end end # Let the daemon run again, freely in the filesystem. Frolick, little # daemon! def enable lockfile.unlock(:anonymous => true) end # Stop the daemon from making any catalog runs. def disable lockfile.lock(:anonymous => true) end # Retrieve the config from a remote server. If this fails, then # use the cached copy. def getconfig dostorage() facts = nil Puppet::Util.benchmark(:debug, "Retrieved facts") do facts = self.class.facts end raise Puppet::Network::ClientError.new("Could not retrieve any facts") unless facts.length > 0 # Retrieve the plugins. getplugins() if Puppet[:pluginsync] if (self.catalog or FileTest.exist?(self.cachefile)) and self.fresh?(facts) Puppet.info "Configuration is up to date" return if use_cached_config end Puppet.debug("Retrieving catalog") # If we can't retrieve the catalog, just return, which will either # fail, or use the in-memory catalog. unless yaml_objects = get_actual_config(facts) use_cached_config(true) return end begin objects = YAML.load(yaml_objects) rescue => detail msg = "Configuration could not be translated from yaml" msg += "; using cached catalog" if use_cached_config(true) Puppet.warning msg return end self.setclasses(objects.classes) # Clear all existing objects, so we can recreate our stack. clear() if self.catalog # Now convert the objects to a puppet catalog graph. begin @catalog = objects.to_catalog rescue => detail clear() puts detail.backtrace if Puppet[:trace] msg = "Configuration could not be instantiated: %s" % detail msg += "; using cached catalog" if use_cached_config(true) Puppet.warning msg return end if ! @catalog.from_cache self.cache(yaml_objects) end # Keep the state database up to date. @catalog.host_config = true end # A simple proxy method, so it's easy to test. def getplugins self.class.getplugins end # Just so we can specify that we are "the" instance. def initialize(*args) Puppet.settings.use(:main, :ssl, :puppetd) super self.class.instance = self @running = false end # Mark that we should restart. The Puppet module checks whether we're running, # so this only gets called if we're in the middle of a run. def restart # If we're currently running, then just mark for later Puppet.notice "Received signal to restart; waiting until run is complete" @restart = true end # Should we restart? def restart? if defined? @restart @restart else false end end # Retrieve the cached config def retrievecache if FileTest.exists?(self.cachefile) return ::File.read(self.cachefile) else return nil end end # The code that actually runs the catalog. # This just passes any options on to the catalog, # which accepts :tags and :ignoreschedules. def run(options = {}) got_lock = false splay Puppet::Util.sync(:puppetrun).synchronize(Sync::EX) do if !lockfile.lock Puppet.notice "Lock file %s exists; skipping catalog run" % lockfile.lockfile else got_lock = true begin duration = thinmark do self.getconfig end rescue => detail puts detail.backtrace if Puppet[:trace] Puppet.err "Could not retrieve catalog: %s" % detail end if self.catalog @catalog.retrieval_duration = duration Puppet.notice "Starting catalog run" unless @local benchmark(:notice, "Finished catalog run") do @catalog.apply(options) end end # Now close all of our existing http connections, since there's no # reason to leave them lying open. Puppet::Network::HttpPool.clear_http_instances end lockfile.unlock # Did we get HUPped during the run? If so, then restart now that we're # done with the run. if self.restart? Process.kill(:HUP, $$) end end ensure # Just make sure we remove the lock file if we set it. lockfile.unlock if got_lock and lockfile.locked? clear() end def running? lockfile.locked? end # Store the classes in the classfile, but only if we're not local. def setclasses(ary) if @local return end unless ary and ary.length > 0 Puppet.info "No classes to store" return end begin ::File.open(Puppet[:classfile], "w") { |f| f.puts ary.join("\n") } rescue => detail Puppet.err "Could not create class file %s: %s" % [Puppet[:classfile], detail] end end private # Download files from the remote server, returning a list of all # changed files. def self.download(args) hash = { :path => args[:dest], :recurse => true, :source => args[:source], :tag => "#{args[:name]}s", :owner => Process.uid, :group => Process.gid, :purge => true, :force => true, :backup => false, :noop => false } if args[:ignore] hash[:ignore] = args[:ignore].split(/\s+/) end downconfig = Puppet::Node::Catalog.new("downloading") downconfig.add_resource Puppet::Type.type(:file).create(hash) Puppet.info "Retrieving #{args[:name]}s" files = [] begin Timeout::timeout(self.timeout) do downconfig.apply do |trans| trans.changed?.find_all do |resource| yield resource if block_given? files << resource[:path] end end end rescue Puppet::Error, Timeout::Error => detail if Puppet[:debug] puts detail.backtrace end Puppet.err "Could not retrieve #{args[:name]}s: %s" % detail end # Now clean up after ourselves downconfig.clear return files end # Retrieve facts from the central server. def self.getfacts # Download the new facts path = Puppet[:factpath].split(":") files = [] download(:dest => Puppet[:factdest], :source => Puppet[:factsource], :ignore => Puppet[:factsignore], :name => "fact") do |resource| next unless path.include?(::File.dirname(resource[:path])) files << resource[:path] end ensure # Clear all existing definitions. Facter.clear # Reload everything. if Facter.respond_to? :loadfacts Facter.loadfacts elsif Facter.respond_to? :load Facter.load else raise Puppet::Error, "You must upgrade your version of Facter to use centralized facts" end # This loads all existing facts and any new ones. We have to remove and # reload because there's no way to unload specific facts. loadfacts() end # Retrieve the plugins from the central server. We only have to load the # changed plugins, because Puppet::Type loads plugins on demand. def self.getplugins download(:dest => Puppet[:plugindest], :source => Puppet[:pluginsource], :ignore => Puppet[:pluginsignore], :name => "plugin") do |resource| next if FileTest.directory?(resource[:path]) path = resource[:path].sub(Puppet[:plugindest], '').sub(/^\/+/, '') unless Puppet::Util::Autoload.loaded?(path) next end begin Puppet.info "Reloading downloaded file %s" % path load resource[:path] rescue => detail Puppet.warning "Could not reload downloaded file %s: %s" % [resource[:path], detail] end end end def self.loaddir(dir, type) return unless FileTest.directory?(dir) Dir.entries(dir).find_all { |e| e =~ /\.rb$/ }.each do |file| fqfile = ::File.join(dir, file) begin Puppet.info "Loading #{type} %s" % ::File.basename(file.sub(".rb",'')) Timeout::timeout(self.timeout) do load fqfile end rescue => detail Puppet.warning "Could not load #{type} %s: %s" % [fqfile, detail] end end end def self.loadfacts - Puppet[:factpath].split(":").each do |dir| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet[:factpath].split(":").each do |dir| loaddir(dir, "fact") end end def self.timeout timeout = Puppet[:configtimeout] case timeout when String: if timeout =~ /^\d+$/ timeout = Integer(timeout) else raise ArgumentError, "Configuration timeout must be an integer" end when Integer: # nothing else raise ArgumentError, "Configuration timeout must be an integer" end return timeout end loadfacts() # Have the facts changed since we last compiled? def facts_changed?(facts) oldfacts = (Puppet::Util::Storage.cache(:configuration)[:facts] || {}).dup newfacts = facts.dup self.class.dynamic_facts.each do |fact| [oldfacts, newfacts].each do |facthash| facthash.delete(fact) if facthash.include?(fact) end end if oldfacts == newfacts return false else # unless oldfacts # puts "no old facts" # return true # end # newfacts.keys.each do |k| # unless newfacts[k] == oldfacts[k] # puts "%s: %s vs %s" % [k, newfacts[k], oldfacts[k]] # end # end return true end end # Actually retrieve the catalog, either from the server or from a # local master. def get_actual_config(facts) begin Timeout::timeout(self.class.timeout) do return get_remote_config(facts) end rescue Timeout::Error Puppet.err "Configuration retrieval timed out" return nil end end # Retrieve a config from a remote master. def get_remote_config(facts) textobjects = "" textfacts = CGI.escape(YAML.dump(facts)) benchmark(:debug, "Retrieved catalog") do # error handling for this is done in the network client begin textobjects = @driver.getconfig(textfacts, "yaml") begin textobjects = CGI.unescape(textobjects) rescue => detail raise Puppet::Error, "Could not CGI.unescape catalog" end rescue => detail Puppet.err "Could not retrieve catalog: %s" % detail return nil end end return nil if textobjects == "" @compile_time = Time.now Puppet::Util::Storage.cache(:configuration)[:facts] = facts Puppet::Util::Storage.cache(:configuration)[:compile_time] = @compile_time return textobjects end def lockfile unless defined?(@lockfile) @lockfile = Puppet::Util::Pidlock.new(Puppet[:puppetdlockfile]) end @lockfile end # Sleep when splay is enabled; else just return. def splay return unless Puppet[:splay] limit = Integer(Puppet[:splaylimit]) # Pick a splay time and then cache it. unless time = Puppet::Util::Storage.cache(:configuration)[:splay_time] time = rand(limit) Puppet::Util::Storage.cache(:configuration)[:splay_time] = time end Puppet.info "Sleeping for %s seconds (splay is enabled)" % time sleep(time) end private # Use our cached config, optionally specifying whether this is # necessary because of a failure. def use_cached_config(because_of_failure = false) return true if self.catalog if because_of_failure and ! Puppet[:usecacheonfailure] @catalog = nil Puppet.warning "Not using cache on failed catalog" return false end return false unless oldtext = self.retrievecache begin @catalog = YAML.load(oldtext).to_catalog @catalog.from_cache = true @catalog.host_config = true rescue => detail puts detail.backtrace if Puppet[:trace] Puppet.warning "Could not load cached catalog: %s" % detail clear return false end return true end end diff --git a/lib/puppet/network/handler/report.rb b/lib/puppet/network/handler/report.rb index 8ddeed9f6..b92b77ea5 100755 --- a/lib/puppet/network/handler/report.rb +++ b/lib/puppet/network/handler/report.rb @@ -1,86 +1,87 @@ require 'puppet/util/instance_loader' require 'puppet/reports' # A simple server for triggering a new run on a Puppet client. class Puppet::Network::Handler class Report < Handler desc "Accepts a Puppet transaction report and processes it." @interface = XMLRPC::Service::Interface.new("puppetreports") { |iface| iface.add_method("string report(array)") } # Add a new report type. def self.newreport(name, options = {}, &block) Puppet.warning "The interface for registering report types has changed; use Puppet::Reports.register_report for report type %s" % name Puppet::Reports.register_report(name, options, &block) end def initialize(*args) super Puppet.settings.use(:main, :reporting, :metrics) end # Accept a report from a client. def report(report, client = nil, clientip = nil) # Unescape the report unless @local report = CGI.unescape(report) end Puppet.info "Processing reports %s for %s" % [reports().join(", "), client] begin process(report) rescue => detail Puppet.err "Could not process report for %s: %s" % [client, detail] if Puppet[:trace] puts detail.backtrace end end end private # Process the report using all of the existing hooks. def process(yaml) return if Puppet[:reports] == "none" # First convert the report to real objects begin report = YAML.load(yaml) rescue => detail Puppet.warning "Could not load report: %s" % detail return end # Used for those reports that accept yaml client = report.host reports().each do |name| if mod = Puppet::Reports.report(name) # We have to use a dup because we're including a module in the # report. newrep = report.dup begin newrep.extend(mod) newrep.process rescue => detail if Puppet[:trace] puts detail.backtrace end Puppet.err "Report %s failed: %s" % [name, detail] end else Puppet.warning "No report named '%s'" % name end end end # Handle the parsing of the reports attribute. def reports - Puppet[:reports].gsub(/(^\s+)|(\s+$)/, '').split(/\s*,\s*/) + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet[:reports].gsub(/(^\s+)|(\s+$)/, '').split(/\s*,\s*/) end end end diff --git a/lib/puppet/network/http/mongrel/rest.rb b/lib/puppet/network/http/mongrel/rest.rb index db63613ab..6c24e360c 100644 --- a/lib/puppet/network/http/mongrel/rest.rb +++ b/lib/puppet/network/http/mongrel/rest.rb @@ -1,37 +1,39 @@ require 'puppet/network/http/handler' class Puppet::Network::HTTP::MongrelREST < Puppet::Network::HTTP::Handler private def register_handler @server.register('/' + @handler.to_s, self) @server.register('/' + @handler.to_s + 's', self) end def http_method(request) request.params[Mongrel::Const::REQUEST_METHOD] end def path(request) - '/' + request.params[Mongrel::Const::REQUEST_PATH].split('/')[1] + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = '/' + request.params[Mongrel::Const::REQUEST_PATH].split('/')[1] end def request_key(request) - request.params[Mongrel::Const::REQUEST_PATH].split('/')[2] + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = request.params[Mongrel::Const::REQUEST_PATH].split('/')[2] end def body(request) request.body end def params(request) Mongrel::HttpRequest.query_parse(request.params["QUERY_STRING"]) end def encode_result(request, response, result, status = 200) response.start(status) do |head, body| body.write(result) end end end diff --git a/lib/puppet/network/http/webrick/rest.rb b/lib/puppet/network/http/webrick/rest.rb index dd0c84d61..8cda079e2 100644 --- a/lib/puppet/network/http/webrick/rest.rb +++ b/lib/puppet/network/http/webrick/rest.rb @@ -1,41 +1,43 @@ require 'puppet/network/http/handler' class Puppet::Network::HTTP::WEBrickREST < Puppet::Network::HTTP::Handler # WEBrick uses a service() method to respond to requests. Simply delegate to the handler response() method. def service(request, response) process(request, response) end private def register_handler @server.mount('/' + @handler.to_s, self) @server.mount('/' + @handler.to_s + 's', self) end def http_method(request) request.request_method end def path(request) - '/' + request.path.split('/')[1] + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = '/' + request.path.split('/')[1] end def request_key(request) - request.path.split('/')[2] + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = request.path.split('/')[2] end def body(request) request.body end def params(request) request.query end def encode_result(request, response, result, status = 200) response.status = status response.body = result end -end \ No newline at end of file +end diff --git a/lib/puppet/node/environment.rb b/lib/puppet/node/environment.rb index 2a314803f..343720a62 100644 --- a/lib/puppet/node/environment.rb +++ b/lib/puppet/node/environment.rb @@ -1,46 +1,47 @@ # Model the environment that a node can operate in. This class just # provides a simple wrapper for the functionality around environments. class Puppet::Node::Environment # Return the list of valid environments. Just looks them up in # the settings. def self.valid - Puppet.settings.value(:environments).split(",").collect { |e| e.to_sym } + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = Puppet.settings.value(:environments).split(",").collect { |e| e.to_sym } end # Is the provided environment valid? def self.valid?(name) return false if name.to_s == "" valid.include?(name.to_sym) end @seen = {} # Return an existing environment instance, or create a new one, # validating the environment name. def self.new(name = nil) name ||= Puppet.settings.value(:environment) raise ArgumentError, "Environment name must be specified" unless name raise(ArgumentError, "'%s' is not a valid environment" % name) unless valid?(name) symbol = name.to_sym return @seen[symbol] if @seen[symbol] obj = self.allocate obj.send :initialize, symbol @seen[symbol] = obj end attr_reader :name # Return an environment-specific setting. def [](param) Puppet.settings.value(param, self.name) end def initialize(name) @name = name end end diff --git a/lib/puppet/provider/mailalias/aliases.rb b/lib/puppet/provider/mailalias/aliases.rb index 85dec29cc..8b5c45617 100755 --- a/lib/puppet/provider/mailalias/aliases.rb +++ b/lib/puppet/provider/mailalias/aliases.rb @@ -1,30 +1,33 @@ require 'puppet/provider/parsedfile' Puppet::Type.type(:mailalias).provide(:aliases, :parent => Puppet::Provider::ParsedFile, :default_target => "/etc/aliases", :filetype => :flat ) do text_line :comment, :match => /^#/ text_line :blank, :match => /^\s*$/ record_line :aliases, :fields => %w{name recipient}, :separator => /\s*:\s*/, :block_eval => :instance do def post_parse(record) - record[:recipient] = record[:recipient].split(/\s*,\s*/).collect { |d| d.gsub(/^['"]|['"]$/, '') } + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + # It's not sufficient to assign to an existing hash. + recipient = record[:recipient].split(/\s*,\s*/).collect { |d| d.gsub(/^['"]|['"]$/, '') } + record[:recipient] = recipient record end def to_line(record) dest = record[:recipient].collect do |d| # Quote aliases that have non-alpha chars if d =~ /[^-\w@.]/ '"%s"' % d else d end end.join(",") return "%s: %s" % [record[:name], dest] end end end diff --git a/lib/puppet/provider/zone/solaris.rb b/lib/puppet/provider/zone/solaris.rb index dd8698b92..be2dd97f9 100644 --- a/lib/puppet/provider/zone/solaris.rb +++ b/lib/puppet/provider/zone/solaris.rb @@ -1,242 +1,243 @@ Puppet::Type.type(:zone).provide(:solaris) do desc "Provider for Solaris Zones." commands :adm => "/usr/sbin/zoneadm", :cfg => "/usr/sbin/zonecfg" defaultfor :operatingsystem => :solaris mk_resource_methods # Convert the output of a list into a hash def self.line2hash(line) fields = [:id, :name, :ensure, :path] properties = {} line.split(":").each_with_index { |value, index| properties[fields[index]] = value } # Configured but not installed zones do not have IDs if properties[:id] == "-" properties.delete(:id) end properties[:ensure] = symbolize(properties[:ensure]) return properties end def self.instances - adm(:list, "-cp").split("\n").collect do |line| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = adm(:list, "-cp").split("\n").collect do |line| new(line2hash(line)) end end # Perform all of our configuration steps. def configure # If the thing is entirely absent, then we need to create the config. str = %{create -b set zonepath=%s } % @resource[:path] # Then perform all of our configuration steps. It's annoying # that we need this much internal info on the resource. @resource.send(:properties).each do |property| if property.is_a? ZoneConfigProperty and ! property.insync?(properties[property.name]) str += property.configtext + "\n" end end str += "commit\n" setconfig(str) end def destroy zonecfg :delete, "-F" end def exists? properties[:ensure] != :absent end # Clear out the cached values. def flush @property_hash.clear end def install zoneadm :install end # Look up the current status. def properties if @property_hash.empty? @property_hash = status || {} if @property_hash.empty? @property_hash[:ensure] = :absent else @resource.class.validproperties.each do |name| @property_hash[name] ||= :absent end end end @property_hash.dup end # We need a way to test whether a zone is in process. Our 'ensure' # property models the static states, but we need to handle the temporary ones. def processing? if hash = status() case hash[:ensure] when "incomplete", "ready", "shutting_down" true else false end else false end end # Collect the configuration of the zone. def getconfig output = zonecfg :info name = nil current = nil hash = {} output.split("\n").each do |line| case line when /^(\S+):\s*$/: name = $1 current = nil # reset it when /^(\S+):\s*(.+)$/: hash[$1.intern] = $2 when /^\s+(\S+):\s*(.+)$/: if name unless hash.include? name hash[name] = [] end unless current current = {} hash[name] << current end current[$1.intern] = $2 else err "Ignoring '%s'" % line end else debug "Ignoring zone output '%s'" % line end end return hash end # Execute a configuration string. Can't be private because it's called # by the properties. def setconfig(str) command = "#{command(:cfg)} -z %s -f -" % @resource[:name] debug "Executing '%s' in zone %s with '%s'" % [command, @resource[:name], str] IO.popen(command, "w") do |pipe| pipe.puts str end unless $? == 0 raise ArgumentError, "Failed to apply configuration" end end def start # Check the sysidcfg stuff if cfg = @resource[:sysidcfg] path = File.join(@resource[:path], "root", "etc", "sysidcfg") unless File.exists?(path) begin File.open(path, "w", 0600) do |f| f.puts cfg end rescue => detail if Puppet[:debug] puts detail.stacktrace end raise Puppet::Error, "Could not create sysidcfg: %s" % detail end end end zoneadm :boot end # Return a hash of the current status of this zone. def status begin output = adm "-z", @resource[:name], :list, "-p" rescue Puppet::ExecutionFailure return nil end main = self.class.line2hash(output.chomp) # Now add in the configuration information config_status.each do |name, value| main[name] = value end main end def stop zoneadm :halt end def unconfigure zonecfg :delete, "-F" end def uninstall zoneadm :uninstall, "-F" end private # Turn the results of getconfig into status information. def config_status config = getconfig() result = {} result[:autoboot] = config[:autoboot] ? config[:autoboot].intern : :absent result[:pool] = config[:pool] result[:shares] = config[:shares] if dir = config["inherit-pkg-dir"] result[:inherit] = dir.collect { |dirs| dirs[:dir] } end if net = config["net"] result[:ip] = net.collect { |params| "%s:%s" % [params[:physical], params[:address]] } end result end def zoneadm(*cmd) begin adm("-z", @resource[:name], *cmd) rescue Puppet::ExecutionFailure => detail self.fail "Could not %s zone: %s" % [cmd[0], detail] end end def zonecfg(*cmd) # You apparently can't get the configuration of the global zone return "" if self.name == "global" begin cfg("-z", self.name, *cmd) rescue Puppet::ExecutionFailure => detail self.fail "Could not %s zone: %s" % [cmd[0], detail] end end end diff --git a/lib/puppet/resource_reference.rb b/lib/puppet/resource_reference.rb index 771a91be7..44b518816 100644 --- a/lib/puppet/resource_reference.rb +++ b/lib/puppet/resource_reference.rb @@ -1,79 +1,80 @@ # # Created by Luke Kanies on 2007-11-28. # Copyright (c) 2007. All rights reserved. require 'puppet' # A simple class to canonize how we refer to and retrieve # resources. class Puppet::ResourceReference attr_reader :type attr_accessor :title, :catalog def initialize(type, title) # This will set @type if it looks like a resource reference. self.title = title # Don't override whatever was done by setting the title. self.type = type if self.type.nil? @builtin_type = nil end # Find our resource. def resolve if catalog return catalog.resource(to_s) end # If it's builtin, then just ask for it directly from the type. if t = builtin_type t[@title] else # Else, look for a component with the full reference as the name. Puppet::Type::Component[to_s] end end # If the title has square brackets, treat it like a reference and # set things appropriately; else, just set it. def title=(value) if value =~ /^([^\[\]]+)\[(.+)\]$/ self.type = $1 @title = $2 else @title = value end end # Canonize the type so we know it's always consistent. def type=(value) if value.nil? or value.to_s.downcase == "component" @type = "Class" else - @type = value.to_s.split("::").collect { |s| s.capitalize }.join("::") + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = @type = value.to_s.split("::").collect { |s| s.capitalize }.join("::") end end # Convert to the standard way of referring to resources. def to_s "%s[%s]" % [@type, @title] end private def builtin_type? builtin_type ? true : false end def builtin_type if @builtin_type.nil? if @type =~ /::/ @builtin_type = false elsif klass = Puppet::Type.type(@type.to_s.downcase) @builtin_type = klass else @builtin_type = false end end @builtin_type end end diff --git a/lib/puppet/type/host.rb b/lib/puppet/type/host.rb index 868e41629..53365bf40 100755 --- a/lib/puppet/type/host.rb +++ b/lib/puppet/type/host.rb @@ -1,103 +1,104 @@ module Puppet newtype(:host) do ensurable newproperty(:ip) do desc "The host's IP address, IPv4 or IPv6." validate do |value| unless value =~ /((([0-9a-fA-F]+:){7}[0-9a-fA-F]+)|(([0-9a-fA-F]+:)*[0-9a-fA-F]+)?::(([0-9a-fA-F]+:)*[0-9a-fA-F]+)?)|((25[0-5]|2[0-4][\d]|[0-1]?\d?\d)(\.(25[0-5]|2[0-4]\d|[0-1]?\d?\d)){3})/ raise Puppet::Error, "Invalid IP address" end end end newproperty(:alias) do desc "Any alias the host might have. Multiple values must be specified as an array. Note that this state has the same name as one of the metaparams; using this state to set aliases will make those aliases available in your Puppet scripts and also on disk." def insync?(is) is == @should end def is_to_s(currentvalue = @is) currentvalue = [currentvalue] unless currentvalue.is_a? Array currentvalue.join(" ") end def retrieve is = super case is when String is = is.split(/\s*,\s*/) when Symbol: is = [is] when Array # nothing else raise Puppet::DevError, "Invalid @is type %s" % is.class end return is end # We actually want to return the whole array here, not just the first # value. def should if defined? @should if @should == [:absent] return :absent else return @should end else return nil end end def should_to_s(newvalue = @should) newvalue.join(" ") end validate do |value| if value =~ /\s/ raise Puppet::Error, "Aliases cannot include whitespace" end end end newproperty(:target) do desc "The file in which to store service information. Only used by those providers that write to disk (i.e., not NetInfo)." defaultto { if @resource.class.defaultprovider.ancestors.include?(Puppet::Provider::ParsedFile) @resource.class.defaultprovider.default_target else nil end } end newparam(:name) do desc "The host name." isnamevar validate do |value| - value.split('.').each do |hostpart| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = value.split('.').each do |hostpart| unless hostpart =~ /^([\d\w]+|[\d\w][\d\w\-]+[\d\w])$/ raise Puppet::Error, "Invalid host name" end end end end @doc = "Installs and manages host entries. For most systems, these entries will just be in ``/etc/hosts``, but some systems (notably OS X) will have different solutions." end end diff --git a/lib/puppet/type/service.rb b/lib/puppet/type/service.rb index 1b625cc41..560d81bae 100644 --- a/lib/puppet/type/service.rb +++ b/lib/puppet/type/service.rb @@ -1,192 +1,194 @@ # This is our main way of managing processes right now. # # a service is distinct from a process in that services # can only be managed through the interface of an init script # which is why they have a search path for initscripts and such module Puppet newtype(:service) do @doc = "Manage running services. Service support unfortunately varies widely by platform -- some platforms have very little if any concept of a running service, and some have a very codified and powerful concept. Puppet's service support will generally be able to make up for any inherent shortcomings (e.g., if there is no 'status' command, then Puppet will look in the process table for a command matching the service name), but the more information you can provide the better behaviour you will get. Or, you can just use a platform that has very good service support. Note that if a ``service`` receives an event from another resource, the service will get restarted. The actual command to restart the service depends on the platform. You can provide a special command for restarting with the ``restart`` attribute." feature :refreshable, "The provider can restart the service.", :methods => [:restart] feature :enableable, "The provider can enable and disable the service", :methods => [:disable, :enable, :enabled?] feature :controllable, "The provider uses a control variable." newproperty(:enable, :required_features => :enableable) do desc "Whether a service should be enabled to start at boot. This property behaves quite differently depending on the platform; wherever possible, it relies on local tools to enable or disable a given service." newvalue(:true, :event => :service_enabled) do provider.enable end newvalue(:false, :event => :service_disabled) do provider.disable end def retrieve return provider.enabled? end end # Handle whether the service should actually be running right now. newproperty(:ensure) do desc "Whether a service should be running." newvalue(:stopped, :event => :service_stopped) do provider.stop end newvalue(:running, :event => :service_started) do provider.start end aliasvalue(:false, :stopped) aliasvalue(:true, :running) def retrieve return provider.status end def sync event = super() if property = @resource.property(:enable) val = property.retrieve property.sync unless property.insync?(val) end return event end end newparam(:binary) do desc "The path to the daemon. This is only used for systems that do not support init scripts. This binary will be used to start the service if no ``start`` parameter is provided." end newparam(:hasstatus) do desc "Declare the the service's init script has a functional status command. Based on testing, it was found that a large number of init scripts on different platforms do not support any kind of status command; thus, you must specify manually whether the service you are running has such a command (or you can specify a specific command using the ``status`` parameter). If you do not specify anything, then the service name will be looked for in the process table." newvalues(:true, :false) end newparam(:name) do desc "The name of the service to run. This name is used to find the service in whatever service subsystem it is in." isnamevar end newparam(:path) do desc "The search path for finding init scripts. Multiple values should be separated by colons or provided as an array." munge do |value| value = [value] unless value.is_a?(Array) - paths = value.flatten.collect { |p| p.split(":") }.flatten.find_all do |path| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + # It affects stand-alone blocks, too. + paths = value.flatten.collect { |p| x = p.split(":") }.flatten.find_all do |path| if FileTest.directory?(path) true else if FileTest.exist?(path) and ! FileTest.directory?(path) @resource.debug "Search path %s is not a directory" % [path] else @resource.debug("Search path %s does not exist" % [path]) end false end end paths end defaultto { provider.class.defpath if provider.class.respond_to?(:defpath) } end newparam(:pattern) do desc "The pattern to search for in the process table. This is used for stopping services on platforms that do not support init scripts, and is also used for determining service status on those service whose init scripts do not include a status command. If this is left unspecified and is needed to check the status of a service, then the service name will be used instead. The pattern can be a simple string or any legal Ruby pattern." defaultto { @resource[:binary] || @resource[:name] } end newparam(:restart) do desc "Specify a *restart* command manually. If left unspecified, the service will be stopped and then started." end newparam(:start) do desc "Specify a *start* command manually. Most service subsystems support a ``start`` command, so this will not need to be specified." end newparam(:status) do desc "Specify a *status* command manually. If left unspecified, the status method will be determined automatically, usually by looking for the service in the process table." end newparam(:stop) do desc "Specify a *stop* command manually." end newparam(:control) do desc "The control variable used to manage services (originally for HP-UX). Defaults to the upcased service name plus ``START`` replacing dots with underscores, for those providers that support the ``controllable`` feature." defaultto { resource.name.gsub(".","_").upcase + "_START" if resource.provider.controllable? } end newparam :hasrestart do desc "Specify that an init script has a ``restart`` option. Otherwise, the init script's ``stop`` and ``start`` methods are used." newvalues(:true, :false) end # Basically just a synonym for restarting. Used to respond # to events. def refresh # Only restart if we're actually running if (@parameters[:ensure] || newattr(:ensure)).retrieve == :running provider.restart else debug "Skipping restart; service is not running" end end end end diff --git a/lib/puppet/util.rb b/lib/puppet/util.rb index 2e83a08e9..560afe10b 100644 --- a/lib/puppet/util.rb +++ b/lib/puppet/util.rb @@ -1,463 +1,464 @@ # A module to collect utility functions. require 'sync' require 'puppet/external/lock' module Puppet # A command failed to execute. class ExecutionFailure < Puppet::Error end module Util require 'benchmark' require 'puppet/util/posix' extend Puppet::Util::POSIX # Create a hash to store the different sync objects. @@syncresources = {} # Return the sync object associated with a given resource. def self.sync(resource) @@syncresources[resource] ||= Sync.new return @@syncresources[resource] end # Change the process to a different user def self.chuser if Facter["operatingsystem"].value == "Darwin" $stderr.puts "Ruby on darwin is broken; puppetmaster will not set its UID to 'puppet' and must run as root" return end if group = Puppet[:group] group = self.gid(group) unless group raise Puppet::Error, "No such group %s" % Puppet[:group] end unless Puppet::Util::SUIDManager.gid == group begin Puppet::Util::SUIDManager.egid = group Puppet::Util::SUIDManager.gid = group rescue => detail Puppet.warning "could not change to group %s: %s" % [group.inspect, detail] $stderr.puts "could not change to group %s" % group.inspect # Don't exit on failed group changes, since it's # not fatal #exit(74) end end end if user = Puppet[:user] user = self.uid(user) unless user raise Puppet::Error, "No such user %s" % Puppet[:user] end unless Puppet::Util::SUIDManager.uid == user begin Puppet::Util::SUIDManager.uid = user Puppet::Util::SUIDManager.euid = user rescue $stderr.puts "could not change to user %s" % user exit(74) end end end end # Create a shared lock for reading def self.readlock(file) self.sync(file).synchronize(Sync::SH) do File.open(file) { |f| f.lock_shared { |lf| yield lf } } end end # Create an exclusive lock for writing, and do the writing in a # tmp file. def self.writelock(file, mode = 0600) tmpfile = file + ".tmp" unless FileTest.directory?(File.dirname(tmpfile)) raise Puppet::DevError, "Cannot create %s; directory %s does not exist" % [file, File.dirname(file)] end self.sync(file).synchronize(Sync::EX) do File.open(file, "w", mode) do |rf| rf.lock_exclusive do |lrf| File.open(tmpfile, "w", mode) do |tf| yield tf end begin File.rename(tmpfile, file) rescue => detail Puppet.err "Could not rename %s to %s: %s" % [file, tmpfile, detail] end end end end end # Create instance methods for each of the log levels. This allows # the messages to be a little richer. Most classes will be calling this # method. def self.logmethods(klass, useself = true) Puppet::Util::Log.eachlevel { |level| klass.send(:define_method, level, proc { |args| if args.is_a?(Array) args = args.join(" ") end if useself Puppet::Util::Log.create( :level => level, :source => self, :message => args ) else Puppet::Util::Log.create( :level => level, :message => args ) end }) } end # Proxy a bunch of methods to another object. def self.classproxy(klass, objmethod, *methods) classobj = class << klass; self; end methods.each do |method| classobj.send(:define_method, method) do |*args| obj = self.send(objmethod) obj.send(method, *args) end end end # Proxy a bunch of methods to another object. def self.proxy(klass, objmethod, *methods) methods.each do |method| klass.send(:define_method, method) do |*args| obj = self.send(objmethod) obj.send(method, *args) end end end # XXX this should all be done using puppet objects, not using # normal mkdir def self.recmkdir(dir,mode = 0755) if FileTest.exist?(dir) return false else tmp = dir.sub(/^\//,'') path = [File::SEPARATOR] tmp.split(File::SEPARATOR).each { |dir| path.push dir if ! FileTest.exist?(File.join(path)) Dir.mkdir(File.join(path), mode) elsif FileTest.directory?(File.join(path)) next else FileTest.exist?(File.join(path)) raise "Cannot create %s: basedir %s is a file" % [dir, File.join(path)] end } return true end end # Execute a given chunk of code with a new umask. def self.withumask(mask) cur = File.umask(mask) begin yield ensure File.umask(cur) end end def benchmark(*args) msg = args.pop level = args.pop object = nil if args.empty? if respond_to?(level) object = self else object = Puppet end else object = args.pop end unless level raise Puppet::DevError, "Failed to provide level to :benchmark" end unless level == :none or object.respond_to? level raise Puppet::DevError, "Benchmarked object does not respond to %s" % level end # Only benchmark if our log level is high enough if level != :none and Puppet::Util::Log.sendlevel?(level) result = nil seconds = Benchmark.realtime { yield } object.send(level, msg + (" in %0.2f seconds" % seconds)) return seconds else yield end end def binary(bin) if bin =~ /^\// if FileTest.exists? bin return bin else return nil end else - ENV['PATH'].split(":").each do |dir| + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = ENV['PATH'].split(":").each do |dir| if FileTest.exists? File.join(dir, bin) return File.join(dir, bin) end end return nil end end module_function :binary # Execute the provided command in a pipe, yielding the pipe object. def execpipe(command, failonfail = true) if respond_to? :debug debug "Executing '%s'" % command else Puppet.debug "Executing '%s'" % command end output = open("| #{command} 2>&1") do |pipe| yield pipe end if failonfail unless $? == 0 raise ExecutionFailure, output end end return output end def execfail(command, exception) begin output = execute(command) return output rescue ExecutionFailure raise exception, output end end # Execute the desired command, and return the status and output. # def execute(command, failonfail = true, uid = nil, gid = nil) def execute(command, arguments = {:failonfail => true}) if command.is_a?(Array) command = command.flatten.collect { |i| i.to_s } str = command.join(" ") else # We require an array here so we know where we're incorrectly # using a string instead of an array. Once everything is # switched to an array, we might relax this requirement. raise ArgumentError, "Must pass an array to execute()" end if respond_to? :debug debug "Executing '%s'" % str else Puppet.debug "Executing '%s'" % str end if arguments[:uid] arguments[:uid] = Puppet::Util::SUIDManager.convert_xid(:uid, arguments[:uid]) end if arguments[:gid] arguments[:gid] = Puppet::Util::SUIDManager.convert_xid(:gid, arguments[:gid]) end @@os ||= Facter.value(:operatingsystem) output = nil child_pid, child_status = nil # There are problems with read blocking with badly behaved children # read.partialread doesn't seem to capture either stdout or stderr # We hack around this using a temporary file # The idea here is to avoid IO#read whenever possible. output_file="/dev/null" if ! arguments[:squelch] require "tempfile" output_file = Tempfile.new("puppet") end oldverb = $VERBOSE $VERBOSE = false child_pid = Kernel.fork $VERBOSE = oldverb if child_pid # Parent process executes this child_status = Process.waitpid2(child_pid)[1] else # Child process executes this Process.setsid begin $stdin.reopen("/dev/null") $stdout.reopen(output_file) $stderr.reopen(output_file) if arguments[:gid] Process.egid = arguments[:gid] Process.gid = arguments[:gid] unless @@os == "Darwin" end if arguments[:uid] Process.euid = arguments[:uid] Process.uid = arguments[:uid] unless @@os == "Darwin" end ENV['LANG'] = ENV['LC_ALL'] = ENV['LC_MESSAGES'] = ENV['LANGUAGE'] = 'C' if command.is_a?(Array) Kernel.exec(*command) else Kernel.exec(command) end rescue => detail puts detail.to_s exit!(1) end # begin; rescue end # if child_pid # read output in if required if ! arguments[:squelch] # Make sure the file's actually there. This is # basically a race condition, and is probably a horrible # way to handle it, but, well, oh well. unless FileTest.exists?(output_file.path) Puppet.warning "sleeping" sleep 0.5 unless FileTest.exists?(output_file.path) Puppet.warning "sleeping 2" sleep 1 unless FileTest.exists?(output_file.path) Puppet.warning "Could not get output" output = "" end end end unless output # We have to explicitly open here, so that it reopens # after the child writes. output = output_file.open.read # The 'true' causes the file to get unlinked right away. output_file.close(true) end end if arguments[:failonfail] unless child_status == 0 raise ExecutionFailure, "Execution of '%s' returned %s: %s" % [str, child_status, output] end end return output end module_function :execute # Create an exclusive lock. def threadlock(resource, type = Sync::EX) Puppet::Util.sync(resource).synchronize(type) do yield end end # Because some modules provide their own version of this method. alias util_execute execute module_function :benchmark def memory unless defined? @pmap pmap = %x{which pmap 2>/dev/null}.chomp if $? != 0 or pmap =~ /^no/ @pmap = nil else @pmap = pmap end end if @pmap return %x{pmap #{Process.pid}| grep total}.chomp.sub(/^\s*total\s+/, '').sub(/K$/, '').to_i else 0 end end def symbolize(value) if value.respond_to? :intern value.intern else value end end def symbolizehash(hash) newhash = {} hash.each do |name, val| if name.is_a? String newhash[name.intern] = val else newhash[name] = val end end end def symbolizehash!(hash) hash.each do |name, val| if name.is_a? String hash[name.intern] = val hash.delete(name) end end return hash end module_function :symbolize, :symbolizehash, :symbolizehash! # Just benchmark, with no logging. def thinmark seconds = Benchmark.realtime { yield } return seconds end module_function :memory, :thinmark end end require 'puppet/util/errors' require 'puppet/util/methodhelper' require 'puppet/util/metaid' require 'puppet/util/classgen' require 'puppet/util/docs' require 'puppet/util/execution' require 'puppet/util/logging' require 'puppet/util/package' require 'puppet/util/warnings' diff --git a/lib/puppet/util/constant_inflector.rb b/lib/puppet/util/constant_inflector.rb index 8b083951f..eeeaa0632 100644 --- a/lib/puppet/util/constant_inflector.rb +++ b/lib/puppet/util/constant_inflector.rb @@ -1,14 +1,15 @@ # Created on 2008-02-12 # Copyright Luke Kanies # A common module for converting between constants and # file names. module Puppet::Util::ConstantInflector def file2constant(file) - file.split("/").collect { |name| name.capitalize }.join("::").gsub(/_+(.)/) { |term| $1.capitalize } + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = file.split("/").collect { |name| name.capitalize }.join("::").gsub(/_+(.)/) { |term| $1.capitalize } end def constant2file(constant) constant.to_s.gsub(/([a-z])([A-Z])/) { |term| $1 + "_" + $2 }.gsub("::", "/").downcase end end diff --git a/lib/puppet/util/fileparsing.rb b/lib/puppet/util/fileparsing.rb index 8e39719c8..23d02ea60 100644 --- a/lib/puppet/util/fileparsing.rb +++ b/lib/puppet/util/fileparsing.rb @@ -1,398 +1,399 @@ # A mini-language for parsing files. This is only used file the ParsedFile # provider, but it makes more sense to split it out so it's easy to maintain # in one place. # # You can use this module to create simple parser/generator classes. For instance, # the following parser should go most of the way to parsing /etc/passwd: # # class Parser # include Puppet::Util::FileParsing # record_line :user, :fields => %w{name password uid gid gecos home shell}, # :separator => ":" # end # # You would use it like this: # # parser = Parser.new # lines = parser.parse(File.read("/etc/passwd")) # # lines.each do |type, hash| # type will always be :user, since we only have one # p hash # end # # Each line in this case would be a hash, with each field set appropriately. # You could then call 'parser.to_line(hash)' on any of those hashes to generate # the text line again. require 'puppet/util/methodhelper' module Puppet::Util::FileParsing include Puppet::Util attr_writer :line_separator, :trailing_separator class FileRecord include Puppet::Util include Puppet::Util::MethodHelper attr_accessor :absent, :joiner, :rts, :separator, :rollup, :name, :match, :block_eval attr_reader :fields, :optional, :type INVALID_FIELDS = [:record_type, :target, :on_disk] # Customize this so we can do a bit of validation. def fields=(fields) @fields = fields.collect do |field| r = symbolize(field) if INVALID_FIELDS.include?(r) raise ArgumentError.new("Cannot have fields named %s" % r) end r end end def initialize(type, options = {}, &block) @type = symbolize(type) unless [:record, :text].include?(@type) raise ArgumentError, "Invalid record type %s" % @type end set_options(options) if self.type == :record # Now set defaults. self.absent ||= "" self.separator ||= /\s+/ self.joiner ||= " " self.optional ||= [] unless defined? @rollup @rollup = true end end if block_given? @block_eval ||= :process # Allow the developer to specify that a block should be instance-eval'ed. if @block_eval == :instance instance_eval(&block) else meta_def(@block_eval, &block) end end end # Convert a record into a line by joining the fields together appropriately. # This is pulled into a separate method so it can be called by the hooks. def join(details) joinchar = self.joiner fields.collect { |field| # If the field is marked absent, use the appropriate replacement if details[field] == :absent or details[field] == [:absent] or details[field].nil? if self.optional.include?(field) self.absent else raise ArgumentError, "Field '%s' is required" % field end else details[field].to_s end }.reject { |c| c.nil?}.join(joinchar) end # Customize this so we can do a bit of validation. def optional=(optional) @optional = optional.collect do |field| symbolize(field) end end # Create a hook that modifies the hash resulting from parsing. def post_parse=(block) meta_def(:post_parse, &block) end # Create a hook that modifies the hash just prior to generation. def pre_gen=(block) meta_def(:pre_gen, &block) end # Are we a text type? def text? type == :text end def to_line=(block) meta_def(:to_line, &block) end end # Clear all existing record definitions. Only used for testing. def clear_records @record_types.clear @record_order.clear end def fields(type) if record = record_type(type) record.fields.dup else nil end end # Try to match a specific text line. def handle_text_line(line, record) if line =~ record.match return {:record_type => record.name, :line => line} else return nil end end # Try to match a record. def handle_record_line(line, record) ret = nil if record.respond_to?(:process) if ret = record.send(:process, line.dup) unless ret.is_a?(Hash) raise Puppet::DevError, "Process record type %s returned non-hash" % record.name end else return nil end elsif regex = record.match # In this case, we try to match the whole line and then use the # match captures to get our fields. if match = regex.match(line) fields = [] ret = {} record.fields.zip(match.captures).each do |field, value| if value == record.absent ret[field] = :absent else ret[field] = value end end else nil end else ret = {} sep = record.separator # String "helpfully" replaces ' ' with /\s+/ in splitting, so we # have to work around it. if sep == " " sep = / / end line_fields = line.split(sep) record.fields.each do |param| value = line_fields.shift if value and value != record.absent ret[param] = value else ret[param] = :absent end end if record.rollup and ! line_fields.empty? last_field = record.fields[-1] val = ([ret[last_field]] + line_fields).join(record.joiner) ret[last_field] = val end end if ret ret[:record_type] = record.name return ret else return nil end end def line_separator unless defined?(@line_separator) @line_separator = "\n" end @line_separator end # Split text into separate lines using the record separator. def lines(text) # Remove any trailing separators, and then split based on them - text.sub(/#{self.line_separator}\Q/,'').split(self.line_separator) + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + x = text.sub(/#{self.line_separator}\Q/,'').split(self.line_separator) end # Split a bunch of text into lines and then parse them individually. def parse(text) count = 1 lines(text).collect do |line| count += 1 if val = parse_line(line) val else error = Puppet::Error.new("Could not parse line %s" % line.inspect) error.line = count raise error end end end # Handle parsing a single line. def parse_line(line) unless records? raise Puppet::DevError, "No record types defined; cannot parse lines" end @record_order.each do |record| # These are basically either text or record lines. method = "handle_%s_line" % record.type if respond_to?(method) if result = send(method, line, record) if record.respond_to?(:post_parse) record.send(:post_parse, result) end return result end else raise Puppet::DevError, "Somehow got invalid line type %s" % record.type end end return nil end # Define a new type of record. These lines get split into hashes. Valid # options are: # * :absent: What to use as value within a line, when a field is # absent. Note that in the record object, the literal :absent symbol is # used, and not this value. Defaults to "". # * :fields: The list of fields, as an array. By default, all # fields are considered required. # * :joiner: How to join fields together. Defaults to '\t'. # * :optional: Which fields are optional. If these are missing, # you'll just get the 'absent' value instead of an ArgumentError. # * :rts: Whether to remove trailing whitespace. Defaults to false. # If true, whitespace will be removed; if a regex, then whatever matches # the regex will be removed. # * :separator: The record separator. Defaults to /\s+/. def record_line(name, options, &block) unless options.include?(:fields) raise ArgumentError, "Must include a list of fields" end record = FileRecord.new(:record, options, &block) record.name = symbolize(name) new_line_type(record) end # Are there any record types defined? def records? defined?(@record_types) and ! @record_types.empty? end # Define a new type of text record. def text_line(name, options, &block) unless options.include?(:match) raise ArgumentError, "You must provide a :match regex for text lines" end record = FileRecord.new(:text, options, &block) record.name = symbolize(name) new_line_type(record) end # Generate a file from a bunch of hash records. def to_file(records) text = records.collect { |record| to_line(record) }.join(line_separator) if trailing_separator text += line_separator end return text end # Convert our parsed record into a text record. def to_line(details) unless record = record_type(details[:record_type]) raise ArgumentError, "Invalid record type %s" % details[:record_type].inspect end if record.respond_to?(:pre_gen) details = details.dup record.send(:pre_gen, details) end case record.type when :text: return details[:line] else if record.respond_to?(:to_line) return record.to_line(details) end line = record.join(details) if regex = record.rts # If they say true, then use whitespace; else, use their regex. if regex == true regex = /\s+$/ end return line.sub(regex,'') else return line end end end # Whether to add a trailing separator to the file. Defaults to true def trailing_separator if defined? @trailing_separator return @trailing_separator else return true end end def valid_attr?(type, attr) type = symbolize(type) if record = record_type(type) and record.fields.include?(symbolize(attr)) return true else if symbolize(attr) == :ensure return true else false end end end private # Define a new type of record. def new_line_type(record) @record_types ||= {} @record_order ||= [] if @record_types.include?(record.name) raise ArgumentError, "Line type %s is already defined" % record.name end @record_types[record.name] = record @record_order << record return record end # Retrieve the record object. def record_type(type) @record_types[symbolize(type)] end end diff --git a/lib/puppet/util/tagging.rb b/lib/puppet/util/tagging.rb index 8a50f3458..e06e13a2c 100644 --- a/lib/puppet/util/tagging.rb +++ b/lib/puppet/util/tagging.rb @@ -1,39 +1,40 @@ # Created on 2008-01-19 # Copyright Luke Kanies # A common module to handle tagging. module Puppet::Util::Tagging # Add a tag to our current list. These tags will be added to all # of the objects contained in this scope. def tag(*ary) @tags ||= [] qualified = [] ary.collect { |tag| tag.to_s.downcase }.each do |tag| fail(Puppet::ParseError, "Invalid tag %s" % tag.inspect) unless valid_tag?(tag) qualified << tag if tag.include?("::") @tags << tag unless @tags.include?(tag) end - qualified.collect { |name| name.split("::") }.flatten.each { |tag| @tags << tag unless @tags.include?(tag) } + # LAK:NOTE See http://snurl.com/21zf8 [groups_google_com] + qualified.collect { |name| x = name.split("::") }.flatten.each { |tag| @tags << tag unless @tags.include?(tag) } end # Are we tagged with the provided tag? def tagged?(tag) defined?(@tags) and @tags.include?(tag.to_s) end # Return a copy of the tag list, so someone can't ask for our tags # and then modify them. def tags @tags ||= [] @tags.dup end private def valid_tag?(tag) tag =~ /^\w[-\w:.]*$/ end end diff --git a/test/ral/providers/mailalias/aliases.rb b/test/ral/providers/mailalias/aliases.rb index 663d4359c..ec8b84684 100755 --- a/test/ral/providers/mailalias/aliases.rb +++ b/test/ral/providers/mailalias/aliases.rb @@ -1,56 +1,55 @@ #!/usr/bin/env ruby require File.dirname(__FILE__) + '/../../../lib/puppettest' require 'puppettest' require 'puppettest/fileparsing' -require 'puppet/type/mailalias' class TestMailaliasAliasesProvider < Test::Unit::TestCase include PuppetTest include PuppetTest::FileParsing def setup super @provider = Puppet.type(:mailalias).provider(:aliases) @oldfiletype = @provider.filetype @alias = mkalias end def teardown Puppet::Util::FileType.filetype(:ram).clear @provider.filetype = @oldfiletype @provider.clear super end def mkalias(name = "me") if defined? @pcount @pcount += 1 else @pcount = 1 end args = { :name => name, :recipient => %w{here there} } fakeresource = fakeresource(:mailalias, args[:name]) key = @provider.new(fakeresource) args.each do |p,v| key.send(p.to_s + "=", v) end return key end def test_data_parsing_and_generating fakedata("data/types/mailalias").each { |file| fakedataparse(file) } end end