diff --git a/README.queueing b/README.queueing index 8c4a18029..83a8e19c0 100644 --- a/README.queueing +++ b/README.queueing @@ -1,126 +1,126 @@ *PUPPET QUEUEING Puppet Queueing is a feature which is designed to take some load off of the PuppetMaster by transferring the task of updating the database to a separate program which is named puppetqd (Puppet Queue Daemon). Currently this is only supported for "Storeconfigs" which is documented at: -http://reductivelabs.com/trac/puppet/wiki/UsingStoredConfiguration +http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration In the future this feature can be extended to any new puppet data which involves storage in a database. *OPERATION In a nutshell: puppetmasterd -> stomp -> service -> stomp -> puppetqd -> database At the moment the only messaging protocol supported is "stomp". Although others could be implemented, stomp is considered by many as the default queueing mechanism for Ruby and Rails applications. It is distributed as a Ruby gem and is easily installed. (The queueing code inside Puppet has been written so that when other interfaces and protocols are implemented they will be easy to use by changing settings in puppet.conf). The "service" in the diagram above is any queueing service that supports the Stomp API. For details refer to: http://xircles.codehaus.org/projects/stomp Both puppetmasterd and puppetqd subscribe to the same queueing service using the stomp interface. As puppetmasterd posts data to the queue, puppetqd receives it and stores it. The details of how to connect to the service and the name of the queue to use are set in puppet.conf: [main] queue_type = stomp queue_source = stomp://localhost:61613 [puppetmasterd] async_storeconfigs = true Note: since puppetmasterd needs to recover the data being stored at a later time, both puppetmasterd and puppetqd need to work with the same database as defined in the STORECONFIGS setup. *QUEUEING SERVICES As mentioned previously any queueing service that supports the Stomp protocol can be used. Which one you use depends on your needs. We have tested with two of the most popular services - StompServer and ActiveMQ. + StompServer http://rubyforge.org/projects/stompserver/ StompServer is a lightweight queueing service written in Ruby which is suitable for testing or low volume puppet usage. Works well when both puppetmasterd and puppetd are running on the same machine that it's running on but we encountered some problems when using it from multiple machines. Just install the stompserver gem and run 'stompserver'. + Apache ActiveMQ http://activemq.apache.org Considered by many to be the most popular message service in use today, ActiveMQ has hundreds of features for scaling, persistence and so on. Although installation is fairly simple, the configuration can seem quite intimidating, but for our use a one line change to the standard configuration is all that is required and is explained at: http://activemq.apache.org/stomp.html Other customization of the internal workings of ActiveMQ, if any, will depend on your needs and deployment. A quick skimming of the ActiveMQ documentation will give you enough info to decide. Others We have looked at but not tried some other queuing services which are compatible with the Stomp API: + POE Component Message Queue + JBoss Messaging (with 3rd party support for Stomp) *SCALING For StoreConfigs you basically need to have the catalog for a node stored in the database before the next time the node connects and asks for a new catalog. If the puppetd on your nodes is set to check every 30 minutes, then it would seem that there is no problem. However if you have 3000 nodes you have a LOT of catalogs to store and it is possible you will not get a catalog saved in time. Running puppetmaster, your queueing service and puppetqd on the same machine means that they are all competing for the same CPU cycles. Bumping up the power of the server they are running on may be enough to handle even fairly large deployments. However since most queueing services (even StompServer) are designed to deliver messages from a "queue" to whoever asks for the next message you can split things up between machines: puppetmaster1 --\ /-- puppetqd1 -\ puppetmaster2 ----> ActiveMQ ---> puppetqd2 ---> database puppetmaster3 --/ \-- puppetqd33 -/ \- puppetqd4-/ This is, of course a totally contrived example, but it gets the point across. As long as the data gets to the database, it doesn't matter which machines or services it goes through. Although for StoreConfigs absolute reliability is not a requirement as a new catalog will be sent the next time a node connects, some amount of persistence should some process crash may be desirable. Both ActiveMQ and MySQL (and other databases) have these kind of features built in which can be activated as needed. diff --git a/examples/modules/sample-module/README.txt b/examples/modules/sample-module/README.txt index cd35c83a1..233e54b8d 100644 --- a/examples/modules/sample-module/README.txt +++ b/examples/modules/sample-module/README.txt @@ -1,17 +1,17 @@ Jeff McCune 2007-08-14 This small, sample module demonstrates how to extend the puppet language with a new parser function. See: manifests/init.pp lib/puppet/parser/functions/hostname_to_dn.rb templates/sample.erb Note the consistent naming of files for Puppet::Util::Autoload Reference Documents: -http://reductivelabs.com/trac/puppet/wiki/ModuleOrganisation +http://docs.puppetlabs.com/guides/modules.html http://docs.puppetlabs.com/guides/custom_functions.html -http://reductivelabs.com/trac/puppet/wiki/FunctionReference +http://docs.puppetlabs.com/references/latest/function.html diff --git a/ext/puppetstoredconfigclean.rb b/ext/puppetstoredconfigclean.rb index 16f39efa1..dcbefa816 100644 --- a/ext/puppetstoredconfigclean.rb +++ b/ext/puppetstoredconfigclean.rb @@ -1,91 +1,91 @@ #!/usr/bin/env ruby # Script to clean up stored configs for (a) given host(s) # # Credits: -# Script was taken from http://reductivelabs.com/trac/puppet/attachment/wiki/UsingStoredConfiguration/kill_node_in_storedconfigs_db.rb +# Script was taken from http://reductivelabs.com/trac/puppet/attachment/wiki/UsingStoredConfiguration/kill_node_in_storedconfigs_db.rb (link no longer valid), # which haven been initially posted by James Turnbull # duritong adapted and improved the script a bit. require 'getoptlong' config = '/etc/puppet/puppet.conf' def printusage(error_code) puts "Usage: #{$0} [ list of hostnames as stored in hosts table ]" puts "\n Options:" puts "--config " exit(error_code) end opts = GetoptLong.new( [ "--config", "-c", GetoptLong::REQUIRED_ARGUMENT ], [ "--help", "-h", GetoptLong::NO_ARGUMENT ], [ "--usage", "-u", GetoptLong::NO_ARGUMENT ], [ "--version", "-v", GetoptLong::NO_ARGUMENT ] ) begin opts.each do |opt, arg| case opt when "--config" config = arg when "--help" printusage(0) when "--usage" printusage(0) when "--version" puts "#{Puppet.version}" exit end end rescue GetoptLong::InvalidOption => detail $stderr.puts "Try '#{$0} --help'" exit(1) end printusage(1) unless ARGV.size > 0 require 'puppet/rails' Puppet[:config] = config Puppet.parse_config pm_conf = Puppet.settings.instance_variable_get(:@values)[:master] adapter = pm_conf[:dbadapter] args = {:adapter => adapter, :log_level => pm_conf[:rails_loglevel]} case adapter when "sqlite3" args[:dbfile] = pm_conf[:dblocation] when "mysql", "postgresql" args[:host] = pm_conf[:dbserver] unless pm_conf[:dbserver].to_s.empty? args[:username] = pm_conf[:dbuser] unless pm_conf[:dbuser].to_s.empty? args[:password] = pm_conf[:dbpassword] unless pm_conf[:dbpassword].to_s.empty? args[:database] = pm_conf[:dbname] unless pm_conf[:dbname].to_s.empty? args[:port] = pm_conf[:dbport] unless pm_conf[:dbport].to_s.empty? socket = pm_conf[:dbsocket] args[:socket] = socket unless socket.to_s.empty? else raise ArgumentError, "Invalid db adapter #{adapter}" end args[:database] = "puppet" unless not args[:database].to_s.empty? ActiveRecord::Base.establish_connection(args) ARGV.each { |hostname| if @host = Puppet::Rails::Host.find_by_name(hostname.strip) print "Killing #{hostname}..." $stdout.flush @host.destroy puts "done." else puts "Can't find host #{hostname}." end } exit 0 diff --git a/lib/puppet/type/file/content.rb b/lib/puppet/type/file/content.rb index cf924f371..63c0aaf4d 100755 --- a/lib/puppet/type/file/content.rb +++ b/lib/puppet/type/file/content.rb @@ -1,217 +1,216 @@ require 'net/http' require 'uri' require 'tempfile' require 'puppet/util/checksums' require 'puppet/network/http/api/v1' require 'puppet/network/http/compression' module Puppet Puppet::Type.type(:file).newproperty(:content) do include Puppet::Util::Diff include Puppet::Util::Checksums include Puppet::Network::HTTP::API::V1 include Puppet::Network::HTTP::Compression.module attr_reader :actual_content desc "Specify the contents of a file as a string. Newlines, tabs, and spaces can be specified using the escaped syntax (e.g., \\n for a newline). The primary purpose of this parameter is to provide a kind of limited templating:: define resolve(nameserver1, nameserver2, domain, search) { $str = \"search $search domain $domain nameserver $nameserver1 nameserver $nameserver2 \" file { \"/etc/resolv.conf\": content => $str } } - This attribute is especially useful when used with - `PuppetTemplating templating`:trac:." + This attribute is especially useful when used with templating." # Store a checksum as the value, rather than the actual content. # Simplifies everything. munge do |value| if value == :absent value elsif checksum?(value) # XXX This is potentially dangerous because it means users can't write a file whose # entire contents are a plain checksum value else @actual_content = value resource.parameter(:checksum).sum(value) end end # Checksums need to invert how changes are printed. def change_to_s(currentvalue, newvalue) # Our "new" checksum value is provided by the source. if source = resource.parameter(:source) and tmp = source.checksum newvalue = tmp end if currentvalue == :absent return "defined content as '#{newvalue}'" elsif newvalue == :absent return "undefined content from '#{currentvalue}'" else return "content changed '#{currentvalue}' to '#{newvalue}'" end end def checksum_type if source = resource.parameter(:source) result = source.checksum else checksum = resource.parameter(:checksum) result = resource[:checksum] end if result =~ /^\{(\w+)\}.+/ return $1.to_sym else return result end end def length (actual_content and actual_content.length) || 0 end def content self.should end # Override this method to provide diffs if asked for. # Also, fix #872: when content is used, and replace is true, the file # should be insync when it exists def insync?(is) if resource.should_be_file? return false if is == :absent else return true end return true if ! @resource.replace? result = super if ! result and Puppet[:show_diff] write_temporarily do |path| print diff(@resource[:path], path) end end result end def retrieve return :absent unless stat = @resource.stat ftype = stat.ftype # Don't even try to manage the content on directories or links return nil if ["directory","link"].include?(ftype) begin resource.parameter(:checksum).sum_file(resource[:path]) rescue => detail raise Puppet::Error, "Could not read #{ftype} #{@resource.title}: #{detail}" end end # Make sure we're also managing the checksum property. def should=(value) @resource.newattr(:checksum) unless @resource.parameter(:checksum) super end # Just write our content out to disk. def sync return_event = @resource.stat ? :file_changed : :file_created # We're safe not testing for the 'source' if there's no 'should' # because we wouldn't have gotten this far if there weren't at least # one valid value somewhere. @resource.write(:content) return_event end def write_temporarily tempfile = Tempfile.new("puppet-file") tempfile.open write(tempfile) tempfile.close yield tempfile.path tempfile.delete end def write(file) resource.parameter(:checksum).sum_stream { |sum| each_chunk_from(actual_content || resource.parameter(:source)) { |chunk| sum << chunk file.print chunk } } end def self.standalone? Puppet.settings[:name] == "apply" end def each_chunk_from(source_or_content) if source_or_content.is_a?(String) yield source_or_content elsif source_or_content.nil? && resource.parameter(:ensure) && [:present, :file].include?(resource.parameter(:ensure).value) yield '' elsif source_or_content.nil? yield read_file_from_filebucket elsif self.class.standalone? yield source_or_content.content elsif source_or_content.local? chunk_file_from_disk(source_or_content) { |chunk| yield chunk } else chunk_file_from_source(source_or_content) { |chunk| yield chunk } end end private def chunk_file_from_disk(source_or_content) File.open(source_or_content.full_path, "r") do |src| while chunk = src.read(8192) yield chunk end end end def chunk_file_from_source(source_or_content) request = Puppet::Indirector::Request.new(:file_content, :find, source_or_content.full_path.sub(/^\//,'')) connection = Puppet::Network::HttpPool.http_instance(source_or_content.server, source_or_content.port) connection.request_get(indirection2uri(request), add_accept_encoding({"Accept" => "raw"})) do |response| case response.code when "404"; nil when /^2/; uncompress(response) { |uncompressor| response.read_body { |chunk| yield uncompressor.uncompress(chunk) } } else # Raise the http error if we didn't get a 'success' of some kind. message = "Error #{response.code} on SERVER: #{(response.body||'').empty? ? response.message : uncompress_body(response)}" raise Net::HTTPError.new(message, response) end end end def read_file_from_filebucket raise "Could not get filebucket from file" unless dipper = resource.bucket sum = should.sub(/\{\w+\}/, '') dipper.getfile(sum) rescue => detail fail "Could not retrieve content for #{should} from filebucket: #{detail}" end end end diff --git a/tasks/rake/git_workflow.rake b/tasks/rake/git_workflow.rake index b2f96c603..980d2fbce 100644 --- a/tasks/rake/git_workflow.rake +++ b/tasks/rake/git_workflow.rake @@ -1,121 +1,121 @@ # This set of tasks helps automate the workflow as described on -# http://reductivelabs.com/trac/puppet/wiki/Development/DevelopmentLifecycle +# http://projects.puppetlabs.com/projects/puppet/wiki/Development_Lifecycle def find_start(start) # This is a case statement, as we might want to map certain # git tags to starting points that are not currently in git. case start when nil?: when @next_release: return "master" else return start end end desc "Set up git for working with Puppet" task :git_setup do # This should be changed as new versions get released @next_release = '0.26.x' @remote = {} default_remote = {} default_remote[:url] = 'git://github.com/reductivelabs/puppet' default_remote[:name] = 'origin' @remote[:name] = %x{git config puppet.defaultremote}.chomp @remote[:name] = @remote[:name].empty? ? default_remote[:name] : @remote[:name] @remote[:url] = default_remote[:url] if @remote[:name] == default_remote[:name] default_fetch = '+refs/heads/*:refs/remotes/puppet/*' @remote[:fetch] = %x{git config puppet.#{@remote[:name]}.fetch}.chomp @remote[:fetch] = @remote[:fetch].empty? ? default_fetch : @remote[:fetch] end desc "Start work on a feature" task :start_feature, [:feature,:remote,:branch] => :git_setup do |t, args| args.with_defaults(:remote => @remote[:name]) args.with_defaults(:branch => @next_release) start_at = find_start(args.branch) branch = "feature/#{start_at}/#{args.feature}" sh "git checkout -b #{branch} #{start_at}" do |ok, res| if ! ok raise < :git_setup do |t, args| args.with_defaults(:remote => @remote[:name]) args.with_defaults(:branch => @next_release) start_at = find_start(args.branch) branch = "tickets/#{start_at}/#{args.ticket}" sh "git checkout -b #{branch} #{start_at}" do |ok, res| unless ok raise < 0 raise "Patches already exist matching '00*.patch'; clean up first" end unless %x{git status} =~ /On branch (.+)/ raise "Could not get branch from 'git status'" end branch = $1 unless branch =~ %r{^([^\/]+)/([^\/]+)/([^\/]+)$} raise "Branch name does not follow // model; cannot autodetect parent branch" end type, parent, name = $1, $2, $3 # Create all of the patches sh "git format-patch -C -M -s -n --subject-prefix='PATCH/puppet' #{parent}..HEAD" # And then mail them out. # If we've got more than one patch, add --compose if Dir.glob("00*.patch").length > 1 compose = "--compose" else compose = "" end # Now send the mail. sh "git send-email #{compose} --no-signed-off-by-cc --suppress-from --to puppet-dev@googlegroups.com 00*.patch" # Finally, clean up the patches sh "rm 00*.patch" end