diff --git a/README.queueing b/README.queueing index 8c4a18029..83a8e19c0 100644 --- a/README.queueing +++ b/README.queueing @@ -1,126 +1,126 @@ *PUPPET QUEUEING Puppet Queueing is a feature which is designed to take some load off of the PuppetMaster by transferring the task of updating the database to a separate program which is named puppetqd (Puppet Queue Daemon). Currently this is only supported for "Storeconfigs" which is documented at: -http://reductivelabs.com/trac/puppet/wiki/UsingStoredConfiguration +http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration In the future this feature can be extended to any new puppet data which involves storage in a database. *OPERATION In a nutshell: puppetmasterd -> stomp -> service -> stomp -> puppetqd -> database At the moment the only messaging protocol supported is "stomp". Although others could be implemented, stomp is considered by many as the default queueing mechanism for Ruby and Rails applications. It is distributed as a Ruby gem and is easily installed. (The queueing code inside Puppet has been written so that when other interfaces and protocols are implemented they will be easy to use by changing settings in puppet.conf). The "service" in the diagram above is any queueing service that supports the Stomp API. For details refer to: http://xircles.codehaus.org/projects/stomp Both puppetmasterd and puppetqd subscribe to the same queueing service using the stomp interface. As puppetmasterd posts data to the queue, puppetqd receives it and stores it. The details of how to connect to the service and the name of the queue to use are set in puppet.conf: [main] queue_type = stomp queue_source = stomp://localhost:61613 [puppetmasterd] async_storeconfigs = true Note: since puppetmasterd needs to recover the data being stored at a later time, both puppetmasterd and puppetqd need to work with the same database as defined in the STORECONFIGS setup. *QUEUEING SERVICES As mentioned previously any queueing service that supports the Stomp protocol can be used. Which one you use depends on your needs. We have tested with two of the most popular services - StompServer and ActiveMQ. + StompServer http://rubyforge.org/projects/stompserver/ StompServer is a lightweight queueing service written in Ruby which is suitable for testing or low volume puppet usage. Works well when both puppetmasterd and puppetd are running on the same machine that it's running on but we encountered some problems when using it from multiple machines. Just install the stompserver gem and run 'stompserver'. + Apache ActiveMQ http://activemq.apache.org Considered by many to be the most popular message service in use today, ActiveMQ has hundreds of features for scaling, persistence and so on. Although installation is fairly simple, the configuration can seem quite intimidating, but for our use a one line change to the standard configuration is all that is required and is explained at: http://activemq.apache.org/stomp.html Other customization of the internal workings of ActiveMQ, if any, will depend on your needs and deployment. A quick skimming of the ActiveMQ documentation will give you enough info to decide. Others We have looked at but not tried some other queuing services which are compatible with the Stomp API: + POE Component Message Queue + JBoss Messaging (with 3rd party support for Stomp) *SCALING For StoreConfigs you basically need to have the catalog for a node stored in the database before the next time the node connects and asks for a new catalog. If the puppetd on your nodes is set to check every 30 minutes, then it would seem that there is no problem. However if you have 3000 nodes you have a LOT of catalogs to store and it is possible you will not get a catalog saved in time. Running puppetmaster, your queueing service and puppetqd on the same machine means that they are all competing for the same CPU cycles. Bumping up the power of the server they are running on may be enough to handle even fairly large deployments. However since most queueing services (even StompServer) are designed to deliver messages from a "queue" to whoever asks for the next message you can split things up between machines: puppetmaster1 --\ /-- puppetqd1 -\ puppetmaster2 ----> ActiveMQ ---> puppetqd2 ---> database puppetmaster3 --/ \-- puppetqd33 -/ \- puppetqd4-/ This is, of course a totally contrived example, but it gets the point across. As long as the data gets to the database, it doesn't matter which machines or services it goes through. Although for StoreConfigs absolute reliability is not a requirement as a new catalog will be sent the next time a node connects, some amount of persistence should some process crash may be desirable. Both ActiveMQ and MySQL (and other databases) have these kind of features built in which can be activated as needed. diff --git a/examples/modules/sample-module/README.txt b/examples/modules/sample-module/README.txt index cd35c83a1..233e54b8d 100644 --- a/examples/modules/sample-module/README.txt +++ b/examples/modules/sample-module/README.txt @@ -1,17 +1,17 @@ Jeff McCune 2007-08-14 This small, sample module demonstrates how to extend the puppet language with a new parser function. See: manifests/init.pp lib/puppet/parser/functions/hostname_to_dn.rb templates/sample.erb Note the consistent naming of files for Puppet::Util::Autoload Reference Documents: -http://reductivelabs.com/trac/puppet/wiki/ModuleOrganisation +http://docs.puppetlabs.com/guides/modules.html http://docs.puppetlabs.com/guides/custom_functions.html -http://reductivelabs.com/trac/puppet/wiki/FunctionReference +http://docs.puppetlabs.com/references/latest/function.html diff --git a/ext/puppetstoredconfigclean.rb b/ext/puppetstoredconfigclean.rb index 16f39efa1..dcbefa816 100644 --- a/ext/puppetstoredconfigclean.rb +++ b/ext/puppetstoredconfigclean.rb @@ -1,91 +1,91 @@ #!/usr/bin/env ruby # Script to clean up stored configs for (a) given host(s) # # Credits: -# Script was taken from http://reductivelabs.com/trac/puppet/attachment/wiki/UsingStoredConfiguration/kill_node_in_storedconfigs_db.rb +# Script was taken from http://reductivelabs.com/trac/puppet/attachment/wiki/UsingStoredConfiguration/kill_node_in_storedconfigs_db.rb (link no longer valid), # which haven been initially posted by James Turnbull # duritong adapted and improved the script a bit. require 'getoptlong' config = '/etc/puppet/puppet.conf' def printusage(error_code) puts "Usage: #{$0} [ list of hostnames as stored in hosts table ]" puts "\n Options:" puts "--config " exit(error_code) end opts = GetoptLong.new( [ "--config", "-c", GetoptLong::REQUIRED_ARGUMENT ], [ "--help", "-h", GetoptLong::NO_ARGUMENT ], [ "--usage", "-u", GetoptLong::NO_ARGUMENT ], [ "--version", "-v", GetoptLong::NO_ARGUMENT ] ) begin opts.each do |opt, arg| case opt when "--config" config = arg when "--help" printusage(0) when "--usage" printusage(0) when "--version" puts "#{Puppet.version}" exit end end rescue GetoptLong::InvalidOption => detail $stderr.puts "Try '#{$0} --help'" exit(1) end printusage(1) unless ARGV.size > 0 require 'puppet/rails' Puppet[:config] = config Puppet.parse_config pm_conf = Puppet.settings.instance_variable_get(:@values)[:master] adapter = pm_conf[:dbadapter] args = {:adapter => adapter, :log_level => pm_conf[:rails_loglevel]} case adapter when "sqlite3" args[:dbfile] = pm_conf[:dblocation] when "mysql", "postgresql" args[:host] = pm_conf[:dbserver] unless pm_conf[:dbserver].to_s.empty? args[:username] = pm_conf[:dbuser] unless pm_conf[:dbuser].to_s.empty? args[:password] = pm_conf[:dbpassword] unless pm_conf[:dbpassword].to_s.empty? args[:database] = pm_conf[:dbname] unless pm_conf[:dbname].to_s.empty? args[:port] = pm_conf[:dbport] unless pm_conf[:dbport].to_s.empty? socket = pm_conf[:dbsocket] args[:socket] = socket unless socket.to_s.empty? else raise ArgumentError, "Invalid db adapter #{adapter}" end args[:database] = "puppet" unless not args[:database].to_s.empty? ActiveRecord::Base.establish_connection(args) ARGV.each { |hostname| if @host = Puppet::Rails::Host.find_by_name(hostname.strip) print "Killing #{hostname}..." $stdout.flush @host.destroy puts "done." else puts "Can't find host #{hostname}." end } exit 0 diff --git a/tasks/rake/git_workflow.rake b/tasks/rake/git_workflow.rake index b2f96c603..980d2fbce 100644 --- a/tasks/rake/git_workflow.rake +++ b/tasks/rake/git_workflow.rake @@ -1,121 +1,121 @@ # This set of tasks helps automate the workflow as described on -# http://reductivelabs.com/trac/puppet/wiki/Development/DevelopmentLifecycle +# http://projects.puppetlabs.com/projects/puppet/wiki/Development_Lifecycle def find_start(start) # This is a case statement, as we might want to map certain # git tags to starting points that are not currently in git. case start when nil?: when @next_release: return "master" else return start end end desc "Set up git for working with Puppet" task :git_setup do # This should be changed as new versions get released @next_release = '0.26.x' @remote = {} default_remote = {} default_remote[:url] = 'git://github.com/reductivelabs/puppet' default_remote[:name] = 'origin' @remote[:name] = %x{git config puppet.defaultremote}.chomp @remote[:name] = @remote[:name].empty? ? default_remote[:name] : @remote[:name] @remote[:url] = default_remote[:url] if @remote[:name] == default_remote[:name] default_fetch = '+refs/heads/*:refs/remotes/puppet/*' @remote[:fetch] = %x{git config puppet.#{@remote[:name]}.fetch}.chomp @remote[:fetch] = @remote[:fetch].empty? ? default_fetch : @remote[:fetch] end desc "Start work on a feature" task :start_feature, [:feature,:remote,:branch] => :git_setup do |t, args| args.with_defaults(:remote => @remote[:name]) args.with_defaults(:branch => @next_release) start_at = find_start(args.branch) branch = "feature/#{start_at}/#{args.feature}" sh "git checkout -b #{branch} #{start_at}" do |ok, res| if ! ok raise < :git_setup do |t, args| args.with_defaults(:remote => @remote[:name]) args.with_defaults(:branch => @next_release) start_at = find_start(args.branch) branch = "tickets/#{start_at}/#{args.ticket}" sh "git checkout -b #{branch} #{start_at}" do |ok, res| unless ok raise < 0 raise "Patches already exist matching '00*.patch'; clean up first" end unless %x{git status} =~ /On branch (.+)/ raise "Could not get branch from 'git status'" end branch = $1 unless branch =~ %r{^([^\/]+)/([^\/]+)/([^\/]+)$} raise "Branch name does not follow // model; cannot autodetect parent branch" end type, parent, name = $1, $2, $3 # Create all of the patches sh "git format-patch -C -M -s -n --subject-prefix='PATCH/puppet' #{parent}..HEAD" # And then mail them out. # If we've got more than one patch, add --compose if Dir.glob("00*.patch").length > 1 compose = "--compose" else compose = "" end # Now send the mail. sh "git send-email #{compose} --no-signed-off-by-cc --suppress-from --to puppet-dev@googlegroups.com 00*.patch" # Finally, clean up the patches sh "rm 00*.patch" end