[Pkg-puppet-devel] RFP: puppetdb-termini -- Enable a Puppet master to connect to PuppetDB

[Apollon Oikonomopoulos]

Control: clone -1 -2
Control: retitle -2 puppet: does not support exporting storeconfigs to PuppetDB
Control: reassign -2 puppet
Control: found -2 puppet/3.7.2-4

Hi micah, all,

On 12:07 Wed 01 Feb     , micah wrote:
> micah <micah at riseup.net> writes:
> 
> > I agree that it doesn't look hard to add the terminus package, but I was
> > hoping we could provide some kind of upgrade path for people to keep
> > their storedconfig database, but I can't seem to figure out what is
> > going on here.
> 
> Ok, I got it working:
> 
> 1. wget http://downloads.puppetlabs.com/puppetdb/puppetdb-2.3.8.tar.gz
> 2. verify and uncompress it
> 3. cp -avp puppetdb-2.3.8/ext/master/lib/puppet/* /usr/lib/ruby/vendor_ruby/puppet/
> 4. copy active record dbadapter details to [main] section of puppet.conf
> 5. apply the attached patch[0] to
> /usr/lib/ruby/vendor_ruby/puppet/face/storeconfigs.rb

> --- storeconfigs.orig.rb	2016-08-24 09:04:48.428728886 +0000
> +++ storeconfigs.rb	2016-08-24 09:51:34.658495419 +0000
> @@ -35,16 +35,15 @@
>          begin
>            Puppet::Rails.connect
>  
> -          # Fetch all nodes, including exported resources and their params
> -          nodes = Puppet::Rails::Host.all(:include => {:resources => [:param_values, :puppet_tags]},
> -                                          :conditions => {:resources => {:exported => true}})
> -
> -          catalogs = nodes.map { |node| node_to_catalog_hash(node) }
> -
>            catalog_dir = File.join(workdir, 'catalogs')
>            FileUtils.mkdir(catalog_dir)
> -
> -          catalogs.each do |catalog|
> +         
> +          nodes = []
> +          # Fetch all nodes, including exported resources and their params
> +          Puppet::Rails::Host.find_each(:include => {:resources => [:param_values, :puppet_tags]},
> +                                        :conditions => {:resources => {:exported => true}}, batch_size: 1) do |node|
> +            catalog = node_to_catalog_hash(node)
> +	    nodes << node[:name]
>              filename = File.join(catalog_dir, "#{catalog[:data][:name]}.json")
>  
>              File.open(filename, 'w') do |file|
> @@ -52,7 +51,7 @@
>              end
>            end
>  
> -          node_names = nodes.map(&:name).sort
> +          node_names = nodes.sort
>  
>            timestamp = Time.now

> 6. puppet storeconfigs export
> 7. copy the exported file to the server running puppetdb software and
> import the data with:
> puppetdb import --infile ./storeconfigs-2017XXXXXX.tar.gz
> 
> This wouldn't be too hard to make work in a debian package, so people
> can actually upgrade, but we need the termini package first.
> 
> micah
> 
> 0. https://tickets.puppetlabs.com/browse/PDB-165

Glad to see this work!

As it seems, we have two issues here:

 1. The Puppet 3.7 master in Jessie does not support `puppet 
    storeconfigs export'
 1. The Puppet 4.8 master in Stretch does not have the PuppetDB terminus 
    readily available

I'm cloning this as a new bug for the export support missing in puppet 
3.7. I think the best course of action is to update puppet in Jessie via 
a stable update to include the following files from the PuppetDB 2.3.8 
source:

 ext/master/lib/face/storeconfigs.rb (patched)
 ext/master/lib/application/storeconfigs.rb
 ext/master/lib/util/puppetdb/*

I have already prepared an update (3.7.2-4+deb8u1, available on [0]) and 
will file for a jessie-pu to get the SRM's opinion on this.

I think we should deal with Puppet 4 supporting PuppetDB the same way 
(i.e. ship the termini directly in puppet) via an unblock request after 
4.8.2-1 has migrated to testing.

Opinions/ideas?

Regards,
Apollon

[0] https://anonscm.debian.org/cgit/pkg-puppet/puppet.git/log/?h=storeconfigs-export