[Pkg-crosswire-devel] Automated Sword module packaging: it works

Mon Jan 26 11:28:25 GMT 2009

Today I have been able to script the process of converting Sword modules
into Debian packages.  While the script is, shall we say, not
necessarily bulletproof (!), it *is* packaging a pile of Sword modules
for me as I type this.

The resulting modules are likely to benefit from manual checking and
probably some manual edits to control files etc. before release, of
course, and Matthew has privately suggested some interesting ideas about
automatically generating lucene indexes to go with them... but I think
the basic principle of automated module to package conversion can now be
considered to be proven.

Below are module2deb which does the conversion/packaging, and
package-modules.sh, a script that grabs the Crosswire archive of Sword
modules and then packages all of them which are labelled as being
"Public Domain".  I hope word-wrap won't destroy them beyond all
recognition.

module2deb expects that whoever runs it has a normal packaging
environment set up, complete with $DEBFULLNAME and $DEBEMAIL, and by
default it does do package signing, so for bulk use you have to set up
gpg-agent or seahorse or whatever so that you don't have to type your
GPG passphrase twice for each package build!  It generates module names
including a component based on ModDrv, so the names try to distinguish
between text and commentary and so forth.  How useful that feature is,
is definitely up for discussion...!

[I see that module2deb seems to have at least one bug; it is not
handling lines in the module *.conf files that contain parentheses
sanely at the moment... probably a quoting issue somewhere.]

The build has now finished, and I have 130 sword-*.deb (and *.dsc, etc)
files!  I'm not about to upload them to my PPA, though.

The lintian warnings are almost all about the changelogs not closing an
ITP bug (impossible to 100% automate that!), and then a few (which I'll
try to check up on) about the description and synopsis fields in the
control file (which module2deb tries to generate from the Description
and About fields in the module .conf file).

NOTE: Please do *not* run package-modules.sh if you do not intend to
work on this stuff, as it will download around 270MB of modules from
Crosswire the first time you run it.  The Crosswire archive does not
seem to have an obvious way to download just the metadata (just the
*.conf files), so one can then download only the Public Domain
modules... so the script downloads all of them, but then only builds the
PD ones.  If there is a better way to do this, please let me know.

Of course, no-one who does objects to them needs to to use these
scripts, or the resulting packages they generate.  No-one is being
forced to do anything, here -- I'm simply providing tools.  The
resulting packages could be submitted to Debian and/or Ubuntu, or could
be put into a PPA or a local "private" repository; none of that is being
dictated by the existence of the tools in any way.

I'm not sure quite where the scripts would belong in the team git
repository, but I'll happily commit them there.  Do we need a new
"modules" chunk of git space for things like this?

Jonathan

#--------------------------------------------------------------------

#!/bin/bash
# module2deb - converts a Sword module .zip file into a Debian source
package
#              Expects to be able to create the package, the orig.tar.gz
and
#              a package dir in the current directory.
#
# Author: Jonathan Marsden <jmarsden at fastmail.fm>
# Copyright: Copyright (C) 2009 Jonathan Marsden
# Licence: GPLv2 or later
# Version: 0.1

distribution="unstable" # Set this to intrepid, jaunty etc. for Ubuntu
debuild_options=""	# Can set to "-us -uc" to avoid GPG signing

[ -z "$1" ] && echo -e "$0: Usage is: $0 zipfile ...\n" && exit 1

function line2var () {
  # function line2var takes one parameter which is the entire line.
  # Most lines in the .conf files are of the form varname=value
  # The first line of each file can be a tag of the form [tag].
  # In this exceptional case, we set conffiletag to the [tag] string.
  varname=${1%%=*}
  value=${1#*=}
  if [ "${varname:0:1}" = "[" ]; then
    value="$varname"
    varname=conffiletag
  fi
  varname=${varname/./_}	# Periods not allowed in bash varnames
  eval ${varname}=\""$value"\"
  #echo "l2v DEBUG $varname is ${!varname}"
}

# Main loop (once through loop for each zipfile)
while [ -n "$1" ]
do
  zipfile="$1"

  # Can we read the file?
  if [ ! -r "$zipfile" -o ! -s "$zipfile" ]; then
    echo "$0: ERROR: $zipfile is not a readable zipfile"
    exit 2
  fi

  # Unzip zipfile into a temporary location
  tempzip="/tmp/$$/$zipfile"
  mkdir -p $tempzip
  if [ ! -d "$tempzip" ];then
    echo "%0: ERROR: $tempzip dir not created"
    exit 3
  fi
  unzip -nqq "$zipfile" -d "$tempzip"

  # Grab data from mods.d/*.conf file into shell variables
  # We need $Version for the version number, and so forth
  unset Version DistributionLicense conffiletag Description About
  Version=${Version:-1.0}	# a few modules have no Version line in their
.conf
  while read -r x ; do line2var "$x" ; done <${tempzip}/mods.d/*.conf
  # Convert ModDrv into a moduletype, removing any leading z
  moduletype=`echo $ModDrv |tr [A-Z] [a-z]`
  [ "${moduletype:0:1}" = "z" ] && moduletype=${moduletype:1}

  # Make sure this module is sufficiently free
  free=no
  case "$DistributionLicense" in
    "Public Domain") free=yes ;;
    "General public license for distribution for any purpose") free=yes ;;
  esac

  if [ "$free" != "yes" ]; then
    echo "$0: ERROR: $zipfile has non-free DistributionLicence of
$DistributionLicense"
    exit 4
  fi

  # Convert file.zip to file-$Version_orig.tar.gz
  stem=`basename $1 .zip`
  stem=`basename $stem .ZIP` # just in case we have any .ZIP files
  stem=`echo $stem |tr [A-Z] [a-z]`
  tarname="sword-${moduletype}-${stem}-$Version"
  origtarname="sword-${moduletype}-${stem}_$Version"
  tardir="$tempzip/$tarname"
  mkdir -p "$tardir"
  unzip -nqq "$zipfile" -d "$tardir"
  tar -z -c -C "$tempzip" -f "$origtarname.orig.tar.gz" "$tarname"

  # Unpack tarball
  [ -d "$tarname" ] && echo "$0: ERROR: directory $tarname already
exists" && exit 5
  tar zxf "$origtarname.orig.tar.gz"

  # Create source package tree debian/ files
  cd $tarname
  mkdir debian
  datestamp=`date -R`

  cat >debian/changelog <<EOF
sword-${moduletype}-$stem ($Version-1) $distribution; urgency=low

  * Initial Release.

 -- $DEBFULLNAME <$DEBEMAIL>  $datestamp
EOF
  echo 5 >debian/compat

  # Word-wrap $About ready for use in control file
  echo "$About" >$tempzip/about
  sed -i 's/\\par/\n\n/g' $tempzip/about	# Deal with \par meaning
paragraph break
  fmt -s $tempzip/about >$tempzip/about2
  sed -i 's/^ *$/./' $tempzip/about2	# Make sure para breaks are exactly "."
  sed -i 's/^/ /' $tempzip/about2	# Indent everything one space

  cat >debian/control <<EOF
Source: sword-${moduletype}-$stem
Section: text
Priority: optional
Maintainer: $DEBFULLNAME <$DEBEMAIL>
Homepage: http://www.crosswire.org/sword/modules/
Build-Depends: debhelper (>= 5)
Standards-Version: 3.8.0

Package: sword-${moduletype}-$stem
Architecture: all
Suggests: libsword7
Provides: sword-text
Enhances: sword-frontend
Description: $Description
EOF
  cat $tempzip/about2 >>debian/control

  cat >debian/copyright <<EOF
This package was debianized by $DEBFULLNAME <$DEBEMAIL> on
$datestamp .

It was downloaded from

  http://crosswire.org/ftpmirror/pub/sword/packages/rawzip/

Based on the information in the upstream .conf file, the
"Distribution Licence" for this module is:

$DistributionLicense

EOF
  #include $About (formatted) if relevant
  grep -sqi copyright $tempzip/about2 &&
    cat $tempzip/about2 >>debian/copyright
  #include info on decompression
  cat >>debian/copyright <<EOF

The text as distributed is in a compressed format.  You can use the mod2imp,
mod2osis or mod2vpl commands from the libsword-tools package in order to
extract the text into an uncompressed form.
EOF

  cat >debian/rules <<EOF
#!/usr/bin/make -f
# -*- makefile -*-
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of
dh-make.

configure: configure-stamp
configure-stamp:
	dh_testdir
	touch configure-stamp

build: build-stamp
build-stamp: configure-stamp
	dh_testdir
	touch build-stamp

clean:
	dh_testdir
	dh_testroot
	rm -f build-stamp configure-stamp
	dh_clean

install: build
	dh_testdir
	dh_testroot
	dh_clean -k
	dh_installdirs
	install -d m* debian/sword-$moduletype-$stem/usr/share/sword

# Build architecture-independent files here.
binary-indep: build install
	dh_testdir
	dh_testroot
	dh_installchangelogs
	dh_installdocs
	dh_compress
	dh_fixperms
	dh_installdeb
	dh_shlibdeps
	dh_gencontrol
	dh_md5sums
	dh_builddeb

# Build architecture-dependent files here.
binary-arch: build install
# We have nothing to do by default.

binary: binary-indep binary-arch
.PHONY: build clean binary-indep binary-arch binary install configure

EOF
  chmod 0700 debian/rules

  # Clean up
  [ -n "$tempzip" -a -d "$tempzip" ] && rm -r "$tempzip"

  # Package it!
  echo "$0: Packaging $1 in `pwd`"
  debuild $debuild_options
  debuild -S -sa $debuild_options

  # Even more clean up (comment out when debugging this script)
  cd ..
  [ -n "$tarname" -a -d "$tarname" ] && rm -r ./"$tarname"

  # Get ready to go back and process the next zipfile
  shift
done

#--------------------------------------------------------------------

#!/bin/bash
# package-modules.sh - downloads and packages packages a pile of Sword
modules

ziparchivedir=~/packages/sword/modules/zipfiles		# Module archive dir
builddir=~/packages/sword/modules			# Package build area
m2d=$builddir/module2deb				# Path to module2deb script

# Create/update zip archive of Sword modules from Crosswire
mkdir -p $ziparchivedir
cd $ziparchivedir
wget -m --no-parent -nH -nd
http://crosswire.org/ftpmirror/pub/sword/packages/rawzip/
rm index.html* robots.txt	# Get rid of junk that is not .zip modules

# First convert KJV to a package (because it is freely redistributable
but not labelled PD)
mkdir -p $builddir
cd $builddir
$m2d $ziparchivedir/KJV.zip

# Then build all the rest of the modules that are labelled as being
public domain
cd $builddir
for i in $ziparchivedir/*zip
do
  unzip -p $i 'mods.d/*.conf' |grep -sqx 'DistributionLicense=Public
Domain' && $m2d $i
done

#--------------------------------------------------------------------