Automating Release Tasks

By | April 11, 2006

I noted earlier today that I’ve made 5 releases of Solar in 7 days. (Clearly I’m a fan of “release early, release often.” ;-) Obviously, I have the Solar-talk subscribers to thank for pointing out bugs and and making functionality requests; these are what drive the need for a release … thanks, guys. :-)

But what I want to talk about in this entry is the release process itself. With the help of Greg Beaver (indirectly) and Clay Loveless (directly), Solar now has a moderate-length PHP script that handles almost all aspects of the release process automatically. Usage is at the command line; issue “php release.php” for a test run, or “php release.php commit” for a full release-and-commit cycle.

With any luck, the lessons I’ve learned here will be of use to someone else; with more luck, perhaps someone else will see possible improvements and mention them here. Read on for a narrative of how the script came to be.

First Iteration

The first person I have to thank here is Greg Beaver and his great work on the PEAR packaging and installation tools. Without his stewardship and maintenenance of this wonderful toolset, the Solar release process would be much more difficult than it is.

Even with the PEAR installer, though, building a package XML release file by hand is a terribly tedious exercise. So the second person I have to thank is Clay Loveless: some months ago, he wrote up the first iteration of the auto-packager for Solar using PEAR_PackageFileManager. I added on a bit to it at that time so that we could keep release information (authors, change notes, etc) in a separate location.

The point of keeping the relase info in a separate location was so we could avoid having to modify the packaging script for every release. The script looks for an “info/” directory; in that directory, there is a sub-directory for every release number, and contained therein are a number text and comma-separated values files. The packaging script then uses those files to generate the “header” portions of the PEAR pacakge file. You can see an example “info/” directory here.

Second Iteration

The PEAR_PackageFileManager has a great auto-replacement utility built in. It can look through the source code at installationpackage time and replace certain keywords, such as “@package_version@”, with the proper value for that particular installation. (UPDATE: [19:33 CST] Greg Beaver pointed out that addGlobalReplacement() does replace at package-time. Once again, Greg proves he has thought of all the right things. Thanks man.)

But there’s a small problem with that for non-PEAR users. Because the replacements happen at installation-time, not at package-time, it means the pearball still has the keywords in place, not the actual values. For methods like Solar::apiVersion() which depend on the automated replacement of “@pacakge_version@” with the correct version number, this is a functionality break for them.

It turns out there’s an easy, if somewhat brute-force, solution: copy the original source to a temporary location, replace keywords yourself, and then have PEAR package that new codebase with the keywords pre-replaced. You can see this portion of the code in release.php at the comment noted “prepare package directory”.

We copy the original source (“src/”) to a packaging directory (“pkg/”) and str_replace() keywords in the new “pkg/” directory. When PEAR_PackageFileManager comes into play, it works on the “pkg/” directory instead of the original source. Note also that the pre-replacements happen in documentation files as well (more on that in a bit).

The end result of this is a pearball/tarball that is equally useful to PEAR and non-PEAR installation processes.

Third (and Current) Iteration

So that takes care of the packaging itself. The last issues to handle are mostly administrative. On a full commit cycle, we need to check a few things before rolling the release:

  • Set Subversion keywords on all files
  • Update the wiki files in “docs/apiref/” (these are built with a custom class that applies the PHP5 Reflection API to each class in Solar; more on that in another post)
  • Make sure all pending changes have been committed to Subversion

If those pre-flight checks pass, we then need to actually build the package (see iterations 1 and 2 above). After that:

  • Tag the release using “svn copy”
  • Commit the tagged release to Subversion
  • Extract the documentation files for updating the Solar website

The release.php script splits this into three task sets.

First, the administrative tasks are handled at the top of the script under “status check”. This sets keywords, builds docs, etc., and then makes sure that there weren’t any changes as a result. If these automated tasks generated changes (as noted by “svn status” returning any ouput at all), the script terminates with a warning to handle outstanding issues.

Second, after the package is built, near the end of the script, we issue a series of “svn” commands to tag the release and commit to the repository.

Finally, we unzip a copy of the pearball and pull out a copy of the docs; these are re-zipped in a separate tarball for upload to the Solar site. We use the packaged docs instead of the source docs so we don’t get all the .svn files mixed in with them.

Manual Tasks

There are still some manual tasks to accomplish in a release, but they’re easy: upload the pearball to the channel server, upload the docs to the website … oh, and write an entry about the release on my blog. ;-)

11 thoughts on “Automating Release Tasks

  1. Ivo Jansch

    For ATK, I’ve automated all but writing the release notes. I just run ./release.sh where is the CVS tag I’ve put on the repository to mark the release. The upload is handled using scp.

    If you are finding yourself releasing 5 releases in 7 days, you might want to consider working with release candidates (people that want stable releases can wait one or two rc’s before downloading the final release) and/or automated nightly builds that people can download.

    Release candidates is useful if you find yourself often fixing bugs right after a release. It ensures more stability for ‘actual’ releases, and you can announce the rc’s to a smaller audience.

    Nightly builds is useful if you commit a lot of features that users want sooner than the next stable release.

    Reply
  2. Ivo Jansch

    That should’ve been “./release.sh tagname where tagname is the CVS tag…” but I had lt/gt’s around it which wordpress decided to strip :)

    Reply
  3. Greg Beaver

    [quote]The PEAR_PackageFileManager has a great auto-replacement utility built in. It can look through the source code at installation time and replace certain keywords, such as “@package_version@”, with the proper value for that particular installation.

    But there’s a small problem with that for non-PEAR users. Because the replacements happen at installation-time, not at package-time, it means… [/quote]

    This is incorrect, for package-info and php-const replacements, all replacements happen at packaging time, but *only* for package.xml 2.0-based packages (which Solar is). Incidentally, this feature is internal to PEAR, PEAR_PackageFileManager simply uses it. If this isn’t happening, we have a problem.

    Reply
  4. pmjones Post author

    Greg, you’re right, it does work as you say. I must have been doing something wrong in my earlier experiments with addGlobalReplacement(). I have updated the entry to reflect your advice; thanks. :-)

    Reply
  5. Rick

    Thanks for the notes. I have been going through the book “Practices of an Agile Developer” and was wondering how a system could auto-build php rather than compiling Java apps and such. Looks like I have some pear reading to do.

    On a side note as I’ve also started doing tests, do you integrate the testing into the release process or just make sure they all go before initiating the new build?

    Reply
  6. pmjones Post author

    Hi Ivo — Good comments.

    As far as release candidates are concerned, these are only alpha-stability releases, so I’m OK with lots of little releases. If I were already at “stable” you can bet I’d be making RCs instead.

    As far as the upload, I’ve not taken the time to see how to push that up to the PEAR channel server yet; it’s a trivial task, so I’m not too concerned about it (yet).

    Reply
  7. pmjones Post author

    Hi Rick –

    That’s a good idea (re: run tests are part of the release script). Right now I just make sure they all run before I do the release. But it would be easy to add a a testing script that does the test-run and only generates output on failures, then it can hook it into the release process the same way “svn status” does. (I already have “all.php” to run all the tests, but it generates output.)

    Reply
  8. Pingback: Bien batido y revuelto » Ésta va de PHP

Leave a Reply

Your email address will not be published. Required fields are marked *