BuildBot: Continuous Integration System

I've spent some of this holiday season learning how to set up BuildBot (http://buildbot.net/) which is a Continuous Integration system that is especially aimed at open-source style projects: You set up a central "build master", and one or more "build slaves" - and it is very easy for someone to set up a new build slave, so if you have some new platform you want to test a project on you can add a build slave. The build-master admin has to add your slave on the master side, and the slave needs to be able to access the source code repository (CVS in my case.)

I've got three of my Perl projects from my CVS repo running under BuildBot now, and it's all working, except email notification of build status - no email is getting to the mail server, and i have yet to learn how to debug that.

When I commit a change to my CVS repository for one of these projects that kicks off a build on one or more build slaves, which checks out the latest code, and runs all the unit tests. The build master shows the results in a web page.

I've got about 9.5 hours into it, (out of an initial estimate of 12) include various yak-shaving activities.

For now I have the build status pages for the three projects at:

• Perl-Metrics-Simple - Building using both MakeMaker and Module::Build. This distro provides modules for a utility program to count lines, packages, subs and complexity of Perl files.
• DBIx-Wrapper-VerySimple - Simplify use of DBI.
• Text-TagTemplate - Lightweight flexible template parsing module.

See also my earlier posting with a review of a book on Continuous Integration and a link to a comparison of several CI systems.

2007-12-29 update: Starting to add notes to the Perl-QA Wiki.
2008-01-06 update: The email issue was caused by a typo in my master.cfg file.
subject='%(builder) BUILD STATUS', WRONG
subject='%(builder)s BUILD STATUS', CORRECT
Note the missing 's' after the closing parenthesis - it is part of a  Python extended printf statement.
Also, I have now added a fourth buildmaster which builds the Parrot project. All my buildbot configurations are now at: http://www.eigenstate.net/buildbot/masters.html
2008-01-17 update: The open source webkit project uses buildbot - you can see their build status at http://build.webkit.org/

Technorati Tags: continuous integration, software development, software testing

Continuous Integration - Improving Software Quality and Reducing Risk

Paul Duvall, Steve Matyas, and Andrew Glover have written a fine book describing the value and practice of Continuous Integration or CI for short. If you have heard of CI and want to learn more about it, or if you want to help educate others about it, this book is a very good place to start.

Continuous Integration is a software development practice intended to notify the development team as soon as possible when a defect is introduced. Typically when CI is being used there is an automated system which the builds the entire project many times each day from its source code to its complete form, and all its automated tests and other automated quality assurance tools can be brought to bear.

The authors of "Continuous Integration" repeatedly emphasize the role that CI has in reducing risk in software development and constantly provide examples of specific practices that support and benefit from CI, for example frequent commits to a version control system, automated tests,  automated code analysis (test coverage, code complexity, duplication, etc.)

I think this book would be great for a leader that is trying to convince their team or their management of the value of CI, as well as for a team implementing CI for the first time as an aid to deciding what system to choose and what aspects to implement first.

Continuous Integration: Improving Software Quality and Reducing Risk
by Paul Duvall, Steve Matyas, Andrew Glover
Paperback: 336 pages
Publisher: Addison-Wesley Professional (July 9, 2007)
Language: English
ISBN-10: 0321336380
ISBN-13: 978-0321336385

See also:

• Codehause.org has a side-by comparison of many Continuous Integration systems.
• Perl QA Continuous Integration wiki page.

Technorati Tags: agile, continuous integration, extreme programming, software development, software testing

Time Machine Very Slow on Leopard

Time Machine will run very slowly if an anti-virus utility such as Norton is scanning each file as it is written or changed on the backup disk.

The fix for this is to go into the Norton Auto-Protect Preference Pane of System Preferences and add your backup disk in the SafeZones tab.

Technorati Tags: Mac OS X

cyradm fails: Can't locate auto/Cyrus/IMAP/imclient_ha.al in @INC

Attempting to login via cyradm fails because of a missing library.


UPDATE 2007-09-15: I believe I have a fix for this. In Cyrus/IMAP/Shell.pm change line 780 from:

if (Cyrus::IMAP::imclient_havetls()) {

to

if (Cyrus::IMAP::havetls()) {

See also: https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=2988

Name : cyrus-imapd
Arch : x86_64
Version: 2.3.9
Release: 6.fc6

$ uname -a
Linux galadriel 2.6.22.1-32.fc6 #1 SMP Wed Aug 1 14:30:16 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux
$ cyradm
cyradm> login matisse
Can't locate auto/Cyrus/IMAP/imclient_ha.al in @INC (@INC contains:
/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi
/usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi
/usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7
/usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5
/usr/lib/perl5/site_perl
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
/usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi
/usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi
/usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7
/usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5
/usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi
/usr/lib/perl5/5.8.8 .) at
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Cyrus/IMAP/Shell.pm
line 780
cyradm>

Evidently I am not the only one getting this problem. See: http://use.perl.org/~jk2addict/journal/32136

Social Dynamics of Pair Programming

The Social Dynamics of Pair Programming

Interesting paper by Jan Chong and Tom Hurlbutt at Stanford University.

Here's the Abstract:

This paper presents data from a four month ethno-
graphic study of professional pair programmers from
two software development teams. Contrary to the cur-
rent conception of pair programmers, the pairs in this
study did not hew to the separate roles of “driver” and
“navigator”. Instead, the observed programmers
moved together through different phases of the task,
considering and discussing issues at the same strategic
“range” or level of abstraction and in largely the same
role. This form of interaction was reinforced by fre-
quent switches in keyboard control during pairing and
the use of dual keyboards. The distribution of expertise
among the members of a pair had a strong influence on
the tenor of pair programming interaction. Keyboard
control had a consistent secondary effect on decision-
making within the pair. These findings have implica-
tions for software development managers and practi-
tioners as well as for the design of software develop-
ment tools.

(Link to PDF version or Google HTML view)

Technorati Tags: agile, extreme programming, pair programming, XP

Finding which jar file provides a Java class.

Given a class name such as:

org.springframework.mail.javamail.MimeMessageHelper

www.jarfinder.com provides a fast easy way to find out which .jar file(s) provide the class, with links to the organization and download site for each one.

For example:

http://www.jarfinder.com/?class=org.springframework.mail.javamail.SmartMimeMessage&submit=search

Technorati Tags: java

Estimating Value of a Possible Software Project

What factors to consider when estimating the value of a possible software project?

I made a list various factors I have considered, more or less formally as the case may be, when estimating the value of undertaking a development project.
When I thought about how I measure these factors I found that I generally think of them as each having a probability-distribution.
Instead of "This feature will cut response time in half", something like "This has a 90% chance of getting of cutting response time by 10%, and maybe a 30% chance of cutting it by 60%."
What factors do you find are most worth spending time to consider?
How might you express those factors in making a go/no-go decision on a project?

Examples, in no particular order:

Value Factors: What will we get from this software?

• capabilities - but then I ask, what is the value of each capability?
• What is the "total market available" for this software/feature?
• How much of that market can we expect to get?
• What new markets will this get us into?
• What cost(s) will this feature help us avoid or reduce?
• What value does this bring to our team - new skills, experience, etc.
• How much fund will this be?
• Synergy with other projects?
• What is the time-to-market value? (what is the value of releasing early vs. later?) Taken to the extreme, that question turns into a cost question: "What is the cost of not releasing at all?"
• Reputation value: What value does this bring to our brand?

Cost Factors: What will it cost us?

• Labor costs
• Hardware costs
• Licensing costs
• Costs associated with various risk factors:
• Reputation cost for overreaching.
• Human cost (burn-out, staff turnover) of doing too much.
• New or increased legal or financial liabilities.
• Delay to other projects.

Technorati Tags: estimating, software development