An excellent idea to track drag...

Kevin E. Schlabach posted about the Snake on the Wall - a great idea for tracking the things that are slowing you down, and giving you a good starting point for your retrospectives.

McDonalds doesn't teach queueing theory

I was in McDonalds today (don't ask), and there was a single line in front of the one attended register. One of the staff opened a second register, but everyone happily stayed in one line. Here's the dialogue that followed:

McDonalds : "Please form a second line here"

Steve : "Why?"

McDonalds : "What?"

Steve : "Why should we form two queues, when one queue is more efficient?"

McDonalds : after 3 or 4 second pause, "Because I say to"

Can't argue with logic like that....

Batch Conversion of Pages file *in Leopard*

I was very proud of my Pages to rtf script, and then I upgraded to Leopard last night and it stopped working! In my search for the answer I came across this post, which solves the problem more thoroughly, though you need to combine it with information from here as well.

In case you don't want to do this yourself, here's a script that will start Pages, prompt you for the type of export you want, and then prompt you for a destination directory, and it works under Leopard. I told you it was more thorough than my last solution :-)

[sourcecode language='jscript']
on open theFiles
tell application "System Events"
if process "Pages" exists then
display dialog "whoops, please close Pages before running droplet!"
end if
end tell
tell application "Pages"
activate
delay 1
close front document
set theList to {"doc", "rtf", "pdf", "txt"}
set theType to (choose from list theList OK button name "Select" with title "Pages Export" with prompt "Choose one or more formats to export using Pages" with multiple selections allowed) as text item
set s to theType as string
set theLocation to choose folder
set theLocation to theLocation as string

repeat with aFile in theFiles
open aFile

if theType contains "doc" then
set asType to "SLDocumentTypeMSWord"
set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to Desktop.
set docPathAndName to theLocation & docNameNew
save front document as asType in docPathAndName

end if
if theType contains "rtf" then
set asType to "SLDocumentTypeRichText"

set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to Desktop.
set docPathAndName to theLocation & docNameNew
save front document as asType in docPathAndName
end if
if theType contains "pdf" then
set asType to "SLDocumentTypePDF"
set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to Desktop.
set docPathAndName to theLocation & docNameNew
set s to save front document as asType in docPathAndName

end if
if theType contains "txt" then
set asType to "SLDocumentTypePlainText"

set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to User Specified Location
set docPathAndName to theLocation & docNameNew
save front document as asType in docPathAndName
end if
try
close front document saving no
end try
end repeat
quit
end tell
end open
[/sourcecode]

Murray Gell-mann, in a TED video....

"In 1957 some of us put forward a partially complete theory of the weak force, in disagreement with the results of seven experiments. It was beautiful and we dared to publish it, believing that all those experiments must be wrong.

In fact, they were all wrong."

I love that in maths and physics truth and beauty are so frequently aligned, and that misalignment often indicates immature understanding. A new insight can reveal new beauty.

I think the same is true of programming. Your code should be at least as elegant as your problem domain (I can't do anything about accounting *s*).

Batch conversion of Pages files

My current assignment makes me a Mac weenie in a Microsoft world, so while I'm happily tootling along in Pages I need to be able to share stuff with my colleagues. At the moment I'm the primary author of 30+ short course descriptions, stored in separate Pages files, and I've been exporting them to RTF when I need to share them. That was initially ok, but got out of hand as the number of files grew, so yesterday I was introduced to AppleScript.

I know nothing about AppleScript, but I was able to find a script on the web (thanks again, Google) that did most of what I want. I'll show you the script, then tell you the parts that weren't obvious to me and slowed me down

[sourcecode language='jscript']
on open theFiles
tell application "Pages"
repeat with aFile in theFiles
open aFile
set docName to name of front document
-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"
-- Add .pdf extension.
set docName to first text item of docName & ".rtf"
set AppleScript's text item delimiters to prevTIDs
-- Save file to Desktop.
set docPathAndName to (path to desktop as string) & "rtfs:" & docName
save front document as ".rtf" in docPathAndName
close front document
end repeat
end tell
end open
[/sourcecode]

My first challenge was that I couldn't figure out how to get this to run! Pressing "Run" in ScriptEditor did nothing. I ended up saving it as an applet (use "Save As", selecting FileFormat => application), and then I could drag and drop the files I wanted to convert onto the applet. Once I put the applet in the dock, that was pretty convenient.

The second thing that slowed me down was figuring out how to save the results to a folder (the original script saved to the desktop). What I needed was the bit ' & "rtfs:"' while setting the docPathAndName. Here I'm dealing with a string, and it already has a trailing ":" - I kept trying to add another one, and AppleScript gave me a message about the target not being writable, rather than not existing, which lead me astray for a while.

Anyway, now I can select my Pages files, drag and drop them onto the applet in my dock, and the rtf files appear in a folder on my desktop. Sweet!

Releases with themes

When I'm doing agile training, I commonly advise customers to give names to releases and iterations to help them decide which stories belong in which iteration, but it's not something that I see many customers do and since I rarely take that role not something I do much myself either. But today I got some direct experience!

Cogent is building a software application to help you organize your personal work, and I'm acting in the sponsor role. The details of the functionality are being sorted out by someone else, but I'm the person focussed on the operational requirements and how we get it to the billable stage as quickly as possible (everyone else in interested in this too, but it's my particular role). Yesterday afternoon I added operational stories to our list, and then prioritised the list story by story, and got what looked like a reasonable response.

Today I went back and created some themed releases, and gave them names.

1. 
   Homely - Ugly, but interesting. This release is usable, but won't win any design awards. We want it first because we're using the application internally now, so functionality is important. This is the release we'll pass to "friendly" testers for feedback, so it needs to support a small set of concurrent users without getting too slow. Codename : Homer


2. 
   Prettified - Like Homer, but certainly nicer looking and generally more usable. We still won't be confident in our backup plan, nor have stress tested the application, but it will look professional and be generally usable. Codename : Marge
   


3. 
   Robust - we need this one to be tough, so it can withstand the abuse of a lot of people. This is the point where we can do an open beta - still not charging, but giving the general public access. Codename : Bart
   


4. 
   Billable - something that we feel we could reasonably charge money for. Codename : Lisa
   

With these releases in mind I went back and looked at my priorities again, and found that some of them were quite different. If you are a customer on an agile project, I strongly encourage you to do the same thing - you may be surprised by the difference it makes.

Another "I wish I had said that"

“Simplicity and elegance are unpopular because they require hard work and discipline to achieve and education to be appreciated.” - Edger W Dijkstra

Adventures in BackgroundRb

We've been asked to produce some "nice" reports in PDF for a client, and getting it all done has been quite an adventure. We use Ruport for the reporting, which unfortunately means we also need to learn quite a bit about PDFWriter to get everything looking spiffy, and the report rendering does take a while, but we had it all working in a controller yesterday and things were looking good. Then we deployed the application to EngineYard.

The first thing that happened was that we got timeout errors that looked like this (courtesy of ExceptionNotifier):

A Mongrel::TimeoutError occurred in reports#cost_detail_pdf:
Mongrel timed out this thread: shutdown
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:221:in `within'

The support folks at EngineYard suggested that we shouldn't be running PDF generation from within our controller, and that we should use something like background job, but since we wanted to report status as the job proceeded and we only had a small number of jobs, we went with BackgroundRb instead. (If you care about the difference, look for 'Backgroundrb and Bj serve different purposes' in this thread).

BackgroundRb works great once you get it going, but there were examples in the documentation that simply didn't work for me. Here's what worked for me in the end:

In my controller:

[sourcecode language='ruby']
MiddleMan.new_worker(:worker => :cost_detail_pdf_worker, :job_key => current_user.id)
MiddleMan.ask_work(:worker => :cost_detail_pdf_worker, :job_key => current_user.id,
:worker_method => :build_report,
:data => [session[:production_id], file_name])
[/sourcecode]

and my worker:

[sourcecode language='ruby']
class CostDetailPdfWorker calculated_report, :template => :default)
register_status("Finished Rendering")
File.open(file_path, "w") do |file|
file << pdf
end
register_status("File available")
end

end
[/sourcecode]

The next part of the adventure was getting this running on our Slicehost slice. Unfortunately, BackgroundRb requires Ruby 1.8.5, and we only had 1.8.4 (on Ubuntu 6.06 LTS). Apt-get for Ubuntu 6.06 doesn't include Ruby 1.8.5, so I needed to install Ruby 1.8.5 from source using these excellent instructions.

Unfortunately, I also needed to reinstall all our gems to go with the new version of Ruby. This was mostly stragithforward, except for Postgres. The postgres gem wouldn't build the native extensions, failing to find pg_config. These instructions were close, but before that would work I also needed to do "apt-get install libpq-dev" (you might also need libpgsql-ruby and/or libpgsql-ruby11.8, but I already had those installed).

Ok, looks pretty good now. I started the backgroundrb server manually as a test, and could produce my pdf files, so the next step was to move to our production environment at Engineyard. This was the first time I tried to stop and start the BackgroundRb using Capistrano - unfortunately you can't simply repeat the command lines in your Capistrano tasks. Fortunately, there's also a description of some tasks that do work (make sure you use the start task in the comments at the end of post).

All in all, a nice outcome but with much more pain than I would have liked.

Design in TDD

There's been a recent discussion in the Melbourne XP Enthusiast Group (MXPEG) mail list about whether or not TDD 'works', with part of the thread also being about Story Driven Development (SDD). Some of the folks have reported a lot of success using SDD and its associated tests (that I would call acceptance and integration tests), to the point where they don't use unit tests at all. The impression I get is that with this approach they're working predominantly top down. I'm glad this works for some people, but I don't think it would work for me, and I wanted to take a few minutes to say why.

I think it's important to have both unit tests (developer tests) and acceptance tests (customer tests), whether they're implemented in the same tools by the same people or quite different tools by quite different people. Notice though that I said "implemented" - to me the two groups of tests have different owners, different perspectives, and serve different purposes.

Customer tests - they tell us if the application is sick or not, but they don't provide any indication of where or why.

Developer tests - they tell us exactly where something is going wrong, but don't necessarily tell us what the impact on the overall application would be.

I would like to have both of these, not one or the other. I also find that I personally don't design entirely either top down or bottom up. I do some top down stuff to get an idea of the problem, then I use that to hypothesise about a design that will meet the requirements. Once I have some confidence that my design looks reasonable, I implement parts of it bottom up, and I want unit tests to confirm that my implementation of each component works the way I expected. I keep switching back and forth between top down and bottom up as I go.

The thing I do badly is that I don't write acceptance tests until the end (if at all). I know this isn't what I'm supposed to do, I know it probably detracts from my overall quality, but it is what I find myself doing. So far I haven't found a tool that makes writing those tests easy enough for me (personal opinion only).

I wish I had said this...

... but Bob Martin did :

One of the developers asked the question point blank: “What do you do when your managers tell you to make a mess?” I responded: “You don’t take it. Behave like a doctor who’s hospital administrator has just told him that hand-washing is too expensive, and he should stop doing it.”

Coupling rollback actions with transaction actions

The system I'm currently developing needs to import a variety of data from another system. The data arrives in batches, and within a batch some items might be valid and some might be invalid - I expect that this is a common problem. An item in the batch might also update many different objects within my domain model, and I won't necessarily know if the item is invalid until I try to commit the changes to the domain model. It became fairly clear that I had some recurring code patterns in my application, so of course I wanted to extract them into some abstraction and Ruby's reflection gave me a nice way to capture this.

What I needed was to be able to associate a piece of code that created or modified some domain objects with the code that cleaned up the changes to the domain model on rollback (assuming that the database transaction handled the persistence parts). I could have used blocks, but I find that unlabelled blocks aren't very expressive so I decided to use methods (and their names) instead of blocks. Here's an example of using my method, #within_transaction_with_rollback; in the example :create_reference_objects and :clear_reference_objects are methods defined somewhere else in the class.

[sourcecode language='ruby']
def process
within_transaction_with_rollback(:create_reference_objects, :clear_reference_objects)
end
[/sourcecode]

And here's the implementation of my method, for those who are interested. I'm sure that there are plenty of other possible implementations :-)

[sourcecode language='ruby']
module Transactions

def within_transaction_with_rollback(transaction_method, rollback_method)
begin
within_transaction(transaction_method)
rescue
self.send(rollback_method)
end
end

def within_transaction(create_method)
self.class.transaction do
self.send(create_method)
end
end

end

[/sourcecode]

Interaction between finder_sql and Active Scaffold

I came across an interesting interaction in Rails that depended on two things that I didn't know (of course, that list is infinite). The two things were:

1. Active Record ignores finder_sql when you do an eager load of an association (though I can't find a definitive reference for this, it is the observed behaviour);
   
2. Active Scaffold uses eager loading by default
   

I didn't have any specs that explicitly tested that my model could eagerly load its associations, but everything else seemed to be working, so I was surprised that Active Scaffold was unhappy. The solution was to tell Active Scaffold not to use eager loading on that particular association, following the instructions in the FAQ under "my database is choking" (which kind of describes what was happening to me, if the choking was fatal).

Need a Rails-savvy, Java tainted, agile bigot?

My involvement in a project has finished unexpectedly early, so I find myself available for a new gig without anything in the pipe - which is unusual for me. If you have anything that you think I could help with, especially short- or part-time, drop me a line (steve at cogentconsulting dot com dot au).

Rspec View specs and integrate_views

We've recently had the pleasure of having Craig Ambrose working on a project for us, and as you'd expect when you bring someone new into the team, we've been doing some storming. One of the topics that comes up repeatedly within development teams I'm familiar with, including ours, is how fanatically we should adhere to clear separation of unit and integration testing.

I'm fairly laissez faire about the issue. I'm quite happy to write tests for Rails models that actually invoke ActiveRecord and interact with the database. I'll also happily set up a network of associated objects in the database rather than mock out everything except the class under test, though I do have some ill-defined limit that makes me uncomfortable.

I'm more strict about controller tests - I want the controller itself to be wafer thin, and I'll generally only test that the controller sets the right instance variables (along with flash notices etc). The tests end up being fairly small, don't touch the view at all, and I think even the evangelical TDD folk would call them unit tests.

However, Craig rightly points out a couple of problems with this. I'm typically working with server side functionality (if I had to say what I was best out, I'd initially claim domain modelling), so I write crap view code and I rarely write tests for it (someone's just going to come along and rewrite it to be presentable anyway). It's a fairly pathetic defence of my laziness, but anecdotally it seems lots of people don't write view specs. So there's a whole area of my application that's not tested very well.

Craig also points out that one of the biggest sources of defects for him is that his controller doesn't set up what the view expects (or you can express if from the opposite perspective if you prefer) - the failure occurs during the interaction between the controller and the view. These sorts of failures are quite difficult to find with unit tests alone, and probably won't get picked up by distinct controller specs and view specs.

We could write separate integration specs, but RSpec gives a simple way to catch at least the most egregious of these problems. Unit testing purists might object, but the laissez faire'sts won't mind. Put "integrate_views" into your descriptions. When integrate_views is specified, RSpec renders the real view rather than mocking out the rendering, and if objects are missing or badly misconfigured you'll get a rendering exception.

I remembered the integrate_views option from earlier versions of RSpec - it's not particularly conspicuous in the current version's documentation, but it can be very useful.

Using the latest attributes in a Rails migration

Occasionally I need to make some change to a table via migrations, and then immediately use the attributes associated with the latest table inside the migration. Most of the time you can do this by defining the aspects of the class inside the migration - for example:

[sourcecode language='ruby']
class AddDepartmentToProducts < ActiveRecord::Migration

class Product < ActiveRecord::Base
belongs_to :department
end

def self.up
add_column :products, :department_id, :integer
# do something with product.department
end

def self.down
#...
end

end
[/sourcecode]

There's also another way that's not as neat, but gives you finer grained control, for example if you need to have different definitions of the class in the up and down migrations:

[sourcecode language='ruby']
class AddDepartmentToProducts < ActiveRecord::Migration

def self.up
add_column :products, :department_id, :integer
Object.class_eval <<-end_eval
class Apparel21::Product < ActiveRecord::Base
belongs_to :department
end
end_eval
# do something with product.department
end

def self.down
#...
end

end
[/sourcecode]

ActiveScaffold reverse associations to models in modules

I really enjoy using ActiveScaffold, and encourage everyone to use it to generate administration style interfaces. I came across an apparent defect today that I wanted to share.

I have a set of models that are contained in a module, and I'm using AS to do administration for them. Mostly fine. However there's a problem when I click a link on an element of a many relationship that's being displayed in a list. In this image,

ranges is the result of a has_many relationship, and when I click on the range 'Summer Selection', I get this:

and the following stacktrace:

[sourcecode language='ruby']
ActionView::TemplateError (undefined method `reflect_on_all_associations' for Range:Class) on line #19 of vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:
16: # determine what constraints we need
17: if column.through_association?
18: @constraints = {
19: association.source_reflection.reverse => {
20: association.through_reflection.reverse => parent_id
21: }
22: }

vendor/plugins/active_scaffold/lib/extensions/reverse_associations.rb:26:in `reverse_matches_for'
vendor/plugins/active_scaffold/lib/extensions/reverse_associations.rb:12:in `reverse'
vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:19:in `_run_erb_47vendor47plugins47active_scaffold47frontends47default47views47_nested46rhtml'
vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:11:in `each'
vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:11:in `_run_erb_47vendor47plugins47active_scaffold47frontends47default47views47_nested46rhtml'
vendor/rails/actionpack/lib/action_view/base.rb:637:in `send'
vendor/rails/actionpack/lib/action_view/base.rb:637:in `compile_and_render_template'
vendor/rails/actionpack/lib/action_view/base.rb:365:in `render_template'
vendor/rails/actionpack/lib/action_view/base.rb:316:in `render_file'
vendor/rails/actionpack/lib/action_view/base.rb:331:in `render_without_active_scaffold'
[/sourcecode]

The problem is that class should be MyModule::Range, not Range.

AS is looking up the class using code on ActiveRecord::Reflection::AssociationReflection, from the file activescaffold/lib/extensions/reverse_associations.rb. The fix (apparently - I haven't tested this exhaustively yet) is to change the code that sends "class_name.constantize" to the association to send "klass" instead (you need to do this in two places). The completed code is attached below. I hope this helps someone else!

[sourcecode language='ruby']
module ActiveRecord
module Reflection
class AssociationReflection #:nodoc:
def reverse_for?(klass)
reverse_matches_for(klass).empty? ? false : true
end

attr_writer :reverse
def reverse
unless @reverse
# Following line changed for compatibility with associations on classes in modules
# reverse_matches = reverse_matches_for(self.class_name.constantize)
reverse_matches = reverse_matches_for(self.klass)
# grab first association, or make a wild guess
@reverse = reverse_matches.empty? ? self.active_record.to_s.pluralize.underscore : reverse_matches.first.name
end
@reverse
end

protected

def reverse_matches_for(klass)
reverse_matches = []

# stage 1 filter: collect associations that point back to this model and use the same primary_key_name
klass.reflect_on_all_associations.each do |assoc|
# skip over has_many :through associations
next if assoc.options[:through]

## Following line changed for compatibility with associations on classes in modules
# next unless assoc.options[:polymorphic] or assoc.class_name.constantize == self.active_record
next unless assoc.options[:polymorphic] or assoc.klass == self.active_record
case [assoc.macro, self.macro].find_all{|m| m == :has_and_belongs_to_many}.length
# if both are a habtm, then match them based on the join table
when 2
next unless assoc.options[:join_table] == self.options[:join_table]

# if only one is a habtm, they do not match
when 1
next

# otherwise, match them based on the primary_key_name
when 0
next unless assoc.primary_key_name.to_sym == self.primary_key_name.to_sym
end

reverse_matches < 1

reverse_matches
end

end
end
end
[/sourcecode]

Uses of OpenID

I'm a user of OpenID, but this video from Google is an excellent introduction to the concept, the uses, and the pros and cons.

Particularly interesting to me as a developer were the ideas of using OpenID as a 'lightweight' authentication mechanism for sites that want a low barrier for registration and access, and recognizing multiple openids so that a site can access id-specific features from each. Highly recommended!

Rspec 1.1.2 and Textmate

A quick warning to everyone - if you upgrade to RSpec 1.1.2 and you use Textmate, you'll probably need to update your textmate bundle as well.

cd ~/Library/Application\ Support/TextMate/Bundles/

svn co svn://rubyforge.org/var/svn/rspec/trunk/RSpec.tmbundle

Freezing gems with architectures

So I don't ever lose this reference again (hopefully), here's a reminder to myself (and maybe to you) that I want to freeze all my gems, and accommodate different architectures, so I should be using gems_with_architecture. See
usage, and browse the
repository.

Beanstalk Dashboard

Since I've failed to get the beanstalk dashboard to display in my comments, here's the "team leader" view that Mark was asking about in the last post:

Excellent support from Beanstalk

We've started evaluating Beanstalk for our subversion repositories, and I had a problem this morning trying to commit to my second repository (the detail is that I was getting the message "Can't create directory '/var/lib/subversion/beanstalk.storage/1911/webistrano/db/transactions/1-1.txn': Permission denied"). Apparently this is a recurring problem on some accounts - probably a teething problem with the service.

Of course I'd rather not have any problems at all, but when I got this message I logged into the Beanstalk Campfire conversation, and within 10 minutes (9 of which were me getting a drink while Chris worked) Chris Nagele has fixed my problem and I was up and running again. Great stuff!

Interesting behaviour for defined?

I've been working with Pete Yandell's Not-a-mock plugin, which I like a lot, and this morning a failing spec led me to some interesting behaviour for the Ruby defined? operator. Try these two things:

defined? arbitrary_attribute

defined? arbitrary_attribute=

With Ruby 1.8.5 on my Mac, the second one fails! From experimenting, it doesn't seem like you can use any setter. The workaround (which I haven't confirmed with Pete yet) is to use 'self.methods.include?("arbitrary_attribute-").

Don't be dogmatic about dogma

A colleague of mine recently passed around a reference to "The Way of Testivus" from Agitar, and as promised on the front page I did indeed find that it was filled with "good advice on developer and unit testing" [although personally I haven't found testing my developers to be so useful :-)]. But there was one page that I thought would be dangerous in the wrong hands, and it started like this....

Don’t get stuck on unit testing dogma

Dogma says:
“Do this.
Do only this.
Do it only this way.
And do it because I tell you.”

Dogma is infl exible.
Testing needs fl exibility.

Dogma kills creativity.
Testing needs creativity.

I've got a good grasp of what the author means, and I agree with the sentiment quite strongly. But I'm also confident that out there somewhere is someone new to automated testing who's saying "oh, I've seen how other people do testing, and how could they have gotten it so wrong? I'm going to do things completely differently - I'm going to be creative". The trouble with rejecting dogma is that "dogma" is often a great approach to 80% of the problem, and it often contains a great deal of wisdom that isn't obvious to a newcomer.

I'm all in favour of rejecting dogma, but only after you've understood why those ideas might have become dogma in the first place, and you're confident that you understand the reasons that particular dogma doesn't apply in your specific context. Otherwise you're just being dogmatic about rejecting dogma.

Installing Postgres on Leopard

Excellent article on installing Postgres from binaries on Leopard. Just make sure you have XCode installed first.