Cogent Values and Goals

Cogent recently had an off-site meeting that we used for a wide-ranging discussion, including a session on what we considered our values and goals as an organisation. We had a fairly spirited exchange on both these topics, constrained by finding at most five points in each area that captured our business. There were many things on the margins that people thought were important, so it was difficult to hone the lists to a concise consensus. In the end we had this:

Cogent Values

• Openness


• Integrity


• Respect


• Craftsmanship


• Collaboration

Cogent Goals

• Make more money than we can from consulting


• Build some kick-ass software


• Work with good people


• Have flexibility in lifestyle and the amount of time I want to work

I was going to add some commentary, but on reflection I think these speak for themselves (but I'm interested in your comments).

For your interest, here are the next group of goals - things that influence us, but that aren't core and didn't make it into the definitive list:

• Prove that you can run a company humanely for both customers and employees


• Learn new things, learn proven things from experienced folk


• Grow as practitioners


• Change the way people develop software


• Inspire one another and others


• To have some fun

So while we would like to change the world and have some fun, we're realistic enough to understand that for us, for the people we are and the stage of life that most of us are in, there are some selfish things (money and flexibility) that are actually more important. That doesn't really make me feel great about myself, but I think it's the right decision and I think it's good that we've been clear and articulated this.

I'd be interested to know the outcomes from any other small firms that do some exercise like this.

Runway

Cogent is very proud to announce the release of our very first product - Runway!

Runway is an action management system. It's inspired by Getting Things Done (GTD), and will eventually grow into a more complete GTD system. Why do you need an action management system? Because your brain is a terrible secretary?Right now you're wasting precious mental energy worrying about things you need to remember to do, which ones are important, which ones are urgent, what needs to be done next, and so on. Runway provides a trusted repository for all the things that you need to do - and trust me, there are hundreds of them.

Runway is designed to be quick and simple to use, and to become intuitive over time. An action is created through a simple, single-line entry field. No check boxes, no drop-down lists, no tabbing between fields. You can also create an action by sending an email to Runway, or by sending a Twitter message. We want to extend Runway to other sources, so that you have ubiquitous capture for your important ideas.

Actions appear in your to-do list right away, and can be prioritised easily with drag-and-drop. You can rapidly filter your actions by when they need to be done, the context you need to be able perform the action, the time required, the energy you need, and any other tags you want to use.




Most people will use Runway to track all their actions, select the actions for the day at the beginning of the day, and then work through that list. I've been using Runway that way for over six months.

I'm really excited by Runway, because it makes my life simpler and helps me get things done. It's also a great piece of software, but I hope that people forget about that, that Runway merges into the background and just becomes an invisible helper. Please visit http://www.runwayapp.com and give it a go. Perhaps you'll have the same experience as Jonathan, one of our beta testers, who says, "Runway has changed my life!"

Cogent Customer Brief

We're not heavily into formal contracts at Cogent, since we think they fail to capture the essence of the relationship. We understand that everyone needs to know how much they're paying and for how long, and we've also had some unfortunate experiences where we didn't do a good job of communicating expectations to the customer, so we've put together a document that describes, in the plainest language we could muster, what to expect then you collaborate on a project with us. You can find it on this page of our website.

Partnerships

My prior post (about excess work) generated a private email suggesting that partnerships would be one way to deal with the problem - that Cogent could enter into an arrangement with one of the larger companies to pass on excess work in exchange for some fee. That would seem to tick most of the boxes - the work gets done by someone we would have referred the customer to anyway, and we get some financial return, so I should feel less like we're "giving something away".

Yet I have some visceral reservations that I need to try to rationalise. I think it's mostly about responsibility and care - if Cogent is going to take money for something then I think we need to stand behind the result, and my experience is that you can lose a lot of money (and reputation) from bailing out one bad gig. If all I say is "here are some folk that have historically done a good job, and we don't have any stake in whether you work with them or not" then I don't feel as much responsibility for what happens downstream.

I think that's the heart of it - thanks for listening/reading while I figured that out.

What we do with excess business

Cogent doesn't do too much in the way of overt marketing and advertising. We spend some money on conference sponsorship, and about $30 per month on Google Adsense, and I think that's about it. Despite that, we get a steady stream of enquiries, and we generally end up with more work than we can handle - particularly if it's urgent work as we're often committed a few months in advance.

If a customer wants to start something right away, we do very little (if anything) to discourage them - instead, we pass along the contact details of the best people we know who can do the work and leave it to the customer to make any further arrangements. This is the simplest approach that can work, since it means that we don't get caught in the middle either managing communications or guaranteeing the quality of the work, and in principle the customer doesn't end up paying two margins so they should get the work done at a reasonable price. We're usually passing the work to companies that are "bigger" than us, and It's also worth saying that Cogent gets referrals in the same kind of way, mostly from independent contractors.

So by all rights I shouldn't be unhappy, but I have to admit to being irked by handing my competitors leads, especially when we don't get a lot of love in return. I understand it logically - if we had available resources, we wouldn't pass work to a competitor either, and these companies are much more likely than us to have available resources - but it still feels wrong or unfair on some level.

I'd be interested to hear how other small companies deal with similar situations.

Failure is mandatory

A few days ago I had a conversation with Travis about failure. In some senses I'm a hard ass about failure - I think if you protect people from failure all the time you also protect them from learning anything. It's still important to protect people from potentially traumatic failure, but traumatic failure is the exception rather than the rule.

The context was how far our duty of care extends as consults, particularly in regard to the transfer of a solution to the customer. There's typically tension around this issue - the customer wants to maximise their investment in the solution itself, and in other projects as well, and that works against either documenting the solution or getting permanent staff involved. One position is that we should go out of our way to document/record knowledge of the solution for handover, even if the customer hasn't made that a priority. I think that's paternalistic (which is ironic given the example I'll use later). I think it's our responsibility to warn the customer that there's a handover risk, and to give some advice on how we'd handle that, but in the end it's up to the customer to decide how to invest their resources, and to wear the consequences of those decisions. I think it would be quite wrong of us to divert resources into documentation against the customers wishes. We might think the customer has made the wrong decision, but it's still their decision and they won't thank you for subverting it.

Another context that we talked about was when you see someone else about to do the "wrong thing". You should definitely draw attention to the problem as you see it, but if the person goes ahead anyway, should you do anything to protect the project from the outcome? My position is that you should not - you should let the project suffer the consequences (again excepting traumatic outcomes). It's possible that your fears won't be realised, in which case you'll learn something. If they are realised, the project team will learn something (including perhaps to pay more attention to your advice) and may be able to avoid similar problems in future. If you just fix it, then what the project team will learn is that there are no negative outcomes from that behaviour, so they'll quite happily repeat, and you'll have the same personal situation later. Ad infinitum.

The same kinds of things apply to my son. I try to protect him from traumatic outcomes, but sometimes I need to let him get hurt (physically or emotionally) so that he learns not to repeat a behaviour.

Cogent is talking about open rates

Some of you will already know this, but Cogent is an ongoing social experiment along the lines of Thoughtworks, but with different axioms. Transparency is one of our axioms, but historically we've applied this internally and the public information about Cogent has been much the same as what you'd know publicly about any other consulting company. We're about to step that up a notch and publish our rates schedule publicly.

As usual, we see pros and cons to this decision. First thing is that no one else seems to do it, so perhaps we're just being stupidly naive and we'll discover the reasons why no one else does it in some catastrophic way.

More obvious to us that we lose one of our supply-and-demand levers. Our supply (our employees) is fixed in the short term, and it's always been hard for us to figure out what the "real" demand for our services is (we get plenty of unsolicited contacts that don't turn into any business). One of the levers we can use to affect demand is price - if we had lots of people doing non-billable work then we'd clearly think about dropping prices to get people onto billable work. Historically this would have been transparent to customers, unless they talked to one another, but with public pricing it will be clear if we offer discounts.

We may also find that we get less contacts because people look at our prices and decide that we're too expensive without even talking to us. It's not clear if that's good or bad - we may actually save time because we don't waste time doing estimates for work that's never going to proceed anyway. Or we may miss out on legitimate business.

And finally, our permanent colleagues on consulting gigs will now have a pretty good idea of what the consultant sitting next to them is being charged out at. That may or may not prove embarrassing (hoprefully not).

So why do it? We have some feedback that clients like simple, public rates schedules. It means less discussion internally about "how much as we going to charge this client?". And it means extending the reach of our core principles, and each time we've done that in the past it's been good - if nothing else it tells the employees that we "walk the walk" consistently.

We've split the rates along two dimensions - the type of work and where the work happens. We do two types of work - agile coaching from senior staff, which involves some hands-on development but is focussed on influencing the entire team rather than producing features directly, and development work, which is mostly focussed on delivering features but also on influencing the entire team to work more effectively. We have different rates for agile coaching and development work, and we also charge less for development work that's performed at our own premises. There are three reasons for charging less for working on our own premises - i) we aren't able to influence the client staff as well, so we deliver less value; ii) we prefer working on our own premises and price our services to encourage this; iii) when we're working on our own premises we're effectively competing against teams all over the world and we need to reduce prices accordingly.

So here are our rates for new work in 2009, all exclusive of GST:

Agile Coaching

• Senior Agile Coach: $1400


• Iteration Manager: $1300

Development (consulting / offsite)

• Tech Lead/Architect: $1200 / $1000


• Senior Developer: $1100 / $ 900


• Developer: $1000 / $ 800


We're interested in what people think of this approach to pricing, so please let us know.

Post-agile, or just agile done well?

Ivar Jacobsen was in Melbourne recently, and did a presentation in conjunction with the ACS saying that development needed to "get smarter". Throughout the presentation he compared Smart development to Agile - the trouble was I couldn't see any difference.

For a few years now I've heard of "post-agilism", which was needed to overcome the woes of agile development - the trouble is I couldn't see any suggestions that weren't compatible with agile as it was described around the time of the Agile Manifesto.

Don't get me wrong - I don't think Agile is the answer to every software development problem. I strongly agree with Alistair Cockburn that the development method you use needs to depend *at least* on the degree of risk and the size of the team. However, I don't think that we need to find new methods just because Agile can be done badly. Maybe we should pause and work a little harder at doing things well before we move on to doing new things.

Experience - who gets the benefit of the doubt?

Bob Martin seems to have created a tempest with his apprenticeship model proposal, but it's worth remembering that it's not a new idea. There was at least one OOPSLA workshop on the Software Studio concept last century, and Ken Auer has implemented a related model at Role Model Software. I think Bob's raising the idea because he's had enough of crappy development and he wants to provoke some thoughtful responses (though that's just my impression). His article also triggered some related thoughts on how experience is treated in our industry.

One response to Bob's suggestion was that hierarchy somehow mistreated or devalued the more junior staff. It's quite true that this is one possible outcome, but the converse is also true. In non-hierarchical teams it's possible for the senior staff to be devalued! You can bring this into focus by thinking about a team of 2, one junior and one senior, and then asking yourself who should get the benefit of the doubt in a deadlock.

My experience is that many "juniors" think that they should get the benefit of the doubt. I've seen this attitude start immediately after graduation. I don't think that's right, or fair (but of course I'm arguing from an old guy's perspective). I've seen someone get upset because they didn't have carte blanche to replace existing frameworks, even though the changes would affect 30 people. I've seen someone simply defy the team conventions and go ahead and do it their preferred way, even after a team-wide discussion of the alternatives.

Everyone needs to understand that part of being a software developer is learning to *convince* people that your way is right - if other people don't understand, then you need to take ownership of the failure and find better ways to convince people. There is always going to be an "incumbent" position - sometimes incumbency is just the way it's done, and sometime incumbency is the older, more experienced person who perhaps can't articulate their position either. Someone needs to get the benefit of the doubt, and my leaning is towards experience.

I can't comment directly on the quality of very recent graduates, because I've stopped advising companies to hire them. My gut feel is that graduates come out with a good foundation for a career in software development, but still have a lot to learn. Sadly (for them) the salary for a graduate seems to be about $50K, and I can get people with 10-15 years of experience for just over $100K, and I'll take the more experienced person every time. I don't believe (as some people have implied) that mastery starts at 4 years and everything past that is wasted.

The cloud...it's full of *people*!

This is the story of the development of a small piece of software to deploy to the cloud, developed by a cloud of people, and how nominally unrelated groups interacted along the way.

I'm interested in having a way to easily create new Rails servers in the cloud (specifically AWS) and to deploy my apps to them. I'm familiar with the bits and pieces that are involved but I hadn't stitched them together. I started to write a Rails app to manage my AWS assets, partially as a path to learning extjs, but when the AWS Console came out I ditched that and started back on the original problem. Here are the different bits that I planned to use, and how I became aware of them:

• Sprinkle, by Marcus Crafter (http://github.com/crafterm/sprinkle/tree/master) - Sprinkle handles deployment of resources on Ubuntu boxes in a pretty easy-to-understand way. Unfortunately I have a strong preference for Postgres over MySql (blame Simon Harris) and the default Sprinkle deployment recipes all use MySql. Still, it didn't take me long to make some extensions to install Postgres instead of MySql.



• Passenger-Stack, by Ben Schwarz (http://github.com/benschwarz/passenger-stack/tree/master) - Sprinkles for Apache, Passenger, Mysql, Memcached & Git. I found out about this via a Ryan Allen tweet.



• Amazon Web Services (http://aws.amazon.com/) - well known, but the fairly recent addition of Elastic Block Storage (EBS) gives a way to easily create persistent data that outlives a given instance, say for a Postgres database.

The last piece to this puzzle is a web site called oDesk (http://www.odesk.com/). I was talking to a client last week and she mentioned that she thought Australian Rails developers were the most expensive in the world. When I asked her what she was doing she said that she was using off-shore Rails developers and was finding good people who would work for US$20/hr, from India but also from the USA. It seemed like the recession in the USA was having interesting side affects. I was definitely curious how using oDesk would work from a client perspective.

So I had this small problem that I didn't have time to work on, and my curiousity about oDesk, and it seemed natural to solve them together. I posted my job on oDesk - produce a script that, given an EC2 instance and an ESB volume, would automatically configure the instance to use Rails with Passenger and Postgres, storing the data on the EBS volume. In addition, provide a test application so I can verify the Capistrano deployment to the instance. I posted this as a fixed price job, and my plan was to use a few different people and compare the outcomes. I posted my job last Sunday (it's currently Thursday).

I had my first response in a few hours, from within Australia! Twenty four hours didn't bring any more responses, so I pushed my job in front of a number of Rails developers who had high ratings. Many declined as they were too busy, some thought the suggested price was too low given the need to learn about AWS, and some still haven't responded. I got two more positive responses, one from an experienced developer in St. Petersburg and one from a less experienced developer on the east coast of the USA. The bids were $333.33 (Australia), $250.00 (Russia), and $166.67 (USA). The expected development times were 1 day, 1 day and 1 week.

I accepted the responses and nominated a starting date of Tuesday. On Wednesday I got my solutions from Australia and St Petersburg. I've tested the Russian solution and it works fine. There seem to be couple of issues in the Australian solution but the developer
is being very responsive, and the USA solution isn't due yet.

What's more interesting is what else happened in the 24 hours of development. The Russian developer forked passenger-stack on GitHub and hosted the fork as a public repository on GitHub (which was fine by me). However, I found out about this through an email from Marcus Crafter, who knew I was looking for a Postgres solution and pointed the fork out to me! Then I saw these tweets from Ben Schwarz (@benschwarz) "@kouky Just so you know, someone forked passenger-stack overnight rolling postgres support in, I'll try and pull everything together :)" and "@atnan @kouky, I've just blindly merged the PostgreSQL stuff as I'm up in the hills. I'll check it all out tomorrow :)" The status message for passenger-stack on GitHub now says "Adding support for PostgreSQL. Sprinkle will prompt the user to select MySQL/PostgreSQL server and the relevant Ruby database drivers."

So it looks like my little experiment ended up sponsoring the change that I wanted in the core implementation of passenger-stack. I suppose I could have asked Ben to do it originally, but that seemed like an imposition and I would have only achieved one of my goals. Now I have a script to configure my EC2 instances automatically, a better feeling for oDesk and off-shore development, and I've discovered that you can always sponsor the features that you want in a piece of open source software. Well worth my costs :-)

An excellent idea to track drag...

Kevin E. Schlabach posted about the Snake on the Wall - a great idea for tracking the things that are slowing you down, and giving you a good starting point for your retrospectives.

McDonalds doesn't teach queueing theory

I was in McDonalds today (don't ask), and there was a single line in front of the one attended register. One of the staff opened a second register, but everyone happily stayed in one line. Here's the dialogue that followed:

McDonalds : "Please form a second line here"

Steve : "Why?"

McDonalds : "What?"

Steve : "Why should we form two queues, when one queue is more efficient?"

McDonalds : after 3 or 4 second pause, "Because I say to"

Can't argue with logic like that....

Batch Conversion of Pages file *in Leopard*

I was very proud of my Pages to rtf script, and then I upgraded to Leopard last night and it stopped working! In my search for the answer I came across this post, which solves the problem more thoroughly, though you need to combine it with information from here as well.

In case you don't want to do this yourself, here's a script that will start Pages, prompt you for the type of export you want, and then prompt you for a destination directory, and it works under Leopard. I told you it was more thorough than my last solution :-)

[sourcecode language='jscript']
on open theFiles
tell application "System Events"
if process "Pages" exists then
display dialog "whoops, please close Pages before running droplet!"
end if
end tell
tell application "Pages"
activate
delay 1
close front document
set theList to {"doc", "rtf", "pdf", "txt"}
set theType to (choose from list theList OK button name "Select" with title "Pages Export" with prompt "Choose one or more formats to export using Pages" with multiple selections allowed) as text item
set s to theType as string
set theLocation to choose folder
set theLocation to theLocation as string

repeat with aFile in theFiles
open aFile

if theType contains "doc" then
set asType to "SLDocumentTypeMSWord"
set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to Desktop.
set docPathAndName to theLocation & docNameNew
save front document as asType in docPathAndName

end if
if theType contains "rtf" then
set asType to "SLDocumentTypeRichText"

set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to Desktop.
set docPathAndName to theLocation & docNameNew
save front document as asType in docPathAndName
end if
if theType contains "pdf" then
set asType to "SLDocumentTypePDF"
set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to Desktop.
set docPathAndName to theLocation & docNameNew
set s to save front document as asType in docPathAndName

end if
if theType contains "txt" then
set asType to "SLDocumentTypePlainText"

set docName to name of front document

-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"

-- Add .doc extension.
set docNameNew to first text item of docName & asType
set AppleScript's text item delimiters to prevTIDs

-- Save file to User Specified Location
set docPathAndName to theLocation & docNameNew
save front document as asType in docPathAndName
end if
try
close front document saving no
end try
end repeat
quit
end tell
end open
[/sourcecode]

Murray Gell-mann, in a TED video....

"In 1957 some of us put forward a partially complete theory of the weak force, in disagreement with the results of seven experiments. It was beautiful and we dared to publish it, believing that all those experiments must be wrong.

In fact, they were all wrong."

I love that in maths and physics truth and beauty are so frequently aligned, and that misalignment often indicates immature understanding. A new insight can reveal new beauty.

I think the same is true of programming. Your code should be at least as elegant as your problem domain (I can't do anything about accounting *s*).

Batch conversion of Pages files

My current assignment makes me a Mac weenie in a Microsoft world, so while I'm happily tootling along in Pages I need to be able to share stuff with my colleagues. At the moment I'm the primary author of 30+ short course descriptions, stored in separate Pages files, and I've been exporting them to RTF when I need to share them. That was initially ok, but got out of hand as the number of files grew, so yesterday I was introduced to AppleScript.

I know nothing about AppleScript, but I was able to find a script on the web (thanks again, Google) that did most of what I want. I'll show you the script, then tell you the parts that weren't obvious to me and slowed me down

[sourcecode language='jscript']
on open theFiles
tell application "Pages"
repeat with aFile in theFiles
open aFile
set docName to name of front document
-- Remove .pages extension.
set prevTIDs to AppleScript's text item delimiters
set AppleScript's text item delimiters to ".pages"
-- Add .pdf extension.
set docName to first text item of docName & ".rtf"
set AppleScript's text item delimiters to prevTIDs
-- Save file to Desktop.
set docPathAndName to (path to desktop as string) & "rtfs:" & docName
save front document as ".rtf" in docPathAndName
close front document
end repeat
end tell
end open
[/sourcecode]

My first challenge was that I couldn't figure out how to get this to run! Pressing "Run" in ScriptEditor did nothing. I ended up saving it as an applet (use "Save As", selecting FileFormat => application), and then I could drag and drop the files I wanted to convert onto the applet. Once I put the applet in the dock, that was pretty convenient.

The second thing that slowed me down was figuring out how to save the results to a folder (the original script saved to the desktop). What I needed was the bit ' & "rtfs:"' while setting the docPathAndName. Here I'm dealing with a string, and it already has a trailing ":" - I kept trying to add another one, and AppleScript gave me a message about the target not being writable, rather than not existing, which lead me astray for a while.

Anyway, now I can select my Pages files, drag and drop them onto the applet in my dock, and the rtf files appear in a folder on my desktop. Sweet!

Releases with themes

When I'm doing agile training, I commonly advise customers to give names to releases and iterations to help them decide which stories belong in which iteration, but it's not something that I see many customers do and since I rarely take that role not something I do much myself either. But today I got some direct experience!

Cogent is building a software application to help you organize your personal work, and I'm acting in the sponsor role. The details of the functionality are being sorted out by someone else, but I'm the person focussed on the operational requirements and how we get it to the billable stage as quickly as possible (everyone else in interested in this too, but it's my particular role). Yesterday afternoon I added operational stories to our list, and then prioritised the list story by story, and got what looked like a reasonable response.

Today I went back and created some themed releases, and gave them names.

1. 
   Homely - Ugly, but interesting. This release is usable, but won't win any design awards. We want it first because we're using the application internally now, so functionality is important. This is the release we'll pass to "friendly" testers for feedback, so it needs to support a small set of concurrent users without getting too slow. Codename : Homer


2. 
   Prettified - Like Homer, but certainly nicer looking and generally more usable. We still won't be confident in our backup plan, nor have stress tested the application, but it will look professional and be generally usable. Codename : Marge
   


3. 
   Robust - we need this one to be tough, so it can withstand the abuse of a lot of people. This is the point where we can do an open beta - still not charging, but giving the general public access. Codename : Bart
   


4. 
   Billable - something that we feel we could reasonably charge money for. Codename : Lisa
   

With these releases in mind I went back and looked at my priorities again, and found that some of them were quite different. If you are a customer on an agile project, I strongly encourage you to do the same thing - you may be surprised by the difference it makes.

Another "I wish I had said that"

“Simplicity and elegance are unpopular because they require hard work and discipline to achieve and education to be appreciated.” - Edger W Dijkstra

Adventures in BackgroundRb

We've been asked to produce some "nice" reports in PDF for a client, and getting it all done has been quite an adventure. We use Ruport for the reporting, which unfortunately means we also need to learn quite a bit about PDFWriter to get everything looking spiffy, and the report rendering does take a while, but we had it all working in a controller yesterday and things were looking good. Then we deployed the application to EngineYard.

The first thing that happened was that we got timeout errors that looked like this (courtesy of ExceptionNotifier):

A Mongrel::TimeoutError occurred in reports#cost_detail_pdf:
Mongrel timed out this thread: shutdown
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:221:in `within'

The support folks at EngineYard suggested that we shouldn't be running PDF generation from within our controller, and that we should use something like background job, but since we wanted to report status as the job proceeded and we only had a small number of jobs, we went with BackgroundRb instead. (If you care about the difference, look for 'Backgroundrb and Bj serve different purposes' in this thread).

BackgroundRb works great once you get it going, but there were examples in the documentation that simply didn't work for me. Here's what worked for me in the end:

In my controller:

[sourcecode language='ruby']
MiddleMan.new_worker(:worker => :cost_detail_pdf_worker, :job_key => current_user.id)
MiddleMan.ask_work(:worker => :cost_detail_pdf_worker, :job_key => current_user.id,
:worker_method => :build_report,
:data => [session[:production_id], file_name])
[/sourcecode]

and my worker:

[sourcecode language='ruby']
class CostDetailPdfWorker calculated_report, :template => :default)
register_status("Finished Rendering")
File.open(file_path, "w") do |file|
file << pdf
end
register_status("File available")
end

end
[/sourcecode]

The next part of the adventure was getting this running on our Slicehost slice. Unfortunately, BackgroundRb requires Ruby 1.8.5, and we only had 1.8.4 (on Ubuntu 6.06 LTS). Apt-get for Ubuntu 6.06 doesn't include Ruby 1.8.5, so I needed to install Ruby 1.8.5 from source using these excellent instructions.

Unfortunately, I also needed to reinstall all our gems to go with the new version of Ruby. This was mostly stragithforward, except for Postgres. The postgres gem wouldn't build the native extensions, failing to find pg_config. These instructions were close, but before that would work I also needed to do "apt-get install libpq-dev" (you might also need libpgsql-ruby and/or libpgsql-ruby11.8, but I already had those installed).

Ok, looks pretty good now. I started the backgroundrb server manually as a test, and could produce my pdf files, so the next step was to move to our production environment at Engineyard. This was the first time I tried to stop and start the BackgroundRb using Capistrano - unfortunately you can't simply repeat the command lines in your Capistrano tasks. Fortunately, there's also a description of some tasks that do work (make sure you use the start task in the comments at the end of post).

All in all, a nice outcome but with much more pain than I would have liked.

Design in TDD

There's been a recent discussion in the Melbourne XP Enthusiast Group (MXPEG) mail list about whether or not TDD 'works', with part of the thread also being about Story Driven Development (SDD). Some of the folks have reported a lot of success using SDD and its associated tests (that I would call acceptance and integration tests), to the point where they don't use unit tests at all. The impression I get is that with this approach they're working predominantly top down. I'm glad this works for some people, but I don't think it would work for me, and I wanted to take a few minutes to say why.

I think it's important to have both unit tests (developer tests) and acceptance tests (customer tests), whether they're implemented in the same tools by the same people or quite different tools by quite different people. Notice though that I said "implemented" - to me the two groups of tests have different owners, different perspectives, and serve different purposes.

Customer tests - they tell us if the application is sick or not, but they don't provide any indication of where or why.

Developer tests - they tell us exactly where something is going wrong, but don't necessarily tell us what the impact on the overall application would be.

I would like to have both of these, not one or the other. I also find that I personally don't design entirely either top down or bottom up. I do some top down stuff to get an idea of the problem, then I use that to hypothesise about a design that will meet the requirements. Once I have some confidence that my design looks reasonable, I implement parts of it bottom up, and I want unit tests to confirm that my implementation of each component works the way I expected. I keep switching back and forth between top down and bottom up as I go.

The thing I do badly is that I don't write acceptance tests until the end (if at all). I know this isn't what I'm supposed to do, I know it probably detracts from my overall quality, but it is what I find myself doing. So far I haven't found a tool that makes writing those tests easy enough for me (personal opinion only).

I wish I had said this...

... but Bob Martin did :

One of the developers asked the question point blank: “What do you do when your managers tell you to make a mess?” I responded: “You don’t take it. Behave like a doctor who’s hospital administrator has just told him that hand-washing is too expensive, and he should stop doing it.”