Need a Rails-savvy, Java tainted, agile bigot?

My involvement in a project has finished unexpectedly early, so I find myself available for a new gig without anything in the pipe - which is unusual for me. If you have anything that you think I could help with, especially short- or part-time, drop me a line (steve at cogentconsulting dot com dot au).

Rspec View specs and integrate_views

We've recently had the pleasure of having Craig Ambrose working on a project for us, and as you'd expect when you bring someone new into the team, we've been doing some storming. One of the topics that comes up repeatedly within development teams I'm familiar with, including ours, is how fanatically we should adhere to clear separation of unit and integration testing.

I'm fairly laissez faire about the issue. I'm quite happy to write tests for Rails models that actually invoke ActiveRecord and interact with the database. I'll also happily set up a network of associated objects in the database rather than mock out everything except the class under test, though I do have some ill-defined limit that makes me uncomfortable.

I'm more strict about controller tests - I want the controller itself to be wafer thin, and I'll generally only test that the controller sets the right instance variables (along with flash notices etc). The tests end up being fairly small, don't touch the view at all, and I think even the evangelical TDD folk would call them unit tests.

However, Craig rightly points out a couple of problems with this. I'm typically working with server side functionality (if I had to say what I was best out, I'd initially claim domain modelling), so I write crap view code and I rarely write tests for it (someone's just going to come along and rewrite it to be presentable anyway). It's a fairly pathetic defence of my laziness, but anecdotally it seems lots of people don't write view specs. So there's a whole area of my application that's not tested very well.

Craig also points out that one of the biggest sources of defects for him is that his controller doesn't set up what the view expects (or you can express if from the opposite perspective if you prefer) - the failure occurs during the interaction between the controller and the view. These sorts of failures are quite difficult to find with unit tests alone, and probably won't get picked up by distinct controller specs and view specs.

We could write separate integration specs, but RSpec gives a simple way to catch at least the most egregious of these problems. Unit testing purists might object, but the laissez faire'sts won't mind. Put "integrate_views" into your descriptions. When integrate_views is specified, RSpec renders the real view rather than mocking out the rendering, and if objects are missing or badly misconfigured you'll get a rendering exception.

I remembered the integrate_views option from earlier versions of RSpec - it's not particularly conspicuous in the current version's documentation, but it can be very useful.

Using the latest attributes in a Rails migration

Occasionally I need to make some change to a table via migrations, and then immediately use the attributes associated with the latest table inside the migration. Most of the time you can do this by defining the aspects of the class inside the migration - for example:

[sourcecode language='ruby']
class AddDepartmentToProducts < ActiveRecord::Migration

class Product < ActiveRecord::Base
belongs_to :department
end

def self.up
add_column :products, :department_id, :integer
# do something with product.department
end

def self.down
#...
end

end
[/sourcecode]

There's also another way that's not as neat, but gives you finer grained control, for example if you need to have different definitions of the class in the up and down migrations:

[sourcecode language='ruby']
class AddDepartmentToProducts < ActiveRecord::Migration

def self.up
add_column :products, :department_id, :integer
Object.class_eval <<-end_eval
class Apparel21::Product < ActiveRecord::Base
belongs_to :department
end
end_eval
# do something with product.department
end

def self.down
#...
end

end
[/sourcecode]

ActiveScaffold reverse associations to models in modules

I really enjoy using ActiveScaffold, and encourage everyone to use it to generate administration style interfaces. I came across an apparent defect today that I wanted to share.

I have a set of models that are contained in a module, and I'm using AS to do administration for them. Mostly fine. However there's a problem when I click a link on an element of a many relationship that's being displayed in a list. In this image,

ranges is the result of a has_many relationship, and when I click on the range 'Summer Selection', I get this:

and the following stacktrace:

[sourcecode language='ruby']
ActionView::TemplateError (undefined method `reflect_on_all_associations' for Range:Class) on line #19 of vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:
16: # determine what constraints we need
17: if column.through_association?
18: @constraints = {
19: association.source_reflection.reverse => {
20: association.through_reflection.reverse => parent_id
21: }
22: }

vendor/plugins/active_scaffold/lib/extensions/reverse_associations.rb:26:in `reverse_matches_for'
vendor/plugins/active_scaffold/lib/extensions/reverse_associations.rb:12:in `reverse'
vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:19:in `_run_erb_47vendor47plugins47active_scaffold47frontends47default47views47_nested46rhtml'
vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:11:in `each'
vendor/plugins/active_scaffold/frontends/default/views/_nested.rhtml:11:in `_run_erb_47vendor47plugins47active_scaffold47frontends47default47views47_nested46rhtml'
vendor/rails/actionpack/lib/action_view/base.rb:637:in `send'
vendor/rails/actionpack/lib/action_view/base.rb:637:in `compile_and_render_template'
vendor/rails/actionpack/lib/action_view/base.rb:365:in `render_template'
vendor/rails/actionpack/lib/action_view/base.rb:316:in `render_file'
vendor/rails/actionpack/lib/action_view/base.rb:331:in `render_without_active_scaffold'
[/sourcecode]

The problem is that class should be MyModule::Range, not Range.

AS is looking up the class using code on ActiveRecord::Reflection::AssociationReflection, from the file activescaffold/lib/extensions/reverse_associations.rb. The fix (apparently - I haven't tested this exhaustively yet) is to change the code that sends "class_name.constantize" to the association to send "klass" instead (you need to do this in two places). The completed code is attached below. I hope this helps someone else!

[sourcecode language='ruby']
module ActiveRecord
module Reflection
class AssociationReflection #:nodoc:
def reverse_for?(klass)
reverse_matches_for(klass).empty? ? false : true
end

attr_writer :reverse
def reverse
unless @reverse
# Following line changed for compatibility with associations on classes in modules
# reverse_matches = reverse_matches_for(self.class_name.constantize)
reverse_matches = reverse_matches_for(self.klass)
# grab first association, or make a wild guess
@reverse = reverse_matches.empty? ? self.active_record.to_s.pluralize.underscore : reverse_matches.first.name
end
@reverse
end

protected

def reverse_matches_for(klass)
reverse_matches = []

# stage 1 filter: collect associations that point back to this model and use the same primary_key_name
klass.reflect_on_all_associations.each do |assoc|
# skip over has_many :through associations
next if assoc.options[:through]

## Following line changed for compatibility with associations on classes in modules
# next unless assoc.options[:polymorphic] or assoc.class_name.constantize == self.active_record
next unless assoc.options[:polymorphic] or assoc.klass == self.active_record
case [assoc.macro, self.macro].find_all{|m| m == :has_and_belongs_to_many}.length
# if both are a habtm, then match them based on the join table
when 2
next unless assoc.options[:join_table] == self.options[:join_table]

# if only one is a habtm, they do not match
when 1
next

# otherwise, match them based on the primary_key_name
when 0
next unless assoc.primary_key_name.to_sym == self.primary_key_name.to_sym
end

reverse_matches < 1

reverse_matches
end

end
end
end
[/sourcecode]

Uses of OpenID

I'm a user of OpenID, but this video from Google is an excellent introduction to the concept, the uses, and the pros and cons.

Particularly interesting to me as a developer were the ideas of using OpenID as a 'lightweight' authentication mechanism for sites that want a low barrier for registration and access, and recognizing multiple openids so that a site can access id-specific features from each. Highly recommended!

Rspec 1.1.2 and Textmate

A quick warning to everyone - if you upgrade to RSpec 1.1.2 and you use Textmate, you'll probably need to update your textmate bundle as well.

cd ~/Library/Application\ Support/TextMate/Bundles/

svn co svn://rubyforge.org/var/svn/rspec/trunk/RSpec.tmbundle

Freezing gems with architectures

So I don't ever lose this reference again (hopefully), here's a reminder to myself (and maybe to you) that I want to freeze all my gems, and accommodate different architectures, so I should be using gems_with_architecture. See
usage, and browse the
repository.

Beanstalk Dashboard

Since I've failed to get the beanstalk dashboard to display in my comments, here's the "team leader" view that Mark was asking about in the last post:

Excellent support from Beanstalk

We've started evaluating Beanstalk for our subversion repositories, and I had a problem this morning trying to commit to my second repository (the detail is that I was getting the message "Can't create directory '/var/lib/subversion/beanstalk.storage/1911/webistrano/db/transactions/1-1.txn': Permission denied"). Apparently this is a recurring problem on some accounts - probably a teething problem with the service.

Of course I'd rather not have any problems at all, but when I got this message I logged into the Beanstalk Campfire conversation, and within 10 minutes (9 of which were me getting a drink while Chris worked) Chris Nagele has fixed my problem and I was up and running again. Great stuff!

Interesting behaviour for defined?

I've been working with Pete Yandell's Not-a-mock plugin, which I like a lot, and this morning a failing spec led me to some interesting behaviour for the Ruby defined? operator. Try these two things:

defined? arbitrary_attribute

defined? arbitrary_attribute=

With Ruby 1.8.5 on my Mac, the second one fails! From experimenting, it doesn't seem like you can use any setter. The workaround (which I haven't confirmed with Pete yet) is to use 'self.methods.include?("arbitrary_attribute-").

Don't be dogmatic about dogma

A colleague of mine recently passed around a reference to "The Way of Testivus" from Agitar, and as promised on the front page I did indeed find that it was filled with "good advice on developer and unit testing" [although personally I haven't found testing my developers to be so useful :-)]. But there was one page that I thought would be dangerous in the wrong hands, and it started like this....

Don’t get stuck on unit testing dogma

Dogma says:
“Do this.
Do only this.
Do it only this way.
And do it because I tell you.”

Dogma is infl exible.
Testing needs fl exibility.

Dogma kills creativity.
Testing needs creativity.

I've got a good grasp of what the author means, and I agree with the sentiment quite strongly. But I'm also confident that out there somewhere is someone new to automated testing who's saying "oh, I've seen how other people do testing, and how could they have gotten it so wrong? I'm going to do things completely differently - I'm going to be creative". The trouble with rejecting dogma is that "dogma" is often a great approach to 80% of the problem, and it often contains a great deal of wisdom that isn't obvious to a newcomer.

I'm all in favour of rejecting dogma, but only after you've understood why those ideas might have become dogma in the first place, and you're confident that you understand the reasons that particular dogma doesn't apply in your specific context. Otherwise you're just being dogmatic about rejecting dogma.

Installing Postgres on Leopard

Excellent article on installing Postgres from binaries on Leopard. Just make sure you have XCode installed first.

Installing Git on my Mac

Inspired by Dr Nic's post I wanted to install git on my Intel-based Mac. Why? There were two motivations - I frequently do work on the train or tram, and less often on planes, which is the reason Nic describes, but I also wanted to be able to commit mini-milestones on my local machine that I'm not really ready to commit to the "point of truth" on svn. This is the point where I reveal that I don't do all of my programming test first, but that's a longer thread. Suffice it to say that there are periods where I need to work to get the test coverage back over the threshold (currently set to 99.2%) and I don't want to commit to svn until the test coverage is back there.

I installed git-core from MacPorts, but when I ran git-svn I didn't have a "clone" command, which is what all the cool kids are talking about. I'm still not sure that I know precisely why that was the case, but in poking around on the web I found a few places with installation instructions, and eventually things were working, so I'll add what I did to the pool (my main motivation is letting you know the problem I was trying to fix though).

1. Followed the instructions from http://www.beyondthetype.com/2007/6/15/guide-to-installing-git-on-a-intel-based-mac to install the latest version of Git from source


2. Re-installed subversion using the package from http://downloads.open.collab.net/binaries.html. Why? Because the last step in the previous instructions (setting the PERL5LIB) didn't help, and I couldn't find the svn-perl libraries anywhere. The instructions on http://speirs.org/2007/07/22/getting-git-svn-working-on-the-mac/ were helpful. After the subversion installation I had a directory /usr/local/lib/svn-perl, which did the job.


3. Added export PERL5LIB='/usr/local/lib/svn-perl' to my .profile


4. Added /usr/local/git-1.5.2.1 to PATH in my ,profile (early on, to make sure git was found here first)


5. When I tried git-svn I got an error concerning Term::ReadKey when perl tried to prompt me for a password. I tried to install Term::ReadKey from CPAN interactively (sudo perl -MCPAN -e 'shell'
   , then 'install Term::ReadKey', but got "Can't exec "./make": No such file or directory at /System/Library/Perl/5.8.6/CPAN.pm line 4566.". The answer was to change directory to ~/.cpan/build/TermReadKey-2.30 and execute the following commands: sudo perl MakeFile.pl; sudo make; sudo make test; sudo make install. Now I have the Term::ReadKey module installed!


6. When I tried to 'gitify' after that, I got this message : "error: More than one value for the key svn-remote.svn.fetch: :refs/remotes/git-svn". This seems to have been caused by previous failed attempts - removing the xxx.git file and running 'gitify' again seemed to work.

For people like me who are less than clear about the result, here's what you get:

1. A copy of your project in '../project_name.git'. This is your git working directory.


2. Your repository is in .git in the git working directory.


3. Change into the git working directory and start doing your stuff


4. You'll probably find these commands useful:
   
   
   
   • git status : tells you what's uncommitted
   
   
   • git add : adds a new file to version control. "git add *" seems to do everything you'd want :-)
   
   
   • git checkout -f : reverts local changes
   
   
   • git commit -a : commits any modifications (but not new files) to the repository
   
   
   • git-svn rebase : merge svn changes into your git repository
   
   
   • git-svn dcommit : commits all your changes back to svn
   
   
   
   

And if I'd known it was going to be this hard when I started, I wouldn't have.

Active Scaffold

After quite a bit of writing and business related non-programming work, I've got my hands back onto Rails. I'm actually crossing a few boundaries by writing an application to help me manage the invoices and employee payments for Cogent (the accounting system does a great job from a purely financial perspective, but I need something that can reflect our internal policies and procedures, which still makes it sound much grander than it is). I've also made a personal commitment to try to understand more about the gems and plugins that I'm using, which brings me to Active Scaffold.

I've used ActiveScaffold in the past for the administrative part consumer facing applications, but this time around I decided to have a closer look at what I could do with it. It's still an exercise in progress, but I've been very impressed with what I've seen so far. Let's look at three simple things first - ordering columns, changing the format of a column value, and filtering the displayed rows.

You tell a controller us use Active Scaffold for CRUD actions by adding a line to your controller. For example, if I want to use ActiveScaffold to maintain the Sale model object then I could set up a controller like this:

active_scaffold :sale

However, the active_scaffold method also takes an optional block, which you can use to customise the behaviour of the scaffold. In particular, there is a method to set the columns in the list view, that you can use like this:

active_scaffold :sale do |config|
config.list.columns = [:invoice_number, :customer, :amount, :status, :salary_package, :entry]
end

In this case I've specified that there will be six columns in the list view, in precisely the order specified in the array, where the symbols in the array correspond to accessor methods on the Sale model object.

Changing the format of a column is done outside the scaffold configuration, in a helper method. The :amount column is some amount of money, so it would be nicer to format is as a currency amount. The formatter needs to be named {column_name}_column, and it receives the entire model object as its only parameter, so a method to format the amount column in the Admin::SalesScaffoldController would look like this:

module Admin::SalesScaffoldHelper
def amount_column(sale)
number_to_currency(sale.amount)
end
end

Finally, filtering the displayed rows is so simple that I stumbled across the functionality by accident. In my Sales model, the status attribute is an enumerated value that contains, among other values, 'Open' and 'Closed'. I set up a link to the controller like this

link_to :controller => 'admin/sales_scaffold', :action => 'index', :status => 'Open'

and expected to need to explicitly provide some filtering inside the controller. Instead, it worked before I'd done any coding at all! It turns out that any parameters that correspond to attributes are used to filter the results in the table. You can specify a single parameter, or you can specify multiple parameters, in which case they'll be and'd together.

I'm really impressed with the capabilities of Active Scaffold, and I intend to explore them in more depth for applications where the tabular presentation of Active Scaffold is a good fit, and for providing functionality quickly while a user interface design is refined.

Introduction to Rails Workshop

It's been quite a while since I blogged, mainly because I've had my head down writing new workshop material. We're running an introductory Ruby On Rails workshop on July 21 (just two weeks away now), and although it's been a lot of effort I'm pretty happy with the way the material is working out.

One day isn't very long to look at the equivalent of Java and a full Java web development stack, so rather than simply repeat what's already available in books and screencasts we're going to cover things that we find useful in our day to day development work. We'll be giving attendees a whirlwind tour of Ruby, a brief guide to Rails, and then we'll focus on tools and practices that aren't quite so accessible to the ruby newbie.

So if you've heard about Ruby and/or Rails but haven't made the plunge yet, or you know someone like that, go have a look at the course description, check out our pricing policies and send us a bid. At the moment all you need to bid is $100, so you may be getting some really cheap training!

New introduction to agile methods

It's been a while since I posted, and it's mainly because I've been busy updating our introduction to agile methods. It's done now, at least well enough to show people. Warning - this is a large (7mb) pdf file. It's not intended to stand alone - the real value comes from someone talking it through - but you may still find it interesting, and there may be parts you can use (with accreditation please).

From Noam Chomsky

Who always says it better than me...

"If you assume that there is no hope, you guarantee that there will be no hope. If you assume that there is an instinct for freedom, there are opportunities to change things, etc., there's a chance to contribute to the making of a better world. That's your choice."

A New Agile Manifesto?

Brian Marick makes some great points here about things that are important to agile development, but weren't the first order of business when our major concern was getting permission to do it. If the default is that agile is acceptable, Brian points out that it's essential to understand what teams need to execute agile processes successfully.

Brian's pulls are "Skill, Disciple, Ease and Joy". Parts of this resonate with something I've said to plenty of people - too many software developers consider pain to be part of their jobs. I've seen some terrible things done and excused with "it's always hard", or "it has to be that way". For change to occur, we first need to see the potential for something better ("it shouldn't hurt like this"), and then we need to be able to envision that better thing (which requires skill). And the result is frequently things that are easier to use for everyone.

The joy thing is going to be even harder though. Programmers seem to have an habit of finding someone else to blame for their pain, rather than looking closely at themselves. If you don't believe me, go read some Dilbert. Dilbert's problems are caused by Catbert, by PHB, sometimes by Wally, and he's generally the good natured, innocent bystander. Now these characters are intentionally caricatures, and I don't want to stretch the analogy too far, but I think they capture the software development zeitgeist. We'd do better if we stopped seeing ourselves as victims, and involved ourselves in finding solutions to the problems we see, especially the ones that impact us! If you think your boss is making stupid decisions, make it your business to understand their perspective on the decision - maybe you're not even looking at the same problems!

Anyway, Brian says things better than I can right now, so go read his blog.

Cogent, Year One

When I started Cogent I had a couple of aims, some of which were poorly articulated at the time. I wanted to create an environment that didn't have the employer/employee tension that I experienced in most workplaces; I wanted to be able to work with the people I wanted to on the jobs I wanted to, and be able to say no to the other stuff; and I wanted to create an environment that was attractive to other developers. I'm pleased to say that at lot of this has been accomplished (I can't say that I've accomplished it since lots of people have helped, particularly Marty Andrews), but I'm now also more aware of the limitations of the approach

It's been just over a year since I started consulting again. After doing some Rails and R&D work at IBS I was really lucky to be able to spend three months in Bangalore working with a Wall St bank. It not only gave me a chance to see the Indian Silicon Valley from the inside, so that it's no longer just an abstract concept, it also gave me a chance to catch up with people I hadn't seen since I left New York in 2001. Then I was able to take six weeks off, which was exactly the flexibility that I was looking for when I started consulting, and now I'm working at Sensis with a bunch of good people, including some people working under the Cogent umbrella.

Over that time Marty has gone from being someone simply billing under the Cogent umbrella to effectively being a partner in the business, which has made things both easier and much more interesting. We've got six people billing effectively full time consulting work through Cogent, and another two people billing part of their work that way. Everyone keeps the vast majority of their own revenue, leaving just a little bit for shared expenses. And I've been learning more about accounting and running a business, which has its good and bad points. That's all the good stuff.

What I've become aware of lately is that while the current business model is 'fair', in the sense that everyone keeps they earn, it has some drawbacks, the main one being that there's no financial incentive for collaboration. There's still room for altruism, and none of us are particularly selfish, but financially there's no reason for me to spend time helping someone else build up their skills, or increase the rate they can charge, especially if means spending less time on something that would help me directly. I can live with that situation, but I don't really feel good about it.

As a result, we've started (just started) thinking about different relationships people might have with Cogent, reflecting different levels of risk and independence. The model we have now is great for people with deep skills who are happy to independently find their own work and bear the associated risks. It's not so good for people who want more predictability, or who would like to work more collaboratively, or have someone who was helping them improve their skills. One possibility is a full employment model, though we'd still have our signature financial transparency and profit sharing of some sort. Cogent takes the risk of finding work and paying salaries, and gets a share of profits in return. The employee gets security and predictability, and gives up some of their revenue in return - it's a pretty traditional model at the core. But we'd expect people to eventually move away from this into a looser arrangement, where they bear more of the risk and get more of the revenue, and eventually into the associate arrangements that we have now. We'd want to have enough flexibility so that no one needed to completely abandon Cogent as their needs and contributions changed.

Early days yet. We'll have to see what happens.

Just when you least expect it...

I rarely post about personal things on this blog, but I'm going to make an exception. What follows is an email that my wife sent to me yesterday. Jamie is our three year old son - I hope that I can just retain a fragment of the magic that he holds lightly in his hands right now, and that he never understands that you can lose it.

---

Tonight while walking home, Jamie asked me where the moon was, and I said, "I don't know - maybe it's behind the clouds." He said, "When I get big and go into space, I will get the moon and bring it down for you. Okay, Mummy?"Awwwwwww.