Costs and Benefits of Unit Testing
Unit testing is someting that brings most value for well-modularized,
reusable (and most imporantly, actually reused!) components, such
as libraries. Say, xml or json parsers or data binders. :-)
And less
(IMO) for higher-level non-reusable things such as business applications
and their user interfaces.
Because of this, I have grown to both do much extensive unit testing with Woodstox and Jackson over time, and like the results. In fact, unit testing is one of 2 primary reasons for implementation quality of these packages (the other is having active user base that finds and reports problems). So whereas initially I mainly just added unit tests to verify bug fixes (and guard against regression) -- and this I still rigorously do -- nowadays I verify new features as well. For example, Woodstox 4.0 Typed Access API implementation has extensive unit test suite
1. Benefits
This level of unit testing has produced many benefits: consider for
example the fact that so far no bugs have been reported by users against
the Typed Access API implementation (to be fair, its usage has probably
been limited so far as well). All bugs reported against Woodstox 4.0 so
far have occured in non-tested (like Maven deployment) or lightly-tested
(W3C Schema validation) areas.
And with Jackson, most issues reported
concern missing features, or non-intuitive bevahior ("broken as
designed"): very few are related to actual implementation flaws.
2. Where's your beef?
So if life is so good, what's with the pickle face? Just this: unit tests don't come free. In fact, for the most part I now seem to spend more time writing unit tests than writing code. And especially so for bug fixes. While this is not based on strict measurements (it would be useful to quantify it of course; but yet another time drain), I have checked out time usage occasionally during development -- when you only have an hour to use on a project on a day, it is good to know where time goes. In general amount of effort required for unit testing is not really linear to amount of code written, but rather amount of code for the feature tested. That explains why bug fixes (which often touch a rather small portion of code) are more test-writing intensive tasks, in cases where no unit tests covered the functionality. I guess you can think of it as paying for test code after-the-fact, when needed (i.e. when a user reports an issue).
3. Is it worth it?
So far I am still ok with the costs, even if speed of progress is probably cut in half (compared to the "devil-may-care" approach). Although I could just take a chance with bug fixes (and in most cases that would actually work ok, based on past experience) and save my own time, that would come at cost for users who would essentially beta-test the product and report (if not completely fed up) bugs. This initial lower quality would be detrimental to perception of project, probably reducing or slowing adoption and indirectly then reducing actual testing.
But another important aspect is that on longer term unit test suite becomes indispensable tool supporting refactoring efforts: you can actually make drastic internal spring cleaning to the codebase, with reasonable confidence that if your changes actually do break something, you will know it via unit test failures. And this is probably the biggest overall benefit for long-lived projects: it helps code stay alive, as no part of code becomes "untouchable"
4. How about "bread-n-butter" code?
One more interesting thing to consider is this: while I maintain that usually unit testing is of less value for your typical day-in day-out corporate code -- at least on sort term -- perhaps it may still valuable for long term. That is, if (and only if!) it is expected that the system in question should become a long-living thing, a strong test suite should be built over time: especially to avoid brittleness that comes with lack of automated testing. And conversely, shorter-lived things just do not warrant big investment for extensive test suite. Less unit testing will be enough, with perhaps added external (QA) testing.
The real trick is of course predicting which kind of a life-cycle a system in development will have. And that is actually a hard problem. :-)
