My last couple of posts on productivity variations among programmers and the Chief Programmer Team model gave rise to some discussion about hazards of measusring software productivity at the individual programmer level. Software engineering studies normally measure productivity in terms of time to complete a specific task, or sometimes in terms of lines of code per effort-hour, staff-month, or some other measure of effort. Regardless of how you choose to measure productivity, there will be issues.
Productivity in Lines of Code Per Staff Month
Software design is a non-determinisitic activity, and researchers have found 10x variations in the code volume that different designer/developers will generate in response to a particular problem specification. If productivity is measured as lines of code per staff month (or equivalent), that implicitly suggests that the programmer who writes 10 times the amount of code to solve a particular problem is more productive than the programmer who writes 1 times the amount of code. That clearly is not right. Some commenters on my previous blog entry asserted that great programmers always write less code. My observation is that there’s a correlation there, but I wouldn’t make that statement that strongly. I would say that great programmers always write clear code, and that often translates to less code. Sometimes the clearest, simplest, and most obvious design takes a little more code than a design that’s more "clever"--in those cases I think the great programmer will write more code to avoid an overly clever design solution. Regardless, the idea that productivity can be measured cleanly as "lines of code per staff month" is subject to problems either way.
The problem with measuring productivity in terms of lines of code per staff month is the old Dilbert joke about Wally coding himself a minivan. If you measure productivity in terms of volume of code generated, some people will optimize for that measure, i.e., they will find ways to write more lines of code, even if more lines of code aren’t needed. This isn’t really a problem with this specific way of measuring productivity. This really just speaks to the management chestnut that "what gets measured gets done," so you need to be careful what you measure.
Productivity in Function Points
Some of the problems of "lines of code per staff month" can be avoided by measuring program size in function points rather than lines of code. Function points are a "synthetic" measure of program size in which inputs, outputs, queries, and files are counted to determine program size. An inefficient design/coding style won’t generate more function points, so function points aren’t subject to the same issues as lines of code. They are however subject to more practical issues, namely that to get an accurate count of function points you need the services of a certified function point counter (which most organizations don’t have available), and the mapping between how function points are counted and individual work packages is rough enough that it becomes impractical to use them to ascertain the productivity of individual programmers.
What about Complexity?
Managers frequently mention this issue: "I always give my best programmer the most difficult/most complex sections of code to work on. His productivity on any measured basis might very well be low compared to programmers who get easier assignments, but my other programmers would take twice as long." Yep. That’s a legitimate issue too.
Is There Any Way to Measure Individual Productivity?
Difficulties like these have led many people to conclude that measuring individual productivity is so fraught with problems that no one should even try. I think it is possible to measure individual productivity meaningfully, as long as you keep several key factors in mind.
1. Don’t expect any single dimensional measure of productivity to give you a very good picture of individual productivity. Think about all the statistics that are collected in sports. We can’t even use a single measure to determine how good a hitter in baseball is. We consider batting average, home runs, runs batted in, on-base percentage, and other factors--and then we still argue about what the numbers mean. If we can’t measure the "good hitter" using a simple measure, why would we expect we could measure something as complex as individual productivity using a simple measure? What we need to do instead is use a combination of measures, which collectively will give us insights into individual productivities. (Measures could include on-time task completion percentage, manager evaluation on a scale of 1-10, peer evaluation on a scale of 1-10, lines of code per staff month, defects reported per line of code, defects, fixed per line of code, bad fix injection rate, etc.)
2. Don’t expect any measures--whether single measures of a combination of measures--to support fine-grain discriminations in productivity among individuals. A good guideline is that measures of individual productivity give you questions to ask but they don’t give you the answers. Using measures of performance for, say, individual performance reviews is both bad management and bad statistics.
3. Remember that trends are usually more important than single-point measures. Measures of individual productivity tend to be far less useful in comparing one individual to another than they are in seeing how one individual is progressing over time.
4. Ask why you need to measure individual productivity at all. In a research setting, researchers need to measure productivity to assess the relative effectiveness of different techniques, and their use of these measures is subject to far fewer problems than measuring individual productivity on real projects is. In a real project environment, what do you want to use the measure(s) for? Performance reviews? Not a good idea for the reasons mentioned above. Task assignments? Most managers I talk with say they *know* who their star contributors are without measuring, and I believe them. Estimation? No, the variations caused by different design approaches, different task difficulty, and related factors make that an ineffective way to build up project estimates.
On real projects it’s hard to find a use for individual productivity measures that is both useful and statistically valid. In my experience, aside from research settings the attempt to measure individual performance arises most often from a desire to do something with the measurements that isn’t statistically valid. So while I see the value of measuring individual performance in research settings, I think it’s difficult to find cases in which the effort is justified on real projects.
(Measuring team productivity and organizational productivity is a different matter -- I'll blog about that soon).