10x Software Development - All Comments

re: Measuring Productivity of Individual Programmers

Solidpoint — Thu, 08 May 2008 09:16:03 GMT

I like the comments of Chris Mountford the best. The only measure of productivity is how much did it cost to deliver the functionality the user wanted - or should have wanted - as is too often the case. A real-life example illustrates this well. When I worked at Citicorp another R&D group spent several million dollars rewriting an ALCO related program to give it a GUI because the users were inputing data in a big, unstructured file and would make input errors like double decimal points, resulting in the failure of a 12 hour job. The GUI, predictably, was very tedious to use and output the very same flat file users had been creating in Notepad for years. Users hated the GUI. They wanted to copy a large file, modify a few things, and know it wouldn't blow up somewhere in the night due to data entry errors. I wrote a recursive routine to check common data entry errors, support embedded comments, and sent it to a friend who was doing damage control on the GUI project. My 3-4 hours coding replaced, and surpassed several million dollars worth of development as the users quickly abandoned using the GUI entirely after a stand-alone validation utility was made available. I could also point to a small CASE system I wrote and used to generate 12 billion lines of prototype code at FIB bank in the 80's...and on and on. Software is congealed intellect. If you don't know the domain you're working in don't be surprise if MOST of what you produce turns out to be redundant. Remember this when some guy is sitting staring at his screen for hours, scribbling on a white board, or is taking long walks with the other super-nerd in your group. It may well be that he's found a way to meet your goal 10,000 times faster. Are you smart enough, and do you know the domain well enough to know when he's bringing you a moon-rock? That is the challenge. Great software isn't a process of saving money, its a process of implementing inspired genius - in exactly the same way that the CFO can never save his company to greatness. Engineering can. Marketing can. The leasing department can. The bean counters can't. Their role is defensive. The best they can do is contribute at the margins by making the process of implementing genius cheaper - right up to the point where they kill it entirely.

re: Productivity Variations Among Software Developers and Teams: The Origin of "10x"

AmazingMo — Thu, 24 Apr 2008 03:16:36 GMT

Joel Spolsky makes a corollary to this point in his blog post "Hitting the High Notes". Joel focusses on the difference between programmers who can solve hard problems, and those who can't, but he also has some data that supports the 10:1 case along the way. If you've read down to this point, you probably will want to look for Joel's post too.

Cheers.

re: Measuring Productivity of Individual Programmers

Mal — Wed, 23 Apr 2008 21:52:32 GMT

Don't take this the wrong way Steve, bu can I have your baby?

Every time, I read one of your books or blogs I walk away incredibly impressed.

re: Measuring Productivity of Individual Programmers

Mitch Barnett — Mon, 21 Apr 2008 22:38:46 GMT

All good stuff, but here is a real world problem, I and others face all of the time. I work for a professional services company and we build large scale, custom ecommerce sites. Customers want us to estimate the effort and therefore cost and schedule to deliver the project on with nothing more than a features list.

As odd as it may sound, each ecommerce site is quite different in functionality (not necessarily content) and in most cases, we may build from the ground up, but it is built on top of an ecommerce framework. Still it may take 60K lines of custom code on top of the framework to implement a project.

So what are the best ways to estimate the effort? Tried FPA, but that requires skills beyond what we have. We have tried other levels of abstractions like use case estimating and the like, but not with a great deal of success. It seems like the only way is back to counting lines of code.

We can equate lines of code to features, which is how the ecommerce project is sold. For example, a PayPal payment feature may be 50 SLOC’s Are there any tools that can help us build a features library and then store historically both the lines of code and the effort it took to write those lines of code, so that over time we can just assemble the estimate from this library?

Would be very curious to hear other options or opinions… Many thanks in advance?

PS. I wish we are at a level that we could worry about programmer productivity, we are still at the stone age level of just trying to grasp the size and effort of our projects…

re: Measuring Productivity of Individual Programmers

Chris Mountford — Sun, 20 Apr 2008 12:08:15 GMT

Why do you hire a programmer? So that your LOC graphs go up?

A good productivity indicator correlates strongly with real business value.

If you cannot measure productivity - and for simplicity's sake I'll assume that degrades into an inability to measure value, then maybe there is no value there! What makes you think there is value? You know there is value and you know you should be able to measure it but why is it so hard?

Perhaps because you're trying to measure some abstract intermediate indicator of progress (such as lines of code). Failing to find a satisfying (reliable, consistent) abstract intermediate indicator of progress does not imply that developer productivity cannot be measured.

My preferred measure of programmer productivity is based on passing tests on shippable software. What kind of tests? All the kinds you need to show your software does what it is supposed to. Tests not enough for you? Where is the problem now?

re: Measuring Productivity of Individual Programmers

chrishmorris — Fri, 18 Apr 2008 19:22:44 GMT

A measure of program size that I like is the number of assertions in an adequate set of test cases.

re: Measuring Productivity of Individual Programmers

Gabriel Belingueres — Fri, 18 Apr 2008 14:54:06 GMT

I would measure productivity of an employee based on the common sense: There are no unproductive people in a project if there are no stressful situations regarding the people in the office.

So I would measure as how many stressful situations are present in the working office because of this employee's work/decisions/behavior.

This of course has sense if you can trace back the problem to the right time and person. For software developers you can see who committed a file in a source repository. Other tools may help but traceability is the king here.

In every project there are little failures and successes so this would be the rules for assigning a "productivity index" to person P:

a) If P's decisions/work (which are OUT of the scope of P) lead to a past/present/future non stressful situation for at least other person in the project, then he has +1 point.

b) If P's decisions/work (which are IN the scope of P) leads to a past/present/future stressful situation for at least other person in the project, then he has -1 point.

c) When the manager is in doubt, run an ANONYMOUS survey asking the team WHO's work/decision leaded to the current stressful situation, and WHY. (Include yourself in the survey.) If 80% of the answers conclude in the same guilty person AND reason then you assign a -1 to this person.

At the end of the period (month/year/etc) sum all the points of each employee and then you have it.

Gabriel

re: Measuring Productivity of Individual Programmers

software_engineer — Fri, 18 Apr 2008 08:35:59 GMT

good post ..

all of us know that measurement is a tricky thing and it influence productivity very much .. every article like that it is very usefull to read ..

thank you.

re: Productivity Variations Among Software Developers and Teams: The Origin of "10x"

Guillermo Schwarz — Tue, 15 Apr 2008 20:06:34 GMT

About the 10% of programmers who couldn't complete the assignment, a few years ago I interviewed 100+ Java developers.

I asked them to reverse a String.

Half of them couldn't complete the assignment and 80% of that half didn't even try.

The test has 24 questions. Only one developer replied 22 correctly but we couldn't afford him. The second best answered 18 correctly and was affordable.

The developer with the lowest ranking that was hired only had 5 correct answers of 24, and the project was a huge success.

Management was totally surprised by the high level of productivity (measured in use cases per month), by the quality of the product and the incredible team motivation.

All we did was to hire the best our money could buy, give them powerful machines, compile automatedly with ant, run unit tests with junit, and functional tests with selenium.

There were other tricks like doing prototypes and converting them into libraries, reducing code duplication, having written and updated code conventions, doing a code review before any check-in, having a team leader in each layer, etc., but having the best people was clearly our most important asset.

re: Productivity Variations Among Software Developers and Teams: The Origin of "10x"

riya — Tue, 15 Apr 2008 02:04:18 GMT

how do we calculate the productivity pf a programmer x who has done module 1 p lines in 2 days and module 2 k lines in 3 days

re: Measuring Productivity of Individual Programmers

Ben Northrop — Mon, 14 Apr 2008 21:09:48 GMT

Excellent post! I think an interesting angle to consider is how developing for maintainability and reuse can affect productivity over time. A developer who solves a problem rather myopically, not considering future requirements/changes, might look like a super-star initially in terms of productivity - he finished his feature on time, wrote a lot of function points, had no bugs, etc. Three months later, however, when a new round of requirements comes in, his productivity may plummet (or at least stagnate), because his prior work didn't make his future work any easier. A good developer knows when to sacrifice productivity in the short-run to realize gains in the long-run, however a manager who uses single, one-point-in-time metrics of productivity might actually discourage developers from writing clean/reusable code - which will of course hurt productivity in the long run.

re: Measuring Productivity of Individual Programmers

Tom Chizek — Mon, 14 Apr 2008 15:03:21 GMT

Great post! Not enough attention is paid to the downside of bad measurement. I have worked several places where great value was placed on cranking out code but little attention was paid to the quality of the code that was cranked. The result was predicable (now that I have lived through it) lots and lots of very buggy code. The turn around was when we started looking at the number of bugs reported along with amount of code produced. With that we (Management - the team already knew but could not get the problem addressed) could see who was just writing junk code and who was writing less code but also fewer bugs. It was really a case of people reacting to measurement.

re: Measuring Productivity of Individual Programmers

Charles Seybold — Mon, 14 Apr 2008 03:22:06 GMT

Great post. I strongly believe that measurement is tricky and can easily influence productivity in the wrong direction. The point Daniel makes is key: “measure variation between the estimate an individual makes and the actual time required”. I’d go one step father and make that idea the basis for almost every task and project: Try a continuous cycle of 1) ranged estimation, 2) execute, 3) look at the variance (together), and 4) repeat weekly until done. This will keep expectations aligned and uncertainty in clear focus. If you do this, I think you’ll be near the high end of productivity for any given team with a minimum amount of overhead. If we’re going to measure, we should reserve the effort for quality metrics not productivity metrics. If people are hitting their honest estimates with good quality, they are doing their best or at least as much as they get paid for.

As an aside, on our press tour for LP, we analysts always asked us if we’d use the estimation history to rate estimators. We told them the point was to help people learn rather than give managers a stick to beat people up. This is the danger of individual metrics, the temptation to misuse them is great. Metrics at the team level, project level, or category level – well that’s a completely different story; looking forward to the next post.

re: Measuring Productivity of Individual Programmers

Simon Parmenter — Sun, 13 Apr 2008 09:56:56 GMT

@Big Maybe: So the super coder does analysis and design where the mediocre coder does not? A better planner as well?

re: Measuring Productivity of Individual Programmers

Daniel Holmes — Fri, 11 Apr 2008 19:17:49 GMT

I think that Dilbert was that the boss was going to improve quality by paying a bonus for every bug the developers fixed. So Wally was going to write some bugs to be fixed.

I agree with you basic observations still.

re: Measuring Productivity of Individual Programmers

Steve McConnell — Thu, 10 Apr 2008 23:01:02 GMT

@Mark's comment about high "productivity" but low quality -- one truism is that work will move from unmeasured categories to measured categories. So if you're measuring quantity but not quality, you'll get quantity but not quality. That's the tricky part of measuring almost anything. You have to define a set of measures that collectively don't leave any room for unwanted behavior to creep in. Arriving at a set of measures that accomplishes that is often an iterative process, and reasonable guesses about what responses a measure will produce are often wrong. The result is that early iterations can miss the mark by a lot.

re: Measuring Productivity of Individual Programmers

Big Maybe — Thu, 10 Apr 2008 16:07:11 GMT

Actually, in my experience, LOC is a pretty good rough measure of productivity.

The mediocre programmer will need to patch together line after line of spaghetti to accomplish the same task that supercoder does in far fewer lines. That much is true.

However, the supercoder factors his code properly and will design tight class hierarchies to eliminate code duplication, raise cohesion and reduce coupling. He will put in exception handling at every point at which one can concievably occur. Then he will code up logging, monitoring, and alerting modules so that program bugs can be easily tracked and fixed.

The super coder checks his calendar and - hey - I've got a few more days, so let's refactor this table structure in a way that will enable useful reporting; let's attribute these structures properly so that they can be serialized to file; let's go ahead and internationalize all I/O; I can continue, but you get the idea. Now hundreds of more LOCs pour in, each one adding quality to the product.

Meantime, mediocre coder is still struggling to get his code to work and adding lines, commenting out other lines, until he can get it to work "on his machine".

re: Measuring Productivity of Individual Programmers

Paul Johnson — Thu, 10 Apr 2008 07:54:53 GMT

Thankyou Steve, and also thankyou Daniel Yokomizo for a great followup.

re: Measuring Productivity of Individual Programmers

Hamish — Thu, 10 Apr 2008 06:44:12 GMT

Another great post Steve, many thanks.

re: Measuring Productivity of Individual Programmers

Mark Roddy — Thu, 10 Apr 2008 06:37:43 GMT

One of the issues that generally disregarded (though not always) is that productivity is also a poor measure if desired level of quality is not taken into account.

A real world example. There was a developer who worked for our department that was known for being SUPER productive. He'd churn out massive volumes of code in time that dwarfed everyone else, but for some reason he was constantly being shifted to different teams. When he was shifted to my team I found out why. Though he did turn out code faster then anyone else I've ever worked with, the quality was so low that the savings of his high productivity was lost on the maintenance several times over.

If we had a position in which most of the work was one off and the code would be thrown out he would have been a godsend, but due to our desired level of quality based on the fact that we are producing applcations expected to live for many years his high productivty was actual resulted in losses in the long run.

I think that any discussion of productivity needs to take into account the level of quality of code produced versus a desired level of quality based on the needs of the project at hand. But then compared to productivity, quality is probably a much harder metric to measure.

re: Measuring Productivity of Individual Programmers

Daniel Yokomizo — Thu, 10 Apr 2008 01:25:30 GMT

A quick note on individual productivity measurement and function points. I was a certified function point specialist by IFPUG by three years and worked a lot with this method, so I know something about it. Anyway conceptually measuring productivity by function points is a wonderful idea, after all function points are an objective measure (when done right), but it faces a few problems that are quite unsolvable:

1. IFPUG's (there are other methods, like COSMIC-FFP) function points (FPs) have a complexity limit, after a certain threshold a function is considered to be complex, without a distinction in how complex it is and two "complex" functions have the same measure in FPs so you can't expect to handle N FPs worth of features to one developer, do the same to another and expect the productivity measure to matter. This is specific to IFPUG's FPs but it's the most widely used today AFAICS. This limit impairs greatly the correct measurement of complex systems, a reason why I started studying COSMIC FFP (which don't have this flaw).

2. FPs are a measure of complexity based on some objective criteria of a feature, but we don't have a comprehensive measure of complexity. Most methods ignore algorithmic complexity which can make a feature much harder to implement, test and understand. Eventually we may figure out a way to assess feature complexity objectively and without problems but today the methods fail at this.

3. FPs are external to software and define the complexity of the feature as a whole. For example if we have two features: create foo and change foo both will share code (e.g. foo's class, persistence layer, UI), so after doing one of them the other will be much easier to implement (compared to the effort of doing it first). As an application grows (hopefully) more code is shared, so the productivity "improves" (i.e. the measure becomes more skewed) for features that are similar to the ones already implemented. Code sharing, frameworks, libraries and tools all affect measurement of productivity of individual features to the point where correlation is insignificant. On the studies I made the data suggested no pattern on the measurement of individual features.

Now when we use FPs to measure teams productivity on bigger sets of features (e.g. 30+ use cases) the correlation becomes significant.

If we want to measure something on an individual basis I would suggest that we should measure variation between the estimate an individual makes and the actual time required, so we can train people to give better estimates and learn how to avoid a few biases in estimation and planning. Other than that there's too much noise.

I did individual productivity measurement (based on a variation of Mark II function points to avoid the complexity limit of IFPUG's FPs) in the study I mentioned in the previous blog post's comments, it was a small team doing four iterations of the entire development process (i.e. from analysis to deployment on each iteration) using an incremental approach (different features on each iteration). Each developer was responsible for a full module on each iteration, so we minimized effects of code sharing and the iterations decreased in business and technical complexity (i.e. most complex and risk at the initial iteration, easier ones later) so we minimized effects of algorithmic complexity within the iterations (roughly every developer got an module with equivalent complexity). The interesting result we got was that the difference in productivity of the programmers didn't vary between them, on each iteration they kept the same relative productivities, but on each iteration as the team got more knowledgeable of the domain, the application's infrastructure become richer and more robust, and the features become easier, their productivity increased by 2.16x (IIRC the values were 1x, 2x, 4.5x, 10.1x). So the entire team doubled productivity on each iteration, but they kept the relative productivities. In the end every programmer implemented the same amount of FPs (within 10% variation) on each iteration. On later projects we found out that the productivity values changed too much, but the relative productivities kept stable. As we had management approval we could divide the features and iterations almost perfectly and we could resist the urges to drop the constraints to keep an absurd schedule or introduce features in the middle of an iteration. In the end the study was considered useful to identify the strengths and weaknesses of the team, but failed to provide a way to make better estimates (the original goal of the study) and we realized the importance of good product and risk management (i.e. developing the riskier features first) and stability of features within iterations.

My conclusion from this study was that we can focus on individual programmers' differences but they don't predict much unless they are supported by a good set of development practices and a sane development process. Also a follow up study, tracking bugs to individual developers (using Subversion) of this project for a period of one year after final deployment, showed that incidence, severity and effort to correct bugs is unrelated to productivity for most programmers, but all of these are things that matter (even more than the productivity IME) in a project and would be important to measure. In the end I gave up on measuring as a career and focused on development practices, product management and tools (languages, libraries, IDEs and frameworks, in that order) to make productivity (and other measures) more predictable.

re: Chief Programmer Team Update

Steve McConnell — Tue, 08 Apr 2008 15:06:28 GMT

@Thomas - I'm on vacation and don't have the original articles with me -- my best recollection is that language lawyer was the term used in the original article. In any case, the *role* of someone who gets into the nitty gritty of the language so that the Chief Programmer doesn't need to was definitely a role on the original CPT.

Now that you point it out, I do find it interesting that the CP isn't a language lawyer himself. Most of the best programmers I've worked with have prided themselves on their detailed knowledge of language minutea. On the other hand, it's hard to know what exactly was needed to be a "language lawyer" in 1968 vs. today. With the internet and ISO standard languages, I would speculate that that information is more readily available today than it was 40 years ago.

re: Productivity Variations Among Software Developers and Teams: The Origin of "10x"

Lance Walton — Mon, 07 Apr 2008 19:43:02 GMT

@JohnRutter: Agreed that everybody put themselves above average.

It is of course possible for most people to be above average if the distribution is severely skewed, but it is not possible to skew it enough for everyone to be above average - at least one person would have to be vastly below average. Presumably that person's meta-cognitive skills would be so lacking that they'd be one of the most arrogant ignoramuses in the history of stupidology.

re: Chief Programmer Team Update

Paul W. Homer — Fri, 04 Apr 2008 16:03:50 GMT

I did five years as essentially the Chief Programmer for a enterprise-class software product in a tiny company. We were highly iterative, with very short implementations (1 to 3 months), and all the members on the team were specialized. I did most of the code (80%), I had a graphic designer, a couple of people building system templates, a writer (sometimes) and someone handling the configuration. Occasionally I'd add a second or third programmer, but the 'projects' were always very targeted. This arrangement worked extremely well.

The thing is, as much as I want to see myself as a "near-genius programmer who was an expert software methodologist, talented writer, exceptionally self-disciplined, and highly motivated", I probably am not that distinct. I code confidently, well and I don't crack under pressure. But it was not necessarily 'who' I was, as more of the way I was going about it.

I think that we should be able to get similar results with bigger teams, so long as the organization doesn't break down into committees. Granted you'll always need to center these groups around 'good programmers', but I don't think they are only dependent on 'great' ones.

Lest I hope not :-)

Paul.

theprogrammersparadox.blogspot.com

re: Chief Programmer Team Update

Thomas Guest — Fri, 04 Apr 2008 12:34:21 GMT

An good read, thanks. I'm interested to see a "language lawyer" on the team and wondered if this was a term used originally by IBM in the team you describe, or whether it's your own term for this role? In either case, it's interesting that the chief programmer doesn't necessarily know the programming language inside-out, and relies on having an expert assistant for this.