Monday, November 12, 2007

To Estimate or Not To Estimate

I have recently been spending a lot of time with two different "project management" applications: FogBugz and Team Foundation Server. FogBugz originally attracted my attention because I read Joel Spolsky's blog, Joel on Software, and he's the CEO of Fog Creek Software which makes FogBugz. TFS attracted my attention because it includes a new Source Control system which is a-w-a-y better than Visual Source Safe. I have been evaluating these products and trying to figure out which I'd rather use, its a harder task than it sounds.

The new version of FogBugz, 6.0, mainly consists of Evidence Based Scheduling. I have to be totally honest and say that EBS is not only very cool but makes total sense and I believe it would totally work. However it requires two things: 1) time tracking and 2) work item estimates.

Joel Spolsky recommends keeping your estimates under 16 hours, anything longer than that doesn't stand a chance of being accurate. That means you need a lot of very detailed tasks entered in the system (ex: develop function foo). Once you have estimates at this level you can start to get good predictions of when you'll ship AND you can start to pick and choose what tasks you want to do based on how much time you have left.

TFS doesn't really have any scheduling. It has Remaining Work and Completed Work fields (but only on the Task work item type, not on the Bug work item type...), but no reporting, no nice views, and no nice way to enter time (you have to subtract from the remaining work field manually when you add Completed Work...). The TFS developers seem unconvinced that estimating work at the level Joel recommends is at all worth while. Witness these articles on the Teams WIT Tools blog.

Personally I think estimating how long a task will take is a great idea. I think creating a list of tasks at that detailed level would not only help a developer think through what they're going to do before they dive into it but also help them get an idea of "where they're at" with all their work. I think it would make managing work loads a lot easier too. Not to mention the fact that the ship date estimates would make it possible for you to determine when your initial "blind" estimates were wrong or when scope creep is starting to get the best of you.

My question for all of you is, what do you do at your work? It doesn't even have to be programming work. Do you estimate work? At the detailed task level, or at a more abstract level? Is it helpful? And if you don't estimate work, do you wish you did?

UPDATE: I've posted a follow up post, Managing Projects


  1. We actually do estimating at a detailed level. We go to the level of the component or general task though. So if im working on a screen to say, take claim information id estimate something like this.

    Create UI - 1 day
    Develop service structure - 1.5 days
    Do crazy thing where i call the mainframe - .5 days


    We call them tasks on a story card. Easy card is a general statement like "Develop claim information page"

    We tend to keep some things internal to the tasks. So for example we dont estimate "write tests for crazy mainframe thing" tests are considered part of the estimate of their corresponding task but if they think they will take longer than usual you just up the estimate.

    Now that all said i shall say i hate this approach. I very much subscribe to the "agile" concept that says to not use time as an estimate at all. The typical agilian aproach (i just invented that word) is to estimate complexity not time. In theory (and in practice) you could have a very simple thing that takes a long time. Also in agile everything is relative. You pick numbers, typically 1, 2, 3, 5, 7, 11, 15, 25, and ? and you assign these "points" to the card based on how difficult one is in relation to another.

    You figure out project schedule by SWAGing it at first (which to be honest is how all projects start regardless) and then changing your plan about 4-6 weeks in based on how much work the team has gotten done up to that point.

    So say you have 1000 points of functionlity and you average 30 points an iteration (2 weeks). In this case your estimate is ~33 iterations or a bit over a years worth of work. As you work more you revise that so if gradually your team gets quicker and you average 50 points your project timeline goes down.

    Over a statistically signifigant number of iterations, and assuming no other signifigant project changes, your average points per iteration stabilizes to a mean and you can be pretty concrete in your projections.

    Now why do i prefer the later over the former. Simple. Neither is accurate. The agile approach accepts this and embraces it by using the inaccurate numbers in a statistically appropriate way. By using project data to drive the end of your project you can back up your assertions, where with the "old way" you back up your assertions by using old projects, typically unrelated, and saying "well project A worked like this and this is KIND of the same so well guess itll take about the same amount of time". Problem is project variables change. Different people, different client, different work, different architecture. Maybe everything is identical but for some reason the team just doesnt come together on this one (last time it was new and cool this time its old and boring). You cant estimate for these very well so embrace the inaccuracy and account for it by not using time.

  2. Cool. Where'd you learn that stuff? Do you know a good book on Agile?

    You should read that link on Evidence Based Scheduling because I'm sure you'll find it really interesting, but I'll quickly summarize.

    Joel Spolsky's point about making sure you're estimates are less than 16 hours is to compensate for two things: misjudging complexity and lack of accuracy.

    EBS basically works by taking the mean just like you described. It looks at how well you've estimated in the past and then runs simulations with that data to guess how accurate your current estimates might be. The result is a probability curve.

    Is saying "This feature is 7 hard" that much different than saying "This feature will take 8 hours"? At that abstract "feature" level you're guessing at the hours relative to the other tasks anyway.

    The problem is that you're at a high abstract level. Joel draws an analogy to engineers building a bridge. They've done it before and its the same thing. The details may change, but you can account for that. In software, if it had been done before you'd just burn a CD with the software on it.

    Since you've never done it before, you're going to misjudge the complexity. However, you have written simple methods to perform discrete tasks. So if you perform your estimates at that level you'll be more accurate. You'll still not get it right all the time, but overall it will be more consistent and the average will come out better.

    Now, that's not my argument, its his argument. It does make a ton of sense to me though. On the other hand, assigning complexity instead of time, like you describe, would clearly work. And it would be easier to be consistent doing it that way than by doing it by hours, which is really all that matters from a statistical standpoint. It may not be as accurate all the time, but how accurate do we actually need to be?

    And another question, how do you use those numbers? Do you use them to help prioritize work? Or is it just a mechanism to keep track of progress and project status?

  3. I learned it from a variety of sources, not the least being the conference i went to in Chicago. Bob Martin is a well known preacher of agile and he's big on relative complexity measurements. I don't know if he has many books but he comes from a company called Object Mentor. You can probably look him up.

    There is a book a read but i don't remember the name. Ill find out after lunch and comment.

    I like Joel's blog, and i think hes getting to the same conclusion the agile guys came too hes just sticking to the time estimate even if it is relative. I have an issue with this. I think years of using time estimating has psychologically set a precident that what you write down is precisley what it should be. From the get go you are using something incredably quantifiable and measurable and i can promise you unless your company culture rocks and your business customer is completley sold on the relative thing someone along the line WILL measure it and WILL ask why things are taking longer than the measure. When you state from the get go that this is "points" not "hours" and they have NO connection there is nothing concrete and real world to compare against. This is good and bad. We have always been taught measurable is good, and as programmers first and mathematicians second measurable is our forte. Problem is an software isnt engineering. Sorry it really isnt. If it was we would all be licenses and have some sort of fancy certification. A LOT of people compare software more to something like gardening. Think about it. How many bridge builder go back and redesign parts of the bridge after its complete or after a signifigant chunk is done? Now how about software engineers. We nurish and improve on our code while the project continues like a gardener does. I know its a weird analogy but in a way it works.

    I have a lot more to say but i have a presentation on the pragmatic programmer to give in 10 mins. Ill write more when i have the name of that book later.

  4. Great Point. One of the big points of EBS seems to be that it doesn't really matter if your estimates are accurate at not. The estimate that EBS produces will be good, but it doesn't matter if the developers estimates were good. But you're right that such a concept could be hard to get through to people in certain environments.

    Speaking of Pragmatic Programmer, they use that gardener example in that book. And don't be confused, Joel's bridge example isn't the typical programming = engineering statement. If anything, its the opposite.

    But again, the point of Joel's < 16 hour estimates is to make them NOT relative because you're not guessing as much. You've written a function to parse and insert data in to a database before, you know how long it will take.

    This is a good discussion.

  5. So my presentation flopped. Not a single person showed up neither on the phone or in person so im a bit pissed. Might as well continue this since its about the only intellectually stimulating thing $lt;/rant>

    Ok so continuing what i was saying...

    I should make it clear we do NOT use pure agile estimates here. We DO point, and we use those for planning but we also use time estimates (something i have been fighting for 2 years). Its inconsistant and allows management to have the iron fist of time (why didnt you get it done in a day if you said its a days worth of work) and the pseudo-flexibility of points. Basically its great for middle management because they can use points to justify missing a date to their bosses, but use time to push us why we arent finishing something.

    So ill answer your questions w/ respect to real agile. Ideally you use the numbers for both purposes. You use it to keep track of how the project is going and to prioritize work. In agile your business customer (or his proxy who has the authority to make decisions on his behalf) if intimatley involved in the process. They go to the meetings, they sit with the team and answer questions, they are ALWAYS there. This person might look at a whole bucket of pointed storycards (or collections of cards) and decide one feature that costs a lot of points doesnt have much priority so they push it till they have time later (if ever). Its true some of the core functionlity will be high points but it will also be high priority regardless.

    Now the next logical thing is time has to come in somewhere. You cant sell a bid on a project by telling the prospective customer you can do it in 300 points. Time comes in by looking at the number of points, comparing it to past projects of similar size and using those as a benchmark when planning. If you need to tell an initial estimate you be less precise (4-6 months) after 2 months you can probably narrow it to say 5 months. Your still in your range and in a way your customer will be happy because they probably anticipated the 6 months.

    The people who preach this stuff really do preach it. It is also used by a lot of big IT companies like google, microsoft (some teams not all), and its quickly becoming an industry standard.

    One other thing i wanted to mention about time estimates. If you know the history of software development from a project management perspective you know that time estimate and waterfall processes actually come from two entities. It was first developed by the department of defense as their "official policy" as to how to manage software projects. IBM picked it up and adapted it for the business world then started consulting on it (like they always do). Here is the irony. Almost every project the DoD has used their waterfall approach on has either failed, come in over budget, over time, or of poor quality. Its a tragically flawed system but it is so widely used no one wants to adapt.

    Now agile hasnt been around long enough to really measure it accuratley. Initial studies show the projects are signifigantly better off than traditional waterfall approaches, but these studies are mostly academic. For example IEEE Software did a study at a university where 2 teams developed the same app, one agile one waterfall. The agile team kicked the other teams asses in time to market and code quality.

    Anyway all we have talked about is the scheduling stuff in agile. There is a lot more too it i could talk about if interested but its not really on topic.

  6. This comment has been removed by the author.

  7. i didn't realize there were the comments before I wrote my response.

    you can read mine here.

    one thing I forgot to mention b/c the post was getting pretty long was that we used some very basic Scrum-oriented web app to manage our estimates/progress.

    At my other clients we didn't have any task based tracking software (WE NEED IT RIGHT NOW!). I have used a tool called JIRA in the past and its a very robust issue tracking web application which I am trying to get them to use now (or something like it)

  8. You should definitely take a look at FogBugz if you like JIRA. I haven't used JIRA, but I watched their demo videos and read about the features. It seems pretty decent, but I would guess FogBugz is a bit nicer. They seem to be at about the same price point too. Just in case you might be interested.

  9. In reference to the original Blog post, Kevin, I noticed you referenced WIT Tools Blog entries. Based on the content of those blog entries it appeared you deducted that TFS Developers didn't think estimating at the smaller level was important.

    I wrote those blog posts, and I certainly don't believe that, so I guess I'm trying to figure out what I said to make someone draw the conclusion you did :-)

    I work on the TFS team, and as for TFS and Estimation, I agree that our story is not complete.

    I do believe that TFS is in a good spot to really nail an estimation solution, and that's what I want to make happen. I have been a long-time believer in estimation, time tracking, and mostly removing the stigma that software engineers are inept at estimating.

    Thanks for posting this blog.
    The conversations on this blog post have been great.

  10. Hey Gregg, thanks for the comment.

    I went and re-read your three posts on Time Tracking. I may in fact have misinterpreted your first post which is listing arguments of why people don't track time and thought those arguments were yours, not other people's. Sorry about that.

    However, I was also responding to the way TFS is currently setup. The default MSF template seems inconsistent with its remaining work/completed work fields and complexity fields. The scheduling fields appear on tasks but not bugs (why?). And there is no "burn down" type report based on remaining work.

    In other words, it feels like there was a plan, but it might not have been completed before release.

    However, again, on second read I think I did misinterpret your stance in those posts. Your part 3 post in particular aligns quite well with what I'd currently like to see.

    I'll write up a follow up to this post including the ideas and goals I've collected as I've been researching and experimenting.