The Hidden Cost Of Estimation

By 5 commentsagile, NoEstimates, planning

“Why would you want a rough estimate, when I can do a more precise one?”

And really, if we can do something better, why do it half way?

There’s a simple answer, but I’ll give it after the long detailed one.

Let’s start by asking again:

Why estimate at all?

There’s a whole #NoEstimates discussion, whether we need estimations or not. Unless your organization is mature enough to handle the truth, someone will want an estimation, believing he can do something with it: approve the task, delay it, budget for it, plan subsequent operations. That someone needs information to make decisions, and is basing them on the numbers we give.

In reality, unless there are orders of magnitude between expected results in estimation it wouldn’t matter. If we had a deadline for 6 months, and the estimation is 8 months, the project will probably be approved, knowing that we can remove  scope from it. If we estimated a project will take a year, there’s going to be a 3 month buffer between it and the next one, because “we know how it works”. Things usually go forward regardless of our estimation.

If however, we estimate we need 5 times the budget than what we thought we needed, this may cause the project to cancel.

In summary, the upfront estimation serves making decision. In fact, if you just go with the discussion, and leave the number out, you can reach the same decisions.

So why do we need the numbers?

Numbers are good proxies. They are simple, manageable, and we can draw wonderful graphs with them. The fact they are wrong, or can be right in very small number of cases is really irrelevant because we like numbers.

Still, someone high up asked for them, shouldn’t we give them the best answer we can?

Indeed we should. But we need to define what is the “best answer”, and how we get it.

How do we estimate?

How do we get to “it will take 3 months” answer?

We rely on past experience. We apply our experience, hopefully, or someone else’s experience to compare similar projects from our past to the ones we’re estimating. We may have even collected data so our estimates are not based on our bad memory.

Software changes all the time, so even past numbers should be modified. We don’t know how to factor in the things we don’t know how to do, or the “unknown unknowns” that will bite us, so we multiply it by a factor until a consensus is reached. We forget stuff, we assume stuff, but in the end we get to the “3 months” answer we can live with. Sometimes.

How about the part we do know about – we can estimate that one more precisely. We can break it down to design, technology, risk and estimate it “better”.

We can. But there’s a catch.

Suppose after we finished the project, we find that 30% of it, included the “unknown unknowns” stuff.  We could have estimated the other 70% very precisely, but the whole estimation would still be volatile.

(I’m being very conservative here, the “unknown unknowns” at the time of estimation is what makes most of a project).

The simple answer

So here is what we know:

  • Estimation is mostly wrong
  • People still want them
  • It takes time to estimate
  • Precise estimation costs more
  • Precise and rough estimation have the same statistical meaning because of unknowns.

That means that we need “good enough” estimates. These are the ones that cost less, and give a good enough, trusted basis for decision for the people who ask for it.

Fredkin’s Paradox talks about how the closer the options we need to decide between, it takes longer for us to decide, while the difference in impact of choosing between the two becomes negligible. Effective estimation recognizes the paradox, and tries the fight it: because the impact of variations in the estimates, there’s no need to further deliberate them.

If you get to the same quality of answer, you should go for the cheaper option. Precise estimates are costly, and you won’t get a benefit from making them more precise. In fact, as a product manager, I wouldn’t ask for precise estimates, because they cost me money and time not being spent on actual delivery.

Working software over comprehensive documentation, remember?

 

Image source: https://openclipart.org/detail/183214/archery-target-by-algotruneman-183214

5 comments on “The Hidden Cost Of Estimation”

  1. Glen Alleman Reply

    Estimates are mostly wrong.

    Do you mean those who have no skill, experience, knowledge, or reference make wrong estimates.

    Because estimates are just a number.

    The root cause of “wrong” estimates is not with the estimates. It with those making the estimates

  2. Gil Zilberfeld Reply

    Glen,

    Thanks for the comment.

    The numbers are wrong, statistically, compared to actual work. They are wrong for all kinds of reasons, primarily because humans make them. It’s experience, bias, ego, and all kinds of elements that affect these numbers.

    Now, if we take the humans out of the equation (which is really hard), we need good reference data. I love data. But it is not so easy to come by, because software projects are different from each other, even those that look similar. Those that are the same – why build if you can buy? And if they are new, we’re back at square one – humans.

  3. Chris Marisic Reply

    The fact that ACCURATE estimation is hard does not mean it should be ignored. The fact that project design is hard does not mean it should be ignored.

    How do you make an estimate wrong? Easy. You just make up the answer. “Landing an estimate.” related it could be a business stakeholder telling you what the estimate is supposed to be.

    To create a reasoned estimate for a project you need to break the project down into well defined chunks based on the system architecture. You need to have an architecture design before you can have a project design. Architecture is not “emergent” it is either planned for or your architect consists of 1 of the 2: big ball of mud or stove pipe architecture (both antipatterns). Having system architecture planned you can then create a project design with numerous activities dependent upon the scope of the project.

    These activities are “small” to be measured in man weeks. 1 man week, 2 man weeks, 6 man weeks. Not multiple man-months for an individual activity.

    Next you need to create a network graph of all the activities and their dependencies. A and B are independent and can be worked on at the same time, C is dependent upon A and B, can only start when A and B are complete. Once you have the network graph, you need to apply float analysis. Using the above example if A is projected to be 2 weeks, and B is 1 week. B has a float of 1 week. It can start 1 week late and not change the overall project timeline. Now because B has float and A has zero float (if A overruns and takes 3 weeks, C can’t start before 3 weeks). A is part of the critical path. http://en.wikipedia.org/wiki/Critical_path_method EVERY system has a critical path, it is there whether you acknowledge it or not.

    Once you’ve defined the whole dependency chain and network graph you can take that information to a tool such as MS Project to get the floats. You can calculate the floats by hand, but MS Project does a great job of basic mathematics and I’ll contend the only purpose of using MS Project ever is for float analysis. It also will calculate the critical path (the network path that has zero float).

  4. Chris Marisic Reply

    Backing up, you might argue “how can you know how long A will take vs B will take?” This is where experience comes into play. It is mostly experience. However suppose it’s something that is brand new or radically different. How can you come up with estimates that are reasonable? There is a technique called broadband estimation. You get your entire team of software developers together, have everyone create their estimate. Then you tabulate the results. You then address the outliers, maybe they have some additional knowledge for shortcuts or see a giant pitfall that no one else did, you discuss. You then execute the estimation again and repeat until there is no more estimates outside of the standard deviation. This is time consuming but can be done for certain tasks or could be done for every single task in the system.

    After you have the network diagram and critical path determined. You can start plotting people to activities. Then you can come to a real date for an estimate, a date that is so accurate an experienced architect will deliver the system on that specific date. This project design also gives you the information you need for how many people should be on the project, it can also show what happens when the project is understaffed. The design demands 5 developers, but you only have 2. So then every item is critical or nearly critical and the risk of project failure is through the roof. Project risk is directly correlated to the amount of float time in the project, when float is 0, risk is infinite. when float is infinite, risk is zero, but cost is infinite.

    The best part about project design. It truly allows you to BE AGILE. Now that you have real project design you can actually track real progress. Going back to my example of A & B. You start A & B at the same time. Given that B should be done in 1 week, if it’s not done in 1 week you now have time to intervene. Do you need to break the activity up from 1 activity to multiple? Do you need to find an expert in _________? At the micro-level this is relatively meaningless but when you have a dozen activities or 6 dozen activities, you can monitor the project as a whole. You can see float disappearing. Float is the buffer you have for the fact humans are inefficient. Even nuclear reactors can’t approach 40% efficiency, so good luck thinking a human can even reach 20% efficiency constantly (the internal combustion clocks around 20%). You have a chance to interdict before the project is failing. Without a project design you’re always 80% done, 100% of the time. A project with no design cannot acknowledge it is off track until it is nearly release day. By the time it is release day it is too late to change the project.

  5. Gil Zilberfeld Reply

    Chris,

    Thank you for the long comment, and a long PM lesson.

    Yes, this is how you’re supposed to do it. Then again, it doesn’t always work. I’m speaking from experience, being both a PM and an architect on several projects.

    Good planning helps the understanding of the project, and therefore should yield a better estimation. Unless for very small projects, the estimation difference from the actual work, and looking back, the effort I put into building those MSProject were a very big waste (we’re talking weeks for large projects).

    Now, the whole point of estimation is to get a go ahead or a cancel. If it’s a go, you’ll get a budget and resources, based on the planning.

    The funny thing is, you’ll get the green light if the number seems plausible to the client, even if they don’t have and lock on reality. It’s like marking the price according to the market, rather than admitting “we don’t know”. I’m not really proud of that, but there was an out-bidding with the competition.

    Planning is good. But estimation can be misused in many ways. And after all, it’s based on all these pesky inefficient humans, who basically guess.

    Thanks again,
    Gil

Leave A Reply

Your email address will not be published. Required fields are marked *