Software and Scheduling
Of all the topics guaranteed to strike fear into the heart of a programmer, asking ``When will it be done?'' is surely the most effective. Figuring out how long building a software system will take is notoriously difficult — so much so that the popular wisdom says it simply cannot be done.
Nonsense. It can be done, it has been done, and it's straightforward to do. Not easy, but straightforward. This is a rant on how to do it.
Why Scheduling is Hard
There are four reasons software scheduling is difficult:
-
Because of the uncertainties and complexities involved in programming, even experienced programmers are pretty bad at estimating the time that an activity will take.
-
There is an enormous range of skill levels across programmers. There is no such thing as an ``average programmer hour.''
-
The schedulable path to software engineering leads to late integration. Knowing that software is unpredictable, upper level management generally demand visible signs of progress in the form of demonstrable deliverables. This actually defeats cost-effective development.
-
Because it takes intense and unpleasant work, software managers (or in some places, team leaders) simply aren't willing to do the planning.
Let's look at each of these issues in greater depth.
Estimates
It's a well known fact: ask people how probable something is, and if they don't know they will say "50/50". Programmers are just like that. Ask them to estimate a task that is too big and you will get a guess. This guess is complete junk. It has absolutely no value for scheduling purposes. It is wildly wrong and most likely optimistic.
But there is some threshold where the task you are asking about is small enough to be envisioned accurately in the programmer's head. For these tasks, programmers can give surprisingly good estimates. Some will be high, some will be low, but on average the estimates will be pretty accurate (though see below about the delusional multiplier).
The trick, as a software manager, is to do two things:
-
Figure out where the envisionable threshold is for your group. For junior groups it will be about half a day. For really experienced teams it's never more than two days. That is: if an experienced programmer gives an answer larger than 2 days for "How long will that take?", what it means is "I don't know".
And they really don't know. Browbeating them won't help. Your job as a manager is to assist them in breaking the task down into units they can estimate, and then asking how long those units will take.
-
Accept the responsibility as manager or team leader to plan. Breaking a project down into manageable pieces is excrutiatingly boring and frustrating work. It is the only way to generate predictable schedules. Figure that if you have a six month project, this is going to take two to three weeks of non-stop arguing up front. Maybe five.
If you aren't willing to do the work, you won't get a predictable result.
Mind you, the two day estimate is wrong. The reason you care is that it's small enough for the programmer to understand what needs doing. Once you have those numbers, you still need to apply the delusional multiplier (see below) for each programmer.
Programmer Productivity
There is an enormous gap in programmer productivity. The top 2.5% of programmers are nearly 10 times as productive as the next 13.5%, who are in turn 10 times as productive as the next 34%. This spread has been measured repeatedly, and it holds independent of programming language or task.
Programmers are delusionally optimistic. If they weren't, they wouldn't be doing this impossible task, now would they? Unfortunately, all programmers believe (or at least behave as though they believe) that they fall in the top 2.5%. The only way to cure them of this habit is measurement. It doesn't actually matter what you measure, so long as you measure consistently.
Measurement requires trust. Most managers aren't trustworthy, and so most programmers resist measurement. As a manager, you have to ask whether you want predictable results or not. In practice, it is almost always faster to seek predictable results. Betting on the star programmer is a lot like betting on a roulette wheel.
As programmer, you need to develop a self-calibration. Before you sit down to each task, get in your head an idea of what you will be doing. Make a guess as to how long it will take you. Write this guess down. Keep track of how long it actually took you. Write this down too. Over time you will discover what your personal ``delusional multiplier'' is. Remember: just because it takes 10 times as long as you thought doesn't make you a weak programmer. It probably takes everybody else 10 times as long as well. The goal here is to be predictable, not to be fast.
Pressure from Above
Every software manager experiences pressure from above in two forms:
-
The boss wants it faster.
-
The boss wants it demonstrated in stages.
The Need for Speed
The reason the boss wants it faster is really simple: programmers are incredibly expensive. If you are in a startup company, every month between start and ship adds 2.5% to the total cost of the company (assumption: 35% annually compounded ROI expectation from the investor). I mean: take all of the money your company has spent since its inception and multiply by 1.025. That is how much a one month delay in product shipment costs. It's a really good reason to want to be done quickly, and it's not just random management nonsense. That money will be important later to market and sell the product, and if you cannot do that, you might as well not build it.
But here is the catch: if I'm a manager, I can flog my troops and get about 5% shaved off of my development time if I am really lucky, but more likely I'm going to render them exhausted and add time to the whole thing. Remember: the programmers are optimists from the start, which means they are probably working as hard as they can humanly work to meet the original schedule they gave you. Adding pressure just isn't going to help. Worse, the time I add is going to be at the end of the process, so it's going to be at the most expensive end of the cost multiplication. Losing a month early, when I still have a lower headcount and burn rate, isn't so bad. Losing it late, after the headcount has built up, is just deadly.
So: tempting as it is to flog your troops, don't. One of my former bosses killed an entire company this way. The only reason he didn't kill my group is that I told him to shove it and faced him down in front of the corporate CEO!
The Need for Demos
Application software can often be built in chunks that are demoable. If this fits the schedule, it's a good thing. Those demos can be used to reveal miscommunication about requirements early. They will cause slips, but in the long term they will bring in a better product.
Certain types of software don't build demos easily. This includes anything where you have to build a lot of ``infrastructure.'' Operating systems, compilers, debuggers, databases, and so forth. If you pressure the software people into demoing, they'll do it. In order to do it, they will shove all kinds of junk into the code base, half of which they will neglect to remove and will turn into bugs in the field later. Understand what you are managing and resist the temptation to push for demos on these projects. Ask the team what the right metrics are. Try to define "demo" correctly — maybe it's simply seeing an early stage bootstrap diagnostic. Seek signs of progress rather than signs of political corruption of your technology.
The Process
The key to making all of this work is dealing with those estimates longer than two days. When you get one of those, you need to ask "Well, what goes in to that? Let's break it down." Eventually you will get down to one or two day units, and at that point you can have some confidence. The ``law of large numbers'' will now work in your favor: some things will go faster, some slower, and on average you'll come out about on target.
This process of breaking things down is going to lead you straight into design discussions. In fact, you will find that you are doing up-front design. Some of this design won't survive contact with reality, but you'll find that the magnitude of the problems won't generally change even if the specific solution does. This is what you are after: not a plan carved in stone, but a plan that is good enough to iterate.
The design process is also going to take substantial time. Programmers will say "I won't know until I build it." Nonsense. Programmers will say "That's impossible to guess at." Not if you can get them to break it down. The design process itself will not be schedulable, but it will take a small fraction of the overall time, and when it is done you will have a workable engineering plan. At that point you can decide whether or not to proceed — while you are still spending cheap money. It won't be a fun process. You'll see screaming, exhaustion, and all kinds of ``darker side'' behavior. It's also a right of passage. At the end, each member of the team will have a pretty good calibration of what the other members can do. That alone is worth it.
Here at Hopkins, and lots of other places, there is a fad going on among people who are in love with shortcuts. It's called Extreme Programming, or sometimes Agile Software Development. There are a lot of good ideas in this methodology, but a tendancy in practice to overlook the hard and ugly part of software development: design up front. Some advocates argue that design up front isn't worthwhile because requirements change. This is just lame excuse making. No, you cannot do a perfect design up front, but you can do a design that is 85% right, and that is plenty good enough to get moving with confidence and adapt as you go.
Programmers hate the discipline of up-front design until they are forced through it and see the results. It is an incredible act of pedagogical irresponsibility that CS departments don't teach any of this. Blame the ABET curriculum, which mostly prevents us from teaching practicum if we want to be ABET-certified. I sometimes think that I should go found a ``practicum'' finishing school for programmers. I'm pretty sure I could get it funded.
Bottom line: you must drive the estimated units down to credible size, or you won't get predictable scheduling. That requires up-front design. Given the costs of delay, this discipline is simply good fiscal sense. If you're a student, your time is still valuable, and predictability is what lets you plan.
Real-World Experience
I used these ideas at SGI when we built the VIEW product line of debugging and performance analysis tools. This was a 10 staff-year effort, and in the end it involved about 550,000 lines of retained code. The final deliverable was two weeks early. It was on budget, and SGI sold about $250M worth of workstations to key customers on the strength of that product at a critical time.
When I took over the software scheduling task for the Xanadu Operating Company in 1991, they had a project that was running about 25 years late. Actually, that's not fair. They had simply never understood what it meant to think in terms of concrete deliverables. The willingness was there, but the necessary discipline hadn't sunk in. Marc Stiegler was just making headway with them when he got pulled into Autodesk to turn them around. In the process, Xanadu was told that if they didn't start making their deliverables on time they would be shut down.
From the time I took over the scheduling task — and briefly, the company — they met every internal commitment to Autodesk on time. Unfortunately, it came too late for Autodesk management to recognize that a change had occurred, and Autodesk decided to spin them out. After the spinout, we went through a long planning exercise to figure out how much money we needed to raise. That was the ``fight through the design'' process, and I was nearly lynched as it occurred (mainly for that, but also for some other reasons; it was a difficult time). But once again they met every deliverable deadline with every functional requirement substantially completed. Afterwards, one of them said that it was the most painful process they had ever been through, but that it worked. Ultimately, the company failed because of the greed and selfishness of a key shareholder, and the software reverted to the old guard who had been fired because they didn't want to ship. The point here is that the scheduling method worked.
When it comes to software scheduling, if there is a substitute for hard work I haven't seen it. But the hard work does pay if you have the stomache to carry it through.
As Steve Jobs puts it: ``Real artists ship.''
Afterthoughts
Obviously, none of this is a secret. It's sitting right there in pretty much any good software engineering book. So why aren't more people doing it?
Because it's hard work, the process of up-front design is painful, and most of us don't have the discipline. Companies and software teams, in this regard, are no different from individuals.
But if you have the discipline, you can deliver on time, every time. Straightforward, but not simple.