Life Is Easier When You Have a Process in Your Pocket (7/8/06)

Construction project management is pretty much down to a science these days. Everything depends on sets of drawings and specifications. Site plans start out as general tools for discussion about parking, building shape, and placement and subsequently evolve into descriptions of utility placement and lot drainage. Scale models of buildings become stacks of separate plans for walls and doors, plumbing, electrical, and HVAC. Many subcontractors report to a single general contractor, who in turn works under the architectural or architectural/engineering firm that is in charge of the project. Somewhere in there is a project manager, sometimes as an employee of the architectural firm but most of the time as an employee of the general contractor. There are tons and tons of things to do, and the project plan can be huge, but here's a little secret: the project plan does not identify every little task to be performed by every worker on the project.

In a past life, I ran a small security systems company. This put me squarely in the middle of a large number of commercial and residential construction projects. The project plans never said "on Tuesday, September 17, Joe Smith will be mounting components 22-36 in the north west wing." Instead, the plans typically had a maximum granularity of one week, with milestones establishing the hierarchy of activities. For example, the ditching for the utilities had to be in place before the utilities could be brought to the site. However, since the plans showed where the utilities would wind up, the electrician was free to rough in the electrical distribution panel with a reasonable assurance that the electrical main would appear at the appropriate point on the opposite side of the wall. However, the distribution panel installation was significantly dependent on the existence of the wall itself. Not a finished wall: this kind of stuff is done when the studs are still exposed.

So the project plan establishes the order of major activities at the project site, and in combination with the architectural and engineering plans, everyone knows what is coming up next and who is responsible. Most subcontractors follow the same general pattern: 1) site survey, 2) site prep, 3) rough-in, 4) finish work, 5) punch list (or system test), and 6) cleanup. For example, electrical site prep and rough-in has to be done after the walls are roughed in, but before the walls are finished. Basically, a set of holes are drilled through the studs and wires are run before the drywall goes up. The net result is a bunch of ugly wires sticking out of the walls at various places.

Again, the project plan does not have a task identifying Joe Smith as actually drilling a bunch of holes in studs on a particular day in September. Instead, the project manager notifies the electrical subcontractor that the walls have been roughed and it is time for them to begin site prep. The details take care of themselves through the subcontractors referencing the architectural plans and using their specific skills to accomplish the necessary activities. The project manager is constantly adjusting the estimated completion date based on actual progress against current milestones. This is what happens when you have a process in place.

This is not what happens with most software projects. In many cases, the developers are in charge of the project. This generally results in the lead developer mumbling something about "agile development" being good and thus justifying an endless prototyping cycle that never appears to really get anywhere. In other cases, a "project manager" is brought on board and they immediately attempt to build out the most wondrous project schedule anyone has ever seen, with complete definitions of just exactly what Joe Smith is going to be doing on September 17. Unfortunately, the structure and milestones are only vaguely related to what is actually going to happen, there are giant holes in the plan because the developers did not tell the project manager about any of the infrastructure tasks associated with the project, and the estimates are all wrong anyway. I remember one frustrated software project manager telling me "it seems I have to take every estimate and multiply by 1000 to get anywhere close to reality!"

Software project success depends on two things: 1) a defined set of high-level activities (a process), and 2) a specification. The process has to rely on the presence of a specification, and a specification will not be created unless it is a prerequisite activity to development as defined by the process. Without a process, there will be no agreement on what needs to be done first, and money will be wasted. Without a specification, there will not be enough detail available to verify that an activity has been completed. The biggest mistake most new project managers make when addressing a software project is to try to define the process within the project schedule, or worse, build some sort of specification into the project schedule. Project managers are not software engineering experts, nor are they intended to serve as systems analysts, but the newbies always try.

Here's a quick gap analysis between construction projects (which are largely successful) and software products (which are usually a significant gamble):

Construction vs. Software
Element Construction Software
Architect Formally educated, licensed, large and in charge. Fee generally based on a percentage of project costs. An occasional powerless systems analyst.
Project manager Reports to architect, acts as coordinator. Either not present, or present and in charge even though they are not qualified.
Contractor Formally licensed, works from the plans, knows how to dance with the others. Rarely has a specification to work from, considers themselves to be in charge anyway, does their own thing in the name of "agile" development.
Process Well understood, common process taught in schools and in the field. Waterfall is bad, agile is good, and the band plays on as the ship slowly sinks.
Specification A standardized, controlling and controlled artifact, provided by the architect, that must be on file before a building permit is issued. A bad thing, because having a specification means you are doing waterfall, which is bad. Besides, there are no standards, and if some systems analyst produces one, it is more of a guidance document to be ignored than a controlling artifact.
Worker qualifications Well-defined, from apprentice to master, licensure requirements for those in charge. The Association of Computing Machinery states: "ACM is opposed to the licensing of software engineers because ACM believes that licensing would not be effective at providing assurances that software engineers could produce reliable, dependable, and usable software systems. When the body of knowledge matures and we have proven experience in certifying software engineering skills, ACM may reconsider its position." In other words, we have produced a huge number of degreed individuals over the past 40 years, but we can't legally rely on any of them because we're still debating about how to do this stuff in the first place.

The hallmarks of a successful software development team or firm include a standard process, real specifications, people that know how to work with both, and an infrastructure to tie it all together. If any of these elements are missing, the software development organization is neither scalable nor sustainable. We all need to stop debating and fill the gaps before somebody powerful gets sick of all this and legislates new career paths for all of us.