Team Work

Productivity

Communication explosion, vicious circle.

The life of a big-team programmer is spent Communicating with other programmers, attending meetings, reading documents, reading email, responding to it, Waiting for others to finish their job, to fix a bug, to implement some vital functionality, to finish building their component, Fighting fires, fixing build breaks, chasing somebody else's bugs, Staring at the computer screen while the machine is compiling, loading a huge program, running test suits, rebooting--and finally, when time permits-- Developing new code.

There are n(n-1)/2 possible connections between n dots. That's of the order of O(n²). By the same token, the number of possible interactions within a group of n programmers is of the order of O(n²). The number of hours they can put out is of the order of O(n). It is thus inevitable that at some point, as the size of the group increases, its members will start spending all their time communicating. In real life, people come up with various communication-overload defense mechanisms. One defense is to ignore incoming messages, another is to work odd hours (nights, weekends), when there aren't that many people around to disturb you (wishful thinking!).

As a programmer you are constantly bombarded with information from every possible direction. They will broadcast messages by email, they will drop printed documents in your mailbox, they will invite you to meetings, they will call you on the phone, or in really urgent cases they will drop by your office or cubicle and talk to you directly.

If programmers were only to write and test code (and it used to be like this not so long ago) the market would be flooded with new operating systems, applications, tools, games, educational programs, and so on, all at ridiculously low prices. As a matter of fact, almost all public domain and shareware programs are written by people with virtually no communication overhead.

The following chart shows the results of a very simple simulation. I assumed that every programmer spends 10 minutes a day communicating with every other programmer in the group. The rest of the time he or she does some real programming. The time spent programming, multiplied by the number of programmers in the group, measures team productivity-- the effective work done by the group every day. Notice that, under these assumptions, the effective work peaks at about 25 people and then starts decreasing.

But wait, there's more! The more people you have in the group, the more complicated the dependency graph. Component A cannot be tested until component B works correctly. Component B needs some new functionality from C. C is blocked waiting for a bug fix in D. People are waiting, they get frustrated, they send more email messages, they drop by each other's offices.

Not enough yet? Consider the reliability of a system with n components. The more components, the more likely it is that one of them will break. When one component is broken, the testing of other components is either impossible or at least impaired. That in turn leads to more buggy code being added to the project causing even more breakages. It seems like all these mechanisms feed on each other in one vicious circle.

In view of all this, the team's productivity curve is much too optimistic.

The other side of the coin is that raising the productivity of a programmer, either by providing better tools, better programming language, better computer, or more help in non-essential tasks, creates a positive feedback loop that amplifies the productivity of the team. If we could raise the productivity of every programmer in a fixed size project, we could reduce the size of the team-- that in turn would lead to decreased communication overhead, further increasing the effective productivity of every programmer. Every dollar invested in programmer's productivity saves several dollars that would otherwise be spent hiring other programmers.

Continuing with our simple simulation--suppose that the goal is to produce 100,000 lines of code in 500 days. I assumed the starting productivity of 16 lines of code per day per programmer, if there were no communication overhead. The following graph shows how the required size of the team shrinks with the increase in productivity.

Notice that, when the curve turns more or less linear (let's say at about 15 programmers), every 3% increase in productivity saves us one programmer, who can then be used in another project.

Several things influence productivity:

• The choice of a programming language and methodology. So far it is hard to beat C++ and object oriented methodology. If the size or speed is not an issue, other specialized higher-level languages may be appropriate (Smalltalk, Prolog, Basic, etc.). The tradeoff is also in the initial investment in the education of the team.
• The choice of project-wide conventions. Such decisions as whether to use exceptions, how to propagate errors, what is the code sharing strategy, how to deal with project-wide header files, etc., are all very difficult to correct during the development. It is much better to think about them up front.
• The choice of a programming environment and tools.
• The choice of hardware for the development. RAM and disk space are of crucial importance. Local Area Network with shared printers and email is a necessity. The need for large monitors is often overlooked.

Team Strategies

In the ideal world we would divide work between small teams and let each team provide a clear and immutable interface to which other teams would write their code. We would couple each interface with a well defined, comprehensive and detailed contract. The interaction between teams would be reduced to the exchange of interface specifications and periodic updates as to which part of the contract has already been implemented.

This ideal situation is, to some extent, realized when the team uses externally created components, such as libraries, operating system API's (application programmer interfaces), etc. Everything whose interface and contract can be easily described is a good candidate for a library. For instance, the string manipulation library, the library of container objects, iostreams, etc., they are all well described either in on-line help, or in compiler manuals, or in other books. Some API's are not that easily described, so using them is often a matter of trial and error (or customer support calls).

In real world, things are more complicated than that. Yes, we do divide work between small teams and they do try to come up with some interfaces and contracts. However, the interfaces are far from immutable, the contracts are far from comprehensive, and they are being constantly re-negotiated between teams. All we can do is to try to smooth out this process as much as possible.

You have to start with a good design. Spend as much time as necessary on designing good hierarchical structure of the product. This structure will be translated into the hierarchical structure of teams. The better the design, the clearer the interfaces and contracts between all parts of the product. That means fewer changes and less negotiating at the later stages of the implementation.

Divide the work in clear correspondence to the structure of the design, taking into account communication needs. Already during the design, as soon as the structure crystallizes, assign team leaders to all the top level components. Let them negotiate the contracts between themselves. Each team leader in turn should repeat this procedure with his or her team, designing the structure of the component and, if necessary, assigning sub-components to leaders of smaller teams.

The whole design should go through several passes. The results of lower level design should serve as feedback to the design of higher level components, and eventually contribute to the design of the whole product. Each team writes its own part of the specification. These specifications are reviewed by other teams responsible for other parts of the same higher level component.

The more negotiating is done up front, during the design, the better the chances of smooth implementation. The negotiations should be structured in such a way that there are only a few people involved at a time. A "plenary" meeting is useful to describe the top level design of the product to all members of all teams, so that everybody knows what the big picture is. Such meetings are also useful during the implementation phase to monitor the progress of the product. They are not a good forum for design discussions.

Contract negotiations during implementation usually look like this: Some member of team A is using one of the team B's interfaces according to his or her understanding of the contract. Unexpectedly the code behind the interface returns an error, asserts, raises an exception, or goes haywire. The member of team A either goes to team B's leader to ask who is responsible for the implementation of the interface in question, or directly to the implementor of this interface to ask what caused the strange behavior. The implementor either clarifies the contract, changes it according to the needs of team A, fixes the implementation to fulfill the contract, or takes over the tracing of the bug. If a change is made to component B, it has to be thoroughly tested to see if it doesn't cause any unexpected problems for other users of B.

During the implementation of some major new functionality it may be necessary to ask other teams to change or extend their interfaces and/or contracts. This is considered a re-design. A re-design, like any other disturbance in the system, produces concentric waves. The feedback process, described previously in the course of the original design, should be repeated again. The interface and contract disturbance are propagated first within the teams that are immediately involved (so that they make sure that the changes are indeed necessary, and to try to describe these changes in some detail.), then up towards the top. Somewhere on the way to the top the disturbances of the design may get dumped completely. Or they may reach the very top and change the way top level objects interact. At that point the changes will be propagated downwards to all the involved teams. Their feedback is then bounced back towards the top, and the process is repeated as many times as is necessary for the changes to stabilize themselves. This "annealing" process ends when the project reaches a new state of equilibrium.