I just had the pleasure of writing a "primer" on how it came to be that most companies have an hierarchic organizational design, and why this is bad. And I also realized why command-and-control companies match so well with the infamous Waterfall project process. So I thought I would share the text here too!
Before the industrial revolution, knowledge on how to build for example a wheelbarrow or a carriage was passed on from generation to generation within families or guilds. This knowledge existed in the craftsmen’s brains and was not documented. Standardization, for example for tools and screws, was non-existent.
However, as a consequence of the industrial revolution, this started to change in the 18th century. In order to be able to mass produce carriages, for example, the knowledge on how to build the carriage had to be documented, so that new workers could learn fast. Standardization of tools etc also became a requirement - imagine the chaos that would come if one attempted to mass produce wheelbarrows without standardized screws!
As the industrial revolution continued, craftsmen were replaced by workers. Where craftsmen saw “the big picture”, workers only saw a small sub part of the entire manufacturing process. “The big picture” was only seen by management. Management became the brain and workers the hands of the industrial revolution.
Hierarchies were built. Work was divided into detailed sequences, which were documented and taught to new workers. This type of organizational structure is called “command and control”; top management has full command and full control over what the workers do. Hence, a manager’s top priority became to control his workers.
The command and control setup might have worked in the Ford factories of the 1920s. However, in a modern world the many problems with an hierarchical organization becomes apparent.
People are often organized in functions and only know of that function. An example of this in the software development context is that in this type of organizational design, requirement people, developers, testers and operations people sit in different departments. Organizing people in this way means that handover of information has to be done between the functions, otherwise it will be impossible for the organization to work together. Very often this type information handover has to flow up and down in the hierarchies. That tends to become very slow and inefficient. Hence, the information flow between these departments becomes sub optimal. As a result, developers may spend weeks developing something that was just a misunderstanding between the requirement people and the developers. In turn, the developers may continue for weeks writing sub-standard code that won’t test correctly, just because they don’t sit next to the testers. And even if the code does test correctly, it might not be possible to install in the production environment, since the operations people never discussed the new operation system with the developers.
Another problem with people sitting in functions is that only top management has “the big picture”. But only in theory, since we just stated that the hierarchical information flow is often slow and inefficient. Which means that not even top management has “the big picture”. Therefore, one might argue that the people in the top don’t have the best knowledge within the organization, but the worst. Still, it is these people that make all the important decisions!
Because of slow and inefficient information flow, it may take really long for a company to correct its course. Let’s say nobody buys the company’s most important product since a competitor just entered the market with a much better and cheaper solution. If information flows slowly it may take months before management acts on this. And then it might be too late!
Essentially, the classic command-and-control organization is an organization where all work has been broken down and sequentially defined. It is an organization without any working feedback loop. The way software development projects are run within this type of command-and-control organization is very similar to the actual organization. Projects in an company where work items are broken down in advance and sequentially defined must of course be… just that. A project in this type of organization therefore normally attempts to follow the Waterfall process.In a waterfall process, a project starts with creating all the requirements. When these have been signed off, the project moves to the next phase: design. In this phase, it is decided exactly how the system shall be built. Huge amounts of documentation is created. When this design documentation has been signed off, it’s time for the actual development, which should - in theory - be very simple since all the design has already been made. And when the development has been completed and signed off, it’s time to test the software. And if it actually tests ok - and why shouldn’t it, as long as the design has been followed :-) - it’s time to sign off the test phase and deploy the product in production.
And of course, this usually fails. Quite often it fails miserably. It fails because you cannot write good requirements without having any real feel for the actual end result. You cannot create a good design without input from the development, test and deployment phases. The waterfall process may look really neat on paper. So does the command-and-control organization. In reality, neither works in the modern fast-changing world!
You may try to get it to work anyway and make a fixed time fixed scope agreement with your customer. It will fail. The time originally designated for testing will be used for developing workarounds and fixes for nasty bugs. And when you finally go into production three months late you will realize that much too much time was spent on writing overly detailed requirements for complicated features that no customers use. While your main competitor managed to deliver a similar but more streamlined and less wasteful solution six months earlier...
