"The Issues of Software Development"




Copyright © 2005-2007, Cybron, Inc.

Introduction

The purpose of this article is to provide some background as to why software can be so complicated and expensive to develop. The target audience is Business Management. This article is not comprehensive. It is also general. However, it should explain the problem space in a way that is understandable, conferrable, and defendable.

The Problems

The problems listed here are ordered according to anticipated priority of concerns. These are not theoretical problems, as the author has witnessed each of them. Note that these problems are more specific to internal Development Groups.

Projects are difficult to manage

Business Management must rely on unverifiable status reports

The information source is often the Development Group. Verification is generally infeasible and may generate resentment. It is not in the interest of Development Groups to provide information that will alarm or invoke scrutiny from Business Management. Such scrutiny inevitably causes further inefficiency for the Development Group.

Business Management must rely on unverifiable council

The source of expert council is generally the Development Group itself. Such council is constrained to the expertise of the Development Group. It is also influenced by the particular interests of the Development Group. The Business Manager must then trust that the council will be defendable under scrutiny. Thus, the Business Manager becomes tied to the technical success of the Development Group. This subtle leverage is quite significant.

Development Group internal dynamics are problematic

Software Development is divergent from other engineering disciplines in regard to the personality types of the people involved. It is a creative discipline that requires individuals who are adept at assimilating abstract and complicated problem spaces. Consequently, the internal dynamics are complicated. External interference of the internal dynamics is rarely productive. That is, Business Management will have limited leverage to address process issues within the Development Group.

Development Group interests are often unaligned to business interests

Development Groups like working on interesting and visible projects. Implementation decisions are often influenced by the technical motivations of the individual members. Tedious and low visibility features receive less attention, despite their business merits. Career interests may direct the choice of inferior new technologies over older proven technologies. Experienced developers often push to implement the more visible features, instead of the more crucial.

Development Groups are rarely held accountable for failures

While Development Groups are cohesive during a project, individual members often move to other groups after completion. The stigma of being on a failed project has little residual effect the individual members. The reasons for failure are cooperatively obscured by the Development Group during the project. It is usually too difficult to determine individual responsibility, let alone hold anyone accountable.

Projects are rarely completed on schedule

The Planning Phase takes longer than anticipated

Unfortunately, Business Management is usually responsible for this. They have a general budget and a sizable list of business goals. The allure of finding an optimal set of features given the various budget constraints is intoxicating. New information and constraints may often change the proposed goals and the resulting Feature Set. Business Management is motivated to postpone feature set decisions, or risk critical peer review for inadequate deliberation. Typically, a determination of a Feature Set is postponed too long, and even then it is not very stable.

The problem is that the schedule and technical assessment of a given Feature Set takes much longer to determine than the Business Manager realizes. This is important: Feature Set changes typically generate the most significant cost, schedule, and quality impacts on a project. It becomes even more costly during Implementation and Testing. Coincidently, it is one of the few variables that the Business Manager has the most control over.

Development Groups are generally over-optimistic

Development Groups are over-optimistic when estimating the time and efforts needed to complete features and projects. They necessarily qualify their estimates with phrases like "if nothing unexpected occurs", "it isnít unreasonable", "it should", or "theoretically".

The number of events that can negatively affect a project is staggering. Historically, doubling a reasonable estimate has been about as accurate a method as any for predicting the actual cost. However, this method rarely gets used, since it is hard to defend. More rigorous methods exist, but they are costly, and generally produce marginally better results.

Unable to clearly explain large buffers in estimates, Development Groups and individual members are essentially forced to provide qualified estimates for best case scenarios they know will rarely occur.

Project Goals are prone to alteration

During the course of a project, business needs will often change. Such changes may result in feature set changes or modifications. The resulting feature changes are expensive and difficult to avoid.

Projects are rarely completed within budget

Unanticipated complications are bound to occur

Complications are often discovered during Implementation and Verification. The worst of them are related to incidental instability and performance issues. Other types are that external components do not function according to their interface or functional descriptions.

Incidental problems are time consuming to reproduce and resolve. They are often left unresolved until late in the project. Their underlying causes may never be determined. Workarounds requiring some amount of redesign may be needed.

Issues with external components may require significant redesign. The resolution of such issues can be quite time consuming, depending on support and licensing issues. Sometimes, such issues can invalidate the entire feasibility of the project.

While some buffer is generally provided to account for such complications, it is rare that an external component or technology is more stable or performs better than expected.

Additional resources are needed to address additional features and constraints.

Changes to features, general constraints, unexpected events, and unanticipated complication all generate additional tasks. If the completion date is already extended as much as possible, the only way to meet the deadline is to cut features or bring in more resources. Using these resources incurs an additional unbudgeted cost to a project.

The results may not satisfy the goals

Performance and stability issues may affect end user adoption

Typical end-user usage and utilization is difficult to predict. Seemingly small aspects in performance may reduce the utility of a feature. For example, if the search mechanism of an inventory control program is too cumbersome, the end-user may avoid using it, preferring other less efficient methods.

Unexpected usage issues may arise

New features may actually generate additional problems. For instance, a search feature may be used in unanticipated ways causing severe performance issues for other applications as it overloads the database server.

New security problems may emerge

The end-user may be able avoid a required, but cumbersome, business workflow using new functionality in intended ways. For example, if creating a product description requires a complex approval process, but new functionality allows an author to directly edit an existing article, an author may just rewrite an existing article, avoiding the review process workflow.

External events may invalidate the initial goals

An external condition which was a basis for the project may change. For example, a new inexpensive external application addresses the very problems that the custom project was designed to address.

The features were not reflective of the actual problems

A portion of this problem is due to the difficulty in prioritizing actual problems and determining an optimal feature set. It is often due to some dynamic unassociated to the core requirements. For instance, upper management may require the inclusion of a technology for strategic reasons, incurring additional costs and possibly restricting the ability to address a core requirement.

The quality of the results may vary greatly

The internal dynamics Development Group can dramatically affect the results

Organizationally, Development Groups are generally uniform. However, given the same project goals and constraints, even similar Development Groups will often produce very different results. Subtle differences in the views of individual members can have quite an effect on component choices and feature implementations. The total amount of divergence is related to the specificity of the project goals and constraints.

Development processes are not uniform across teams

The idea of Software Engineering has been around for a very long time. However, the technical instability of the industry combined with the creative nature of the participants generally repels the ability to enforce standardization. Even minor issues like variable name notation remain indefinitely unresolved.

The result is that quality, reliability, supportability, and efficient development of a project will all differ significantly based on the particular Development Group implementing it.

Work environment inordinately affects Development Group dynamix

Development Groups are full of creative people who are often disproportionately affected by their work environment. Morale and motivation are primary factors of a projects success and efficiency. Organizational, policy, benefit, management, and equipment changes will affect morale and motivation. The affect of morale and motivation on Deliverables is significant.

The longevity of a project's results can vary greatly

External components may become unstable

Either through direct updates, or updates of dependant components, the stability of an external component can change. For instance, your component may relying on a web browser that altered the way it utilizes frames during a minor update, causing your released product to malfunction to a such a degree that patches and new releases must be made.

External process changes reduce the demand for the project results

For example, the project results were a feature rich reporting tool which used a log file generated from a shared web server. However, it was determined that the generation of such log files could be a performance or security issue. So company policy no longer allows the generation of such log files.

Unanticipated security vulnerabilities may emerge

For example, the project design may rely heavily on what was thought to be a secure component. However, after the project is completed, it is discovered that the component exposes security vulnerabilities.

Licensing terms of dependant external components may invalidate the utility of the results

For example, the proprietary voice recognition component utilized by your application has a licensing agreement that is incompatible with the way the product needs to be competitively marketed.

Causes

Like most problems of some scale, the causes of Software Development problems are interrelated. It can best be viewed in terms of chaos theory. That is, determine those variables which have the strongest influence on the system. Granted, other variables may emerge. It is possible that some such variables may create even worse conditions. However, it is the opinion of the author, that such risk is worth taking.

The State of the Industry

Software components, technologies, and operating systems are too instable. They are instable in terms of performance, security, consistency, reliability, and longevity. This creates nearly intractable complexity for Business Manager to address when analyzing the cost-benefit analysis of Software Development.

An understated analogy would be attempting to estimate the cost of a building when it is uncertain what materials will be used, whether those materials will perform as expected, how they will be utilized, and even how long they will last. Fortunately for home owners, the building industry is reasonably constrained to use proven materials and construction methods. Businesses wanting custom software solutions are not so lucky.

Stability is counter to the financial interests of the industry. Stability reduces the frequency at which applications, systems, and system component need upgrades. It is the simply the unwillingness of the industry to act directly against their interests without significant pressure to do so.

An application is dependent on the questionable stability of many external components. The complexity associated with this instability enables the industry to avoid legal and financial liability. Software Licensing Agreements generally state that they may not work correctly, and will assume no liability. Certainly, they cannot assume liability for the instability of external components they must utilize.

Costing and Planning

A key problem stemming from general instability is the difficulty of costing and planning a Software Project with a reasonable amount of confidence. The planning and costing of a project end up being significant portions of the total cost of a project. Whereas, the actual cost of writing the source code for the software is relatively small.

General system instability is not the only complicating factor. Software projects often require solutions which may not have well understood, or existing patterns to follow. They may utilize interfaces and techniques that add additional schedule and cost risk. However, such complexity is less artificial and less avoidable than that associated with general instability.

Business Managers typically make decisions based on cost and benefit analysis. A confident determination of such is typically not prohibitive. For software development, a reasonably confident cost and benefit determination may be too costly to make, or too qualified to be useful.

Inherent Complexity

Software Development is complicated. Even if major strides were made in terms of overall stability of operating systems and components, new complexities would emerge. Hopefully, the new complexities would add greater value to the end results. Software Development follows other industries where the improvements to process and design allow for the addition of features and their corresponding complications.

Therefore, the Business Manager must rely to a great extent on the advice of experts. Too often that advice comes from the Development Groups that are planning and implementing the Software Projects. Such advice is difficult to refute without an independent expertise. This situation is easily leveraged to the continued benefit of the Development Groups.

Conclusions

Consumers and businesses are paying much more than they should for software that assumes little, if any, liability or responsibility for its quality.

The current state of the Software Industry primarily benefits the market leaders both financially and strategically. Free market dynamics have been largely suspended. The inability of the governing authority to provide meaningful remedies is likely to continue. Therefore, the problems will continue.

This isnít to say that Software Development problems cannot be effectively solved in some context. Addressing such problems is the basis for Cybron's business model. However, such solutions are not likely to engineered, implemented, or sustained broadly until the industry problems are remedied.