It is my observation that there is a clear division of software into two very different software industries, which I am defining as Data Processing/Information Technology (DP/IT) and Software Product Engineering (SPE). While I feel most software can be categorised to almost entirely one or the other of these fields, there does also exist also some developments that incorporate elements of both.
I also suggest there are two very different philosophies of software development, and there is a different relationship between each industry and each software development philosophy.
The Two software industries
Why new definitions?
Software has evolved into quite different directions, and Technology Product Software has seen the most significant evolution. The Constant Evolution of the industry overall also means an evolution of terminology, and almost every existing term has a variety of interpretations. So rather than try and appropriate existing terms, I have created new acronyms.
I believe the software industry overall has now become sufficiently large to look at the components, and that the once completely dominant sector of Bespoke Corporate Software Engineering has seen the Technology Product Software Engineering branch of in evolution to the extent that this now could be seen as a different industry.
I am defining terms which I find most useful for 2017. Hopefully these definitions will maintain some relevance for at least a few years, and hopefully they are useful for at least the discussions on these pages.
Bespoke Corporate Software Engineering (BCPE)
Frequently run by IT departments within companies with perhaps a CIO (Chief Information Officer) as the head of the activity, this type of computing has also fits reasonably well with the label data processing.
If you consider a large bank or government department with its own software development team, it could seem that software in this sense is not really an industry, but rather an activity within banking, government or whatever. What has arisen as an industry in its own right is the professional services industry providing software development services as an outsource alternative to banks, government etc, having their own internal software teams. ‘In house’ BCPE can be seen as vertical integration and in reality part of this same industry. This software services industry has large accounting firms, and a number of computer companies who historically sold mainframe computers (Unisys, IBM etc.) as some of the largest players within the industry.
At one time virtually the entire software industry fit in this category, and I suspect this category is still the largest employer. Key characteristics are as follows:
- Software Development within a large non-technology corporation
- Software assists corporation with internal functions
- The software developed is not sold/provided by the corporation as a product
- The IT Crowd as satirical example
- Bespoke business software development for use within a single organisation
- A single customer (sometimes by way of a committee) to accept software
- Also includes outsourced software development for large non-technology companies
- Industry revenue is professional service fees to external providers and salaries for corporate staff
Technology Product Software Engineering (TPSE)
While Bespoke Corporate Software is developed specifically for use within the corporation sponsoring the development, Technology Product Software is produced in order to sell copies of the software developed, or provide a product or service based on the software. This Software is about Technology companies where the software is the product, and the main business, of the company. Classic examples would be a software game to run on a phone, or Adobe Photoshop, newer web examples would be Google and Facebook. The software is produced to used by an entire marketplace, rather than delivered to the specifications agreed with one specific customer. The financial success of Product software projects than projects developing bespoke software, but the potential rewards are far greater. This is a newer industry than bespoke software, but growing rapidly and now of greater economic value than bespoke software, although almost certainly still a smaller employer in total numbers than bespoke software. Characteristics:
The decision to develop the software is made by a business with software as core activity of the business
- Software development by a technology company: the core business of company initiating the software development is software technology
- Any assistance with internal business function becomes secondary to provision of the software to a number of customers
- The software developed is sold/provided by the corporation as a product
- The Internship as comedic example
- The customer is a market segment, not a single business
- Customers can independently select to use the software product or not
- The overall software team is usually also responsible for product design
- Industry is directly a result of the software produced by the company
The Core test: Is development initiated by a Tech company where software is a core business function?
For a bank, the core business is being a bank. Generally, software to assist the bank be a bank can be considered a ‘non-core’ business activity. This means that generally a bank can outsource software development, because it is non-core, while a software technology company would cease to be a software technology company if that company outsourced software development. Companies such as those the bank outsources to are also part of the bespoke software industry under this definition unless they initiate the development themselves. Technology Product software under this definition requires it is the technology company that identifies the need, and funds building the product in order to then promote the product to the chosen market.
When worlds collide: BCSE/TPSE Crossover
Not all software can be categorised as 100% bespoke or 100% product. Consider an internet banking application within a bank. Such an application follows product software engineering in that the customer for the software cannot ‘approve’ the software in the same manner as is possible for internal bespoke software engineering. Does the ‘customer’ of the software have a choice, or does being a customer of the bank remove the customer choice? Certainly the bank customers can elect not the use the app, but these same customers are unlikely to be able to chose an alternative application. The end result is to some degree a mix of bespoke and product software engineering.
Now consider the software inside a toaster which uses a microprocessor. Does the customer choose the application? Or is it the non-software features which fully determine who chooses the product? Perhaps a mix of the two? Again there is some mix of both product and bespoke development to embedded software. Could the company retain its own identify if the software development is outsourced? If yes, then the software is non core to company identity so the bespoke software model will be the best fit.
Who Cares? What does it matter?
I suggest it is important for correct analysis to determine if there are two or more distinct sub-groups. Consider the following:
In my fish tank, the average fish has 1.5 legs.
This perhaps confusing result comes from failing to consider goldfish and starfish are actually two separate groups which deserve their own statistics.
So, as these pages analyse data, recognising there are two different types of software development becomes significant, if the two are actually different. It is my experience these two types of development are different.
I suggest that different skill sets, is part of the reason that India such a significant role in Bespoke Corporate Software Engineering while Silicon Valley USA plays such a significant role in Technology Product Software Engineering.
I have heard statements such as: “the trend is that all software development is being outsourced”. As discussed elsewhere, if software is your core business, you cannot really outsource so this statement specifically applies to bespoke (BCSE), not to the technology product industry. Assuming a trend will apply across all software can be misguided if different factors apply for two different sectors.
Understanding this difference could also be significant those seeking to work in software, who may need to understand what trends actually apply in their chosen sector. Similarly colleges offering courses may need to understand which for which sector their potential graduates should acquire skills.
Two Competing Development Philosophies
There is a famous quote the on the world market being limited to about 5 computers, that at least sounds like at one time it was believable, even if not very astute. For those first few computers, even operating systems were bespoke software, so the software industry began with only bespoke software and all else came later. No only was the first software bespoke, but with the only reference building physical constructions, for ‘building’ software was building a bespoke physical construction.
Construction project development model
Before any construction (for example, the construction of a bridge or building) commences, you need the design and the plans. A construction has steps, similar to these:
- collect design ideas
- budget the project
- finalise a design agreed by stakeholders
- finalise budget of actual design
- construct to match the design
- ensure compliance of the construction to the design and regulatory requirements
- open the construction for use
- now only maintenance if performed unless a specific separate upgrade project is launched
The project can only be ‘open to the public’, after construction is finished. Software can follow this same process, and with software is called ‘waterfall’. Waterfall does have certain advantages:
- it is very easy for non-software people to relate to
- it easily adopts to the bespoke software projects
- the closed nature of the project works with a single budget allocation
- separation of steps may enable use of specialists for each step
- more narrowly skilled and widely available ‘construction workers’ for the construction phase
- can enable outsourcing of specific steps while still retaining design control
This construction project process can be particularly suited to bespoke development, but has also been used for some notable product developments, and sometimes adapted with each product version or major release being considered an separate development with its own set of the ‘waterfall’ steps.
The final test for ”design and regulatory requirements”, is generally know as UAT (User Acceptance Testing) when applied to software.
Although ‘waterfall’ is out of favour as not ‘agile’, if the design requirements are sufficiently concrete from the outset, then ‘agile’ may not be as necessary as for projects which are exploring new and less tried ideas. In the end, all development is either single waterfall, or an series of iterations of the waterfall process.
What does finished mean? With the single waterfall type process, the software is finished at the end of the cycle when development has stopped. But consider in todays world the reaction if a major product software vendor stops development on a software product. For example, if Canonical announced that development of Ubuntu will stop as the product is complete. For a product, stopping development does not signal the product is complete, but rather that the project should be considered dead. For the Ubuntu example, people would look for an alternative product once it has been announced that at some future date work will stop. This may be the opposite of bespoke software waterfall development where managers look forward to the day the development is ‘complete’. Finished with software, can mean something entirely different than it does with construction projects.
Software is all design. A program is essentially a fully detailed set of instructions, specifying exactly the steps that must be performed. This means there is no exact fully detailed specification other than the completed software. While a system design document can be created, an accurate design document is in a sense an overview of the program itself, just lacking detail. The design documents of a construction project do not lack any details, and the ‘build’ phase is about turning the instructions into something physical. The instructions of a computer program can be ‘built’ into the actual program by the computer itself in seconds, the challenge is specifying all the details of what must be built. This fundamentally changes the project to effectively all design, and design is what must be agreed before a construction project can commence. So in essence, the full design cannot be pre-agreed and this leads to problems in construction project model.
Living organism project development
In some cases, software can be considered as more similar to a living organism than to than to a construction project. The steps become:
- The project is conceived but is not really functional at that time
- The project goes through an embryonic phase where it is incapable of any significant functionality
- The project is then ‘born’ and has first contact with the outside world but that contact is best limited as the young project is nurtured
- The project gains capabilities as it matures over time (like the ability to walk or talk) and learns from the environment
- Development continues with milestones reached although exact milestones may seem somewhat artificial (birthdays vs releases)
- It is at some time it is declared the project is mature, although the exact point of maturity may in some ways be arbitrary
- The project continues to develop
- At some point development slows the project gets old
- The project may eventually die although its place may now be filled by its own offspring
Idioms can only run so far, but there is much overlap with software products.
The reality is that waterfall never goes away, it just becomes very short iterations of waterfall, and these short iterations mean every stage of the waterfall must be able to be run very quickly. This has the following consequences:
- Design and development become intertwined, so designer and builder must either be the same person or a very small team with both skills
- A single comprehensive UAT style testing is impractical for all but ‘last’ waterfall cycle, and the ‘last’ cycle may never really exist. A new concept in testing that can occur over and over in a short time is required.
- The living organism allows the project to adapt in an ‘agile’ manner which may see the project differ from the original brief, which can be problematic for a bespoke development contract
Agile / Living organism model and product software
While the living organism model may better match what is required from development, the result can be a challenge for monetising software. While a series of major new product versions can provide points at which customers pay fees, continual product evolution requires either customers pay monthly/annual fees or another revenue model is required. Distinct releases provide for points at which companies can charge for new versions. Despite the need for new business models, product software has changed. This is why Windows 10 is scheduled to continually evolve with updates, rather than be replaced by a new version of windows as has happened in the past. Development tools from Jetbrains all follow monthly or annual licence fees, and Adobe creative cloud invites you to choose a plan rather than purchase a version of a product as in the past.
Agile / Living organism model and bespoke software
Again, the living organism model is a problem for the way projects are run financially. With bespoke software, the challenge of fitting the business model can be even greater than with products. The result is that the while almost all bespoke developments declare they are agile, this ranges from “because we hotdesk” through to actually having some iterations during development, but very rarely to actually adopting a full living organism view of software. There are government departments, with annual IT budgets with little variation and teams of permanently employed outsourced or internal staff building projects, but they still view every IT project as a separate construction project and do not see a these as part of an overall picture where they are funding resources to deliver the continued evolution a set of software continuing to their needs.
There are currently two software industries and two software methodologies.
However, the two industries are in place because of mindset and the pressure of business models, not different business needs. There is a mindset that building software must be like building houses, and a failure to grasp that in essence software is just a detailed design. Yes, for large projects, separating design and then translating that design to code is a task that can be done with lower cost programmers in place of designer/developers, but still evolution will see this decrease. In the same way typists were lower cost than managers, but eventually the combination of technology and education automated typing to a level where managers now type, programmers who are not also designing the code will likely go the way of the specialist typist. The bespoke industry will decline, but t will still take some time. This data from github suggests that the product software industry could be a small as 22% of the overall software industry. The decline is currently more in the wages for these roles and the pressure from low cost off shore developers than the slow move to increase seeing all projects like products.
The two design methodologies follow more from business models that have not adapted, rather than simply being genuine alternatives. As the industry matures further expect software project to continue to move to the living organism model. All programs need to continue to evolve to remain relevant, and the living organism model is the most appropriate development model.
Software is in fact like a living organism in that is needs evolve to continue to reproduce. Virus software may reproduce through its own code, but other software is symbiotic and relies on being chosen by humans to multiply. If users are not happy with the software they do not install copies, so evolution means only liked software reproduces.