Waterfall or Agile?

This page describes both Waterfall and Agile software development philosophies. In reality developments are not 100% waterfall or agile, but a blend of the two.

This is a ‘back to basics’ look at the two philosophies. Separate pages will consider the implications, and necessary modifications, when each is applied to different bespoke software or product software projects.


  • Why yet another page?
  • waterfall: modelled on construction projects
    • We have an agreed design and the process to build this is well known.
  • agile: modelled on engineering design projects
    • no one has ever done this before
  • hybrids?
  • Budgeting
  • The Team

Why yet another page?

In short, to try to separate the ‘religion’ from this topic, and to step back from the ‘bespoke only’ or ‘product software only’ perspective in almost every other page on this topic I have found.

An example of the ‘religion’ attitude is an article here which suggests Apple computer has poor software quality because they use ‘waterfall’ rather than ‘agile’.  It is almost certain that Apple would use a methodology encapsulating some of both ‘agile’ and ‘waterfall’,  but the implication is that label ‘waterfall’ is itself sacrilege.  While a company (and even perhaps Apple) may not embrace sound software methodology, evaluation is more complex than simply a label.

The other issue is to take each concept, and envisage only a chosen adaptation for the authors field of software.  Not only are bespoke and product software almost polar opposites, but even within each category there are factors that affect ideal methodology.  So most pages compare only for a specific environment, which is great if all your projects match that description, but less helpful for those with different or varied projects.

Waterfall:  Modelled on constructions projects.

waterfallsThe process of building something (i.e. a building) has been practiced and refined over centuries.  The waterfall approach is simply taking this construction model, and applying the model to the building of software.  Using a gantt chart,  the steps look more like a cascade of waterfalls.  But if you view a cascade from the front (see pic) then you have the appearance of water falling from step to step.  The waterfall is from step to step, not all at once.

From xbsoftware.com

The principle of waterfall is that each step is completed before the next begins. A complete understanding of requirements should be in place before design starts.  ‘Building” or development, should not commence without a completed design. Once built, tests verify the building meets the design.  These are the same steps that have been proven to work for construction projects over centuries.

Agile: Modelled on engineering design.

While waterfall considers a software project as similar to a construction project, agile effectively considers the entire software project, as being an engineering design phase.

With engineering design, you have a problem, and the challenge is to find a solution.

If it is know that some existing design can be modified to provide a solution, then there is no engineering design required.

An example from the tangible world is the design phase for mass produced products. You have a team of designers who engineer a design, but although you may have a fixed budget and time frame, it is only at the end of the design phase that you know what the final product will look like.

The principles of agile can be considered be:

  • what is being created is new and innovative
  • new ideas will need prototyping and revision
  • the best solution is not always clear at the outset
  • actual product requirements may not even be clear until the product is trialled

The process of agile software development can be considered as:

  • define initial set of requirements
  • build an as simple as possible prototype as quickly as possible
  • now repeat the following steps until satisfied:
    • analyse what how the prototype could be improved, and add those improvements to requirements
    • analyse steps that can be made within 1-2 weeks to meet more requirements and perform those steps

Note: this is representing the agile philosophy itself. Any actual implementation of agile will also depend on some other choices as to how agile will be implemented.

Consider a car manufacturer who produces a new model of a certain car every year.  The design team, with an allocated number of people and resources, is given a fixed time to come up with next years design.  Agile development is similar to this process.

Hybrids? Solutions using a combination.

Waterfall: We have a specification and the process to build this is well known.

With construction projects, you could be building an exact replica in an effectively identical site, but surely there will be something new?  In fact, the more daring the building, the more often that some part of the project will actually encompass an engineering design project, even with prototypes!  The fact that almost every waterfall project encounters some steps which better fit the agile pattern, is the reason so many waterfall projects suffer time and cost blowouts.

Agile: No one has ever done this before.

But Surely someone has done part of it before?  Won’t there are least be construction projects to build the prototypes?

The reality is that both agile and waterfall can start to approach a series of short waterfalls placed back to back. Each just approaches from the opposite side.

A very short waterfall stage loses many of the characteristics of waterfall, and a long build of a single prototype becomes more waterfall.  In the end no project is ‘pure’ and all have elements of both.  Software is always doing something new, but just how new?

The more new, the more no one has genuinely done anything like this before, the greater the need for an agile mindset.  The more ‘just like a previous one but with these straight forward but time consuming changes‘, the more a waterfall mindset may work.


No one can predict the exact cost of an exactly specified system containing elements that no one has ever done before.

As waterfall is based on the construction model, the budgeting process it quite like getting quotes having a house built to match the plans you have had drawn up.  You can have faith you will get something that matches the plans, even if it may be a little late and the real problems only occur if you realise during construction that result will not match what you actually need.

Agile is more like working with the architect.  You may have a fixed quote but you have to commit blindly without knowing in advance if you will like the result.  With agile you can control the budget, but at the expense of accepting what is produced within that budget.

The result is that  many projects are forced into a waterfall mindset from the budgeting phase, and  can only live with an agile mindset for the limited spend of the design phase.

The Team

Just as waterfall breaks a project into steps, it separates the roles needed for those steps.  Design people are expensive, but during ‘construction’ you may only need labourers who cost less.  Similarly a waterfall methodology separates project managers, annalists, and programmers. Programmers who just program become lowest cost resource, and can be more easily outsourced and managed by project managers.

Agile requires the whole team act like engineers.  As the work is always different even the construction has less guidance and more decisions on the fly by the ‘engineer’.  The result is more costly resources and a much flatter team structure.  The team project manages themselves using scrum or other methodology, but this requires additional skills.


Building: No need with python?

Yes, building is needed with python.  but in one special case it can be hidden in the background. This page provides background on building, both with python and kotlin.

It can seem that python programs do not even need building, but the reality is some form of build needed with any program.  The good part of building in python, is building as you develop is so simple you don’t really notice.  The negative is if you later want to package up what you have developed, you may be confronted with one of a wide range of different builds options some of which can be quite complex.

Building Introduction Topics:

  • What is Building?
  • The Two Ways to Build: Environment vs Package Build
    1. Environment Build
    2. Package Build
  • Building in Python
  • Building in Kotlin
  • Conclusion: The different focus of Python and Kotlin

What is Building?

A Definition: Building is the process of putting in place all the components needed for a program, and proving the code with location information for each component.

The components are the resources needed by the program, such as program code for library functions, and images that may be used by the program.

Developing in python, it can seem like there is no such step as ‘building’.  Just type ‘python ‘ and python will do all that is required.  The reality is that this is an environment build approach, and during development any new libraries or other resources are added to the environment. So the steps to prepare the environment happen over time and can be forgotten. It is only when there is a need to run the program on computer other than the development computer, and all the environment install steps need to be repeated, that the environment build gets noticed.

Note also, that reliance on this approach limits the use of python.

In fact while this is true for the simplest way of running python, it is not always true.  That is because the simplest way of running python relies heavily on an ‘environment build’.

The Two Ways to Build

Environment Build vs Package Build

An environment build is where all the resources needed for the program are installed in the environment prior to running the program, enabling different programs to share the same resources.

By contrast, package build is where the resources for the program are packaged inside the program file, ensuring each program is self contained and independent.

1. Environment Build.

For an environment build, it the environment around the program that is built. An environment build means running the program on a new computer will require an ‘install’. The install can either be a series of steps to be followed manually, or there can be an install program to perform those steps.

The advantages of environment build are:

  • no need to re-install resources that have been previously installed in the environment another program(s)
  • during development, each new resource can be installed as required and independently, resulting in only one build for the complete development cycle, spread out over the entire development cycle

Disadvantages of environment build:

  • distribution of the application requires either a manual install process, or building of an installer which can be an additional development step
  • different applications may want different versions of resources to be present in the environment which may cause complex conflicts that are difficult to identify and/or resolve

 2. Package Build

A package build is where the components are combined in a single runnable ‘package’ file containing the components required by the program.

Advantages of package build:

  • distribution of the application can be as simple as just one file
  • each application contains its own components so no version conflicts can occur

java.jar files use a pure package build approach, which does require a build before every run, but means a fully portable self contained program is readily available without extra steps.

Building in Python

Building in python is very different for different uses of python.

Developer is the end user: One of the most common situations with python is the SEAAS developer, who is both the developer of the program and the end user of program. The resources needed by the program are installed as the program is built so there never seems to be a build at all.

Building for python.py files mostly becomes installing python, then ensuring all the imports have all the packages those imports require installed. If the imports all work, the environment for the python.py program has been built.  Once the environment is built, the program can easily be run without further steps.  The disadvantage of the this ‘build the environment’ approach is that for each new version of python the environment to satisfy the all python programs must be rebuilt, or alternatively a virtual environment approach must be adopted for development, and deployment can be problematic.

Web Server: Even if a web server is viewed by millions of people, the application only needs to be installed once per server, and it can be installed by the developer or other very skilled people.  Manual install of the environment for the python code, even if there are several steps, is entirely practical and is done by the developer.

Mobile Applications: Kivy is probably the best mobile framework for python, but like any other mobile app, Kivy apps, like all mobile apps, must be fully packaged.  Building a packaging is not part of standard python workflow, so the specifics of packaging kivy must be learnt and is quite is a complex and often fragile process.

Sharing with other friend/colleague Python Programmers (SEAAS): Simply send the source file, the other programmer probably already has python (probably the right version?) installed.

With simple python programs, there are no other ‘parts’, just python itself and just the one file.  As programs grow, there are imports, and then libraries to install to enable more imports. If the person installing knows pip install, they will quickly satisfy other requirements.  No build by the developer, manual environment build by the end user that is quite straightforward provided the end user can develop in python.

Sharing libraries and/or applications on PyPI: PyPI (or the “cheeseshop”) allows sharing open source python projects ready for simple installation into any computers python environment with a simple ‘pip install’.

Windows / MacOS applications: To distribute an application, to be installed and run by someone who need not know how to program in python, there are a variety of solutions. py2exe and pyinstaller and examples, and in each case the process is equivalent to a build that automates the install for regular users of the program.

Building in Kotlin

The bad new: There is always a build process.

The good news: There are build processes that work for all scenarios and are highly refined.

First, on that bad news.  Even the kotlin “hello world” program needs a build.  The ‘hello world’ program must call a print function, that has to be connected with the program code.  With python, the print function is in the standard library, which was installed in the python environment when python was installed so the developer needs no further build, unless that developer wants to send the program to a friend as a “helloexe” type file……. which would of course need a build.  However for “hello world” , having a “hello.exe” is not needed so you do not notice.

With kotlin, you are going to get the equivalent of “hello.exe”, so you there will be a build. The “hello” code must be built into a file together with the “print” function to be a complete program, and that requires a build.  All kotlin programs are made ready for installation on a computer, without first installing kotlin on any computer which will run the program.  Either as the “hello.exe” with kotlin native, “hello.jar” with the java version or “hello.js” to run in a browser.  Kotlin offers that choice (a level of choice not matched by python) but always requires a build.

Secondly, on the good news: The build is very simple. Because every developer is doing builds all the time, building is highly evolved.  Building can seem complex, because it is very flexible, and to learn all that can be done with build is still huge.  But to learn simple builds, equivalent to the examples for python given above is very easy.  The trap to avoid is the trap of trying to learn all that can be done with builds before getting started.

Conclusion: The different focus of kotlin and python.

Note: What applies here for python, also applies for many other ‘dynamic’ languages.

For someone writing programs for their own use, or for their colleagues who can also program in python, python provides a virtually ‘build free’ development cycle. Building can be virtually ignored.

For web servers, where millions can use the site build in python and installed just once, python also provides a close to a build free experience, and this time even for those who program as a profession.

For applications or mobile apps to be distributed for the mass market, and particularly apps/application by professionals to earn revenue, the build free experience is no longer available and building then can be complex and more difficult than with kotlin.

Python: With respect to building, python is at its best for programs written to be used by the developer and other programming colleagues, or for web servers or web services.

All kotlin programs require building. For “hello world” it can be so automatic you will not notice, but you will notice soon, even if only building programs for you own use.

However, the build systems available with kotlin are more consistent and more powerful. In fact, some notable large scale python projects like linkedin use kotlin ecosystem build systems. Despite the desirability of the kotlin approach, building in python is scattered that developers moving to kotlin can have their be the first ever exposure to build systems when using kotlin.  Note: in an ideal world, all python projects could use the same build system as other languages such as kotlin  (as is the case with linkedin).

There is a learning curve to building with kotlin, and that will be nice shallow curve if you keep it simple or a very steep curve if  try to learn the most advanced building possibilities at the start (when you don’t need them).

Kotlin: you can have one build system to build everything, including professional application, in place of the several different systems needed with python, and the power available is so compelling that advanced python users invest considerable resources to making the same system available with python.


Gradle with Intellij: Using build.gradle

When first opening a kotlin project that has a gradle configuration, the following two messages should appear:

  1.  Configure gradle in project – DO NOT DO THIS!!!
  2. Import gradle project – ONLY DO THIS ONE ONLY

The trap is that the first suggestion modifies the gradle.build file, and in fact our trials, this will normally break the build.gradle file.  If you have selected this, the only solution is to then undo those modifications made by IntelliJ and try again.

The import gradle may require setting the gradle home directory.  Set this to the location where you installed gradle. On a Mac, this is typically /usr/local/opt/gradle/libexec.


Gradle window (view/tool windows/gradle or gradle from right sidebar)


Note if the import gradle option action does not appear, check if the gradle window is already available, in which case import gradle has already taken place.  If not, try closing and reopening the project after checking that the build.gradle file is in the project root.

Once the import gradle is complete (it takes a while),  open the gradle window (view/tool windows/gradle or gradle in the right sidebar) and expand tasks/build and activate the ‘build‘ option under tasks/build.

It might be useful to then select the gradle build in the run debug toolbar (as shown below):


by clicking the dropdown, and selecting an option with the green gradle icon (as shown to the left of Unnamed above) before using the build build/build project or the run or debug symbols from the bar shown above.  Do not use either of these options before running.

Stack Memory and Local Variables

In todays languages, every call to a function adds new data to ‘the stack’ in the form of the local data for the function, and the value of where to return to in the program when the function completes.  To understand the stack, it can be useful to understand the problem the stack solves.

  • Subroutines
  • The problem:  PDP-8
  • The stack solves the problem
  • Parameters on the Stack
  • Local Variables on the Stack
  • Stack Overflow
  • Tracing the stack
  • conclusion


Before functions, there were subroutines.  A program was considered as a ‘routines’ and subroutines were effectively sub-programs.  Blocks of code that could be considered logically as a single instruction by the overall program.  Subroutines quickly evolved by the addition of parameters, local variables and then return values to be what we today generally call functions.  Some languages still have both subroutines (functions with no return value) and functions (with a return value), but many languages now simply think of all as functions.

The problem: PDP-8

The PDP-8 computer was designed with an instruction set that enabled ‘subroutines’ without using a stack.  The memory location immediately prior to the first instruction of the subroutine holds the return address, which means there can only be one return address.  The problem with only being able to hold one return address is, what if the subroutine has been call from two or more different places?  The second call will overwrite the return address for the first call.  Consider a routine such as factorial when implemented recursively.  For recursion to be possible, each call of the function must have its own return address.  Factorial 3 requires three calls, each with its own return address.

The stack Solves the problem.

So the stack was introduced as a concept to provide a new set of data for each call.  With a stack, three call allows three return addresses stacked on top of each other on a stack.

The PDP-8 call instruction does the following:

  • calculate return address as  the instruction following the current instruction
  • save the return address at the location called
  • set the program counter to the location following the location called (the location after where the return address was stored)
  • To return, the code jumps to value stored at the call location

A new stack based call would be as follows:

  • calculate return address as  the instruction following the current instruction
  • save the return address at the location of the stack pointer and subtract one from the stack pointer
  • set the program counter to the location called
  • to return, add one to the stack pointer and jump to the location stored at the location pointed to by the stack pointer.

The stack approach, requires the following:

  • An area of memory allocated as ‘the stack’
  • A different ‘call subroutine’ instruction performing the call steps above
  • A stack pointer (which is normally a register of the CPU), set to the base of the stack before any subroutine calls
  • A return instruction performing as described above


Parameters on the Stack

In the real world of modern programs, subroutines also accept parameters.  Consider the factorial equation, and the call requires the number for the factorial as a parameter. Again, it should be considered that there may be more than one call to the subroutine active, so separate storage is required for each call.   The stack again provides the solution. Simply push the parameter onto the stack prior to calling the routine, and within the routine the parameter can be accessed using an offset from the stack pointer.

Local Variables on the Stack.

The next concept was to add ‘local variables’.  From the beginning a subroutine could have a variable in global storage that only that subroutine made use of as a type of ‘local’ variable,  but again the problem of multiple copies of the subroutine would occur with several calls to the function all using the same memory locations for local variables. The solution it to allocate a block of ‘local variable storage’ when the subroutine is called.  At the start of the subroutine, the stack pointer is adjusted by the number of bytes required to reserve memory for local variables, and a reverse adjustment is required at the end of the subroutine.  The diagram here on Wikipedia illustrates a stack where there has been a subroutines call, and then that subroutines has also made a call.

Stack Overflow.

The amount of information on the stack has grown from the original return address.  With the return address, parameters and space for local variables, the amount of stack space used by any given routine can be significant.  Nesting, or the process of one routine calling another, grows the space required for the stack.  Too many levels of nesting, particularly with routines making use of a lot of local storage, and the stack can run out of room or ‘overflow’.  In reality most stack overflows occur as a result of infinite recursion.

Tracing the stack.

During debugging, tracing back through the stack can be the only way to understand complex problems, and a ‘stack trace’ or listing of the stack contents the only way to determine the exact state of a program.


The stack or ‘call stack’, has become an integral part of how functions and subroutines actually operate.  Understanding the call stack is critical to fully understanding how programs work.


Kotlin Limitations vs Python

social-media-failThis page will serve as repository of limitations found when attempting to move kotlin code to python code. Some limitations are a natural consequence of moving to static types, others are effectively inherited from Java.

The list here is based on placing those I find most significant at the top, and the further down the list the less significant.

base class parameter pass through


Kotlin, like python, allows for optional parameters, a feature not found in Java.  A consequence of this feature is that libraries with heavily used classes can over time evolve to have a large number of optional parameters in the most heavily used methods.  A larger number of parameters should only occur in code in a language that has optional parameters, as without the parameters being optional, every extra parameter  would introduce overhead on every call.  These methods with a large number of parameters are often frequently used methods, and the goal is that well chosen defaults will mean that most of these optional parameters are only provided in exceptional cases.  While a typical instance of the class may require only 2 or 3 parameters, while there may be as many as 20 to chose from.

To find examples, I simply thought: “what are the most widely used python packages I can think of?”.  The first three that occurred to me to check were sqlalchemy,  attr, and Django, and for each of these I looked for the most basic usage cases I could find. The first examples I found are:

  • sqlalchemy:   Table Class  20 optional parameters (see ‘parameters’)
  • attr: atrr.s() function – 9 optional parameters (see attr.s function)
  • Django: Field class – 22 optional parameters (see __init__)

While such a large number of parameters should not normally occur in Java due to lack of optional parameters, I think these example establish that the pattern is common in well regarded python packages.

The Problem.

Consider the Django Fieldclass . This class serves as a base class for several other classes, such as  TextField, TimeField, BinaryField etc.  To code examples like these cleanly, some type of language mechanism is needed which effectively achieves:  “include all fields from the base class constructor (or other method from the base class) as also parameters to this class unless specifically declared in this class“.

Python uses the *args,**kwargs system, which for several reasons is not an appropriate solution for kotlin, but is at least a solution.  There are some more elegant solutions possible for kotlin, and perhaps one will be added at a later time.

In Python the code for defining the BinaryField Class is as follows (not exact code for simplicity):

class BinaryField(Field):

    def __init__(self, *args, **kwargs):
        kwargs['editable'] = False
        super().__init__(*args, **kwargs)
        // other init code goes here

while in kotlin (slightly changed for name conventions and simplicity) the code becomes:

class BinaryField(
        verboseName:String?=null, name:String?=null, primaryKey:Boolean=false,
        maxLength:Int?=null, unique:Boolean=false, blank:Boolean=false,
        nulled:Boolean=false, dbIndex:Boolean=false, rel:String?=null,
        default:Any?=null, //editable:Boolean=true, - force 'false' for editable
        uniqueForYear:Int?=null, choices:String?=null, helpText:String="",
        dbColumn:Int?=null, dbTablespace:Int?=null, autoCreated:Boolean=false,
        validators:List<Any>?=null, errorMessages:String?=null
 ):Field(verboseName=verboseName, name=name, primaryKey=primaryKey,
        maxLength=maxLength, unique=unique, blank=blank, nulled=nulled,
        dbIndex=dbIndex, rel=rel, default=default, editable=false,
        serialize=serialize, uniqueForYear=uniqueForYear, choices=choices,
        helpText=helpText, dbColumn=dbColumn,
        dbTablespace=dbTablespace, autoCreated=autoCreated,
        validators=validators, errorMessages=errorMessages) {
  // class code here

Clearly, the call to the base constructor will be much shorter if not using named parameters, which is a choice, but in a list this long I would use named parameters.

The code (or almost identical code) will be repeated TextField, TimeField and the over 12 other fields that inherit from Field. Any update to the parameter list for the base Field class is tedious to say the least.

This is a case where kotlin requires boilerplate that is not required in python. What is needed is some way to say “accept all parameters to the base default constructor not specifically named in the base call, and pass these parameters through“. Given this problem will not occur in Java libraries which have no default parameters, it may be some time before kotlin designers consider this, if ever.  In the mean time, messy.

Constructor Calls

(to be added: by Nov 13)

Intricate, preset order, class data initialisation

Kotlin has what can appear a rather strange way of implementing the overriding of properties when extending classes.  The result is that when extending classes, the behaviour of initcode can be unexpected.  The way to avoid this is to use lazy properties in place of initialising during init.

open class Base{
    val simple = 3
    open val open = 3
    open val openGet = 3
    open val getter get()= 3
    open val getOpen get()= 3
    open val getDelg by GetDelg(3)
    init {
        println("base simple $simple open $open openG $openGet "+
                "getOpen $getOpen getter $getter "+
                " getDelg $getDelg")
    //open fun add(a:Int,b:Int) = a + b

class SubClass:Base(){
    override val open = 4
    override val openGet get()= 4
    override val getter get() = 4
    override val getOpen = 4
    //override val getDelg by GetDelg(4)  //uncomment for null pointer
    init {
        println("sub simple $simple open $open openG $openGet "+
                "getOpen $getOpen getter $getter "+
                " getDelg $getDelg")

class GetDelg(val value:Int){
    operator fun getValue(thisRef: Any?, property: KProperty<*>): Int {
        return value

The print from instancing a SubClass object is:

base simple 3 open 0 openG 4 getOpen 0 getter 4 getDelg 3
sub simple 3 open 4 openG 4 getOpen 4 getter 4 getDelg 3


Open is 0 in the base class init because the code to set the value has not yet been run, but not it is not 3 as you would expect.

openG, a value overridden by a get() method, perhaps unexpected returns the override value in both init() methods

getOpen, a get() method in the base overridden by a simple initialisation, behaves as a simple initialised value overridden by a new initialise, which is to be unitialised in the base init() method

getter() , a get() method overidden by another get() method returns the override value as does openG

getDelg() actually generates a null pointer exception if called during  the base init() method, as the overridden value has not been calculated

Note: part of this behaviour is base on the fact that overridden properties are actually new distinct properties, so the do not inherit values from base class property which is still accessible via super.<name>.  This means, counterintuitively, that open in the base init() method, returns 0while super.open in the subclass init()will return 3

I will update with more on the use of lazy to avoid this issue, but the main point is to think carefully before initialising values in a base class that are open.

*args, **kwargs (to be added)

A specific use of *args and **kwargs has already been covered in base class parameter pass through.

Outside of that specific use, kotlin does have effective equivalents to most use cases, but may depend on the reflections library, which is not available on all platforms at this time.

vararg parameters capture the equivalent to a *args list, and allows for using the list for calling in a manner very similar to python.

callBy provide most of the functionality of **kwargs when used for calling functions, but some code is needed to map parameter names to KParameters.  A link to such code may be added to this page is someone asks 🙂

For cases where it is desired to capture parameters in map for, using a map actually makes better sense in every case I have found, but I will update this further if I find a case where this is not true.