Waterfall or Agile?

This page describes both Waterfall and Agile software development philosophies. In reality developments are not 100% waterfall or agile, but a blend of the two.

This is a ‘back to basics’ look at the two philosophies. Separate pages will consider the implications, and necessary modifications, when each is applied to different bespoke software or product software projects.


  • Why yet another page?
  • waterfall: modelled on construction projects
    • We have an agreed design and the process to build this is well known.
  • agile: modelled on engineering design projects
    • no one has ever done this before
  • hybrids?
  • Budgeting
  • The Team

Why yet another page?

In short, to try to separate the ‘religion’ from this topic, and to step back from the ‘bespoke only’ or ‘product software only’ perspective in almost every other page on this topic I have found.

An example of the ‘religion’ attitude is an article here which suggests Apple computer has poor software quality because they use ‘waterfall’ rather than ‘agile’.  It is almost certain that Apple would use a methodology encapsulating some of both ‘agile’ and ‘waterfall’,  but the implication is that label ‘waterfall’ is itself sacrilege.  While a company (and even perhaps Apple) may not embrace sound software methodology, evaluation is more complex than simply a label.

The other issue is to take each concept, and envisage only a chosen adaptation for the authors field of software.  Not only are bespoke and product software almost polar opposites, but even within each category there are factors that affect ideal methodology.  So most pages compare only for a specific environment, which is great if all your projects match that description, but less helpful for those with different or varied projects.

Waterfall:  Modelled on constructions projects.

waterfallsThe process of building something (i.e. a building) has been practiced and refined over centuries.  The waterfall approach is simply taking this construction model, and applying the model to the building of software.  Using a gantt chart,  the steps look more like a cascade of waterfalls.  But if you view a cascade from the front (see pic) then you have the appearance of water falling from step to step.  The waterfall is from step to step, not all at once.

From xbsoftware.com

The principle of waterfall is that each step is completed before the next begins. A complete understanding of requirements should be in place before design starts.  ‘Building” or development, should not commence without a completed design. Once built, tests verify the building meets the design.  These are the same steps that have been proven to work for construction projects over centuries.

Agile: Modelled on engineering design.

While waterfall considers a software project as similar to a construction project, agile effectively considers the entire software project, as being an engineering design phase.

With engineering design, you have a problem, and the challenge is to find a solution.

If it is know that some existing design can be modified to provide a solution, then there is no engineering design required.

An example from the tangible world is the design phase for mass produced products. You have a team of designers who engineer a design, but although you may have a fixed budget and time frame, it is only at the end of the design phase that you know what the final product will look like.

The principles of agile can be considered be:

  • what is being created is new and innovative
  • new ideas will need prototyping and revision
  • the best solution is not always clear at the outset
  • actual product requirements may not even be clear until the product is trialled

The process of agile software development can be considered as:

  • define initial set of requirements
  • build an as simple as possible prototype as quickly as possible
  • now repeat the following steps until satisfied:
    • analyse what how the prototype could be improved, and add those improvements to requirements
    • analyse steps that can be made within 1-2 weeks to meet more requirements and perform those steps

Note: this is representing the agile philosophy itself. Any actual implementation of agile will also depend on some other choices as to how agile will be implemented.

Consider a car manufacturer who produces a new model of a certain car every year.  The design team, with an allocated number of people and resources, is given a fixed time to come up with next years design.  Agile development is similar to this process.

Hybrids? Solutions using a combination.

Waterfall: We have a specification and the process to build this is well known.

With construction projects, you could be building an exact replica in an effectively identical site, but surely there will be something new?  In fact, the more daring the building, the more often that some part of the project will actually encompass an engineering design project, even with prototypes!  The fact that almost every waterfall project encounters some steps which better fit the agile pattern, is the reason so many waterfall projects suffer time and cost blowouts.

Agile: No one has ever done this before.

But Surely someone has done part of it before?  Won’t there are least be construction projects to build the prototypes?

The reality is that both agile and waterfall can start to approach a series of short waterfalls placed back to back. Each just approaches from the opposite side.

A very short waterfall stage loses many of the characteristics of waterfall, and a long build of a single prototype becomes more waterfall.  In the end no project is ‘pure’ and all have elements of both.  Software is always doing something new, but just how new?

The more new, the more no one has genuinely done anything like this before, the greater the need for an agile mindset.  The more ‘just like a previous one but with these straight forward but time consuming changes‘, the more a waterfall mindset may work.


No one can predict the exact cost of an exactly specified system containing elements that no one has ever done before.

As waterfall is based on the construction model, the budgeting process it quite like getting quotes having a house built to match the plans you have had drawn up.  You can have faith you will get something that matches the plans, even if it may be a little late and the real problems only occur if you realise during construction that result will not match what you actually need.

Agile is more like working with the architect.  You may have a fixed quote but you have to commit blindly without knowing in advance if you will like the result.  With agile you can control the budget, but at the expense of accepting what is produced within that budget.

The result is that  many projects are forced into a waterfall mindset from the budgeting phase, and  can only live with an agile mindset for the limited spend of the design phase.

The Team

Just as waterfall breaks a project into steps, it separates the roles needed for those steps.  Design people are expensive, but during ‘construction’ you may only need labourers who cost less.  Similarly a waterfall methodology separates project managers, annalists, and programmers. Programmers who just program become lowest cost resource, and can be more easily outsourced and managed by project managers.

Agile requires the whole team act like engineers.  As the work is always different even the construction has less guidance and more decisions on the fly by the ‘engineer’.  The result is more costly resources and a much flatter team structure.  The team project manages themselves using scrum or other methodology, but this requires additional skills.


Gradle with Intellij: Using build.gradle

When first opening a kotlin project that has a gradle configuration, the following two messages should appear:

  1.  Configure gradle in project – DO NOT DO THIS!!!
  2. Import gradle project – ONLY DO THIS ONE ONLY

The trap is that the first suggestion modifies the gradle.build file, and in fact our trials, this will normally break the build.gradle file.  If you have selected this, the only solution is to then undo those modifications made by IntelliJ and try again.

The import gradle may require setting the gradle home directory.  Set this to the location where you installed gradle. On a Mac, this is typically /usr/local/opt/gradle/libexec.


Gradle window (view/tool windows/gradle or gradle from right sidebar)


Note if the import gradle option action does not appear, check if the gradle window is already available, in which case import gradle has already taken place.  If not, try closing and reopening the project after checking that the build.gradle file is in the project root.

Once the import gradle is complete (it takes a while),  open the gradle window (view/tool windows/gradle or gradle in the right sidebar) and expand tasks/build and activate the ‘build‘ option under tasks/build.

It might be useful to then select the gradle build in the run debug toolbar (as shown below):


by clicking the dropdown, and selecting an option with the green gradle icon (as shown to the left of Unnamed above) before using the build build/build project or the run or debug symbols from the bar shown above.  Do not use either of these options before running.

Stack Memory and Local Variables

In todays languages, every call to a function adds new data to ‘the stack’ in the form of the local data for the function, and the value of where to return to in the program when the function completes.  To understand the stack, it can be useful to understand the problem the stack solves.

  • Subroutines
  • The problem:  PDP-8
  • The stack solves the problem
  • Parameters on the Stack
  • Local Variables on the Stack
  • Stack Overflow
  • Tracing the stack
  • conclusion


Before functions, there were subroutines.  A program was considered as a ‘routines’ and subroutines were effectively sub-programs.  Blocks of code that could be considered logically as a single instruction by the overall program.  Subroutines quickly evolved by the addition of parameters, local variables and then return values to be what we today generally call functions.  Some languages still have both subroutines (functions with no return value) and functions (with a return value), but many languages now simply think of all as functions.

The problem: PDP-8

The PDP-8 computer was designed with an instruction set that enabled ‘subroutines’ without using a stack.  The memory location immediately prior to the first instruction of the subroutine holds the return address, which means there can only be one return address.  The problem with only being able to hold one return address is, what if the subroutine has been call from two or more different places?  The second call will overwrite the return address for the first call.  Consider a routine such as factorial when implemented recursively.  For recursion to be possible, each call of the function must have its own return address.  Factorial 3 requires three calls, each with its own return address.

The stack Solves the problem.

So the stack was introduced as a concept to provide a new set of data for each call.  With a stack, three call allows three return addresses stacked on top of each other on a stack.

The PDP-8 call instruction does the following:

  • calculate return address as  the instruction following the current instruction
  • save the return address at the location called
  • set the program counter to the location following the location called (the location after where the return address was stored)
  • To return, the code jumps to value stored at the call location

A new stack based call would be as follows:

  • calculate return address as  the instruction following the current instruction
  • save the return address at the location of the stack pointer and subtract one from the stack pointer
  • set the program counter to the location called
  • to return, add one to the stack pointer and jump to the location stored at the location pointed to by the stack pointer.

The stack approach, requires the following:

  • An area of memory allocated as ‘the stack’
  • A different ‘call subroutine’ instruction performing the call steps above
  • A stack pointer (which is normally a register of the CPU), set to the base of the stack before any subroutine calls
  • A return instruction performing as described above


Parameters on the Stack

In the real world of modern programs, subroutines also accept parameters.  Consider the factorial equation, and the call requires the number for the factorial as a parameter. Again, it should be considered that there may be more than one call to the subroutine active, so separate storage is required for each call.   The stack again provides the solution. Simply push the parameter onto the stack prior to calling the routine, and within the routine the parameter can be accessed using an offset from the stack pointer.

Local Variables on the Stack.

The next concept was to add ‘local variables’.  From the beginning a subroutine could have a variable in global storage that only that subroutine made use of as a type of ‘local’ variable,  but again the problem of multiple copies of the subroutine would occur with several calls to the function all using the same memory locations for local variables. The solution it to allocate a block of ‘local variable storage’ when the subroutine is called.  At the start of the subroutine, the stack pointer is adjusted by the number of bytes required to reserve memory for local variables, and a reverse adjustment is required at the end of the subroutine.  The diagram here on Wikipedia illustrates a stack where there has been a subroutines call, and then that subroutines has also made a call.

Stack Overflow.

The amount of information on the stack has grown from the original return address.  With the return address, parameters and space for local variables, the amount of stack space used by any given routine can be significant.  Nesting, or the process of one routine calling another, grows the space required for the stack.  Too many levels of nesting, particularly with routines making use of a lot of local storage, and the stack can run out of room or ‘overflow’.  In reality most stack overflows occur as a result of infinite recursion.

Tracing the stack.

During debugging, tracing back through the stack can be the only way to understand complex problems, and a ‘stack trace’ or listing of the stack contents the only way to determine the exact state of a program.


The stack or ‘call stack’, has become an integral part of how functions and subroutines actually operate.  Understanding the call stack is critical to fully understanding how programs work.


Kotlin Limitations vs Python

social-media-failThis page will serve as repository of limitations found when attempting to move kotlin code to python code. Some limitations are a natural consequence of moving to static types, others are effectively inherited from Java.

The list here is based on placing those I find most significant at the top, and the further down the list the less significant.

base class parameter pass through


Kotlin, like python, allows for optional parameters, a feature not found in Java.  A consequence of this feature is that libraries with heavily used classes can over time evolve to have a large number of optional parameters in the most heavily used methods.  A larger number of parameters should only occur in code in a language that has optional parameters, as without the parameters being optional, every extra parameter  would introduce overhead on every call.  These methods with a large number of parameters are often frequently used methods, and the goal is that well chosen defaults will mean that most of these optional parameters are only provided in exceptional cases.  While a typical instance of the class may require only 2 or 3 parameters, while there may be as many as 20 to chose from.

To find examples, I simply thought: “what are the most widely used python packages I can think of?”.  The first three that occurred to me to check were sqlalchemy,  attr, and Django, and for each of these I looked for the most basic usage cases I could find. The first examples I found are:

  • sqlalchemy:   Table Class  20 optional parameters (see ‘parameters’)
  • attr: atrr.s() function – 9 optional parameters (see attr.s function)
  • Django: Field class – 22 optional parameters (see __init__)

While such a large number of parameters should not normally occur in Java due to lack of optional parameters, I think these example establish that the pattern is common in well regarded python packages.

The Problem.

Consider the Django Fieldclass . This class serves as a base class for several other classes, such as  TextField, TimeField, BinaryField etc.  To code examples like these cleanly, some type of language mechanism is needed which effectively achieves:  “include all fields from the base class constructor (or other method from the base class) as also parameters to this class unless specifically declared in this class“.

Python uses the *args,**kwargs system, which for several reasons is not an appropriate solution for kotlin, but is at least a solution.  There are some more elegant solutions possible for kotlin, and perhaps one will be added at a later time.

In Python the code for defining the BinaryField Class is as follows (not exact code for simplicity):

class BinaryField(Field):

    def __init__(self, *args, **kwargs):
        kwargs['editable'] = False
        super().__init__(*args, **kwargs)
        // other init code goes here

while in kotlin (slightly changed for name conventions and simplicity) the code becomes:

class BinaryField(
        verboseName:String?=null, name:String?=null, primaryKey:Boolean=false,
        maxLength:Int?=null, unique:Boolean=false, blank:Boolean=false,
        nulled:Boolean=false, dbIndex:Boolean=false, rel:String?=null,
        default:Any?=null, //editable:Boolean=true, - force 'false' for editable
        uniqueForYear:Int?=null, choices:String?=null, helpText:String="",
        dbColumn:Int?=null, dbTablespace:Int?=null, autoCreated:Boolean=false,
        validators:List<Any>?=null, errorMessages:String?=null
 ):Field(verboseName=verboseName, name=name, primaryKey=primaryKey,
        maxLength=maxLength, unique=unique, blank=blank, nulled=nulled,
        dbIndex=dbIndex, rel=rel, default=default, editable=false,
        serialize=serialize, uniqueForYear=uniqueForYear, choices=choices,
        helpText=helpText, dbColumn=dbColumn,
        dbTablespace=dbTablespace, autoCreated=autoCreated,
        validators=validators, errorMessages=errorMessages) {
  // class code here

Clearly, the call to the base constructor will be much shorter if not using named parameters, which is a choice, but in a list this long I would use named parameters.

The code (or almost identical code) will be repeated TextField, TimeField and the over 12 other fields that inherit from Field. Any update to the parameter list for the base Field class is tedious to say the least.

This is a case where kotlin requires boilerplate that is not required in python. What is needed is some way to say “accept all parameters to the base default constructor not specifically named in the base call, and pass these parameters through“. Given this problem will not occur in Java libraries which have no default parameters, it may be some time before kotlin designers consider this, if ever.  In the mean time, messy.

Constructor Calls

(to be added: by Nov 13)

Intricate, preset order, class data initialisation

Kotlin has what can appear a rather strange way of implementing the overriding of properties when extending classes.  The result is that when extending classes, the behaviour of initcode can be unexpected.  The way to avoid this is to use lazy properties in place of initialising during init.

open class Base{
    val simple = 3
    open val open = 3
    open val openGet = 3
    open val getter get()= 3
    open val getOpen get()= 3
    open val getDelg by GetDelg(3)
    init {
        println("base simple $simple open $open openG $openGet "+
                "getOpen $getOpen getter $getter "+
                " getDelg $getDelg")
    //open fun add(a:Int,b:Int) = a + b

class SubClass:Base(){
    override val open = 4
    override val openGet get()= 4
    override val getter get() = 4
    override val getOpen = 4
    //override val getDelg by GetDelg(4)  //uncomment for null pointer
    init {
        println("sub simple $simple open $open openG $openGet "+
                "getOpen $getOpen getter $getter "+
                " getDelg $getDelg")

class GetDelg(val value:Int){
    operator fun getValue(thisRef: Any?, property: KProperty<*>): Int {
        return value

The print from instancing a SubClass object is:

base simple 3 open 0 openG 4 getOpen 0 getter 4 getDelg 3
sub simple 3 open 4 openG 4 getOpen 4 getter 4 getDelg 3


Open is 0 in the base class init because the code to set the value has not yet been run, but not it is not 3 as you would expect.

openG, a value overridden by a get() method, perhaps unexpected returns the override value in both init() methods

getOpen, a get() method in the base overridden by a simple initialisation, behaves as a simple initialised value overridden by a new initialise, which is to be unitialised in the base init() method

getter() , a get() method overidden by another get() method returns the override value as does openG

getDelg() actually generates a null pointer exception if called during  the base init() method, as the overridden value has not been calculated

Note: part of this behaviour is base on the fact that overridden properties are actually new distinct properties, so the do not inherit values from base class property which is still accessible via super.<name>.  This means, counterintuitively, that open in the base init() method, returns 0while super.open in the subclass init()will return 3

I will update with more on the use of lazy to avoid this issue, but the main point is to think carefully before initialising values in a base class that are open.

*args, **kwargs (to be added)

A specific use of *args and **kwargs has already been covered in base class parameter pass through.

Outside of that specific use, kotlin does have effective equivalents to most use cases, but may depend on the reflections library, which is not available on all platforms at this time.

vararg parameters capture the equivalent to a *args list, and allows for using the list for calling in a manner very similar to python.

callBy provide most of the functionality of **kwargs when used for calling functions, but some code is needed to map parameter names to KParameters.  A link to such code may be added to this page is someone asks 🙂

For cases where it is desired to capture parameters in map for, using a map actually makes better sense in every case I have found, but I will update this further if I find a case where this is not true.

Kotlin beyond Android: A risk?

Support for a language is dependant on the popularity of that language.  The only possible risk in choosing kotlin is the project becoming an orphan as kotlin never gains sufficient traction in the relevant space.  Choosing any language on the assumption that popularity will increase is a gamble,  so understanding the popularity of kotlin now and are the prospects for future popularity is essential in understanding the risk of future kotlin support.

Depending on the language popularity indicator you reference, kotlin is either climbing the rankings or already very near to top.  Kotlin on Android is widely tipped to overtake Java as the number 1 language, and choosing the number 1 language cannot really be considered a risk.  Could choosing Kotlin for development other than Android still be considered a risk, given the positive signs for kotlin uptake?

Continue reading “Kotlin beyond Android: A risk?”