Kotlin Limitations vs Python

social-media-failThis page will serve as repository of limitations found when attempting to move kotlin code to python code. Some limitations are a natural consequence of moving to static types, others are effectively inherited from Java.

The list here is based on placing those I find most significant at the top, and the further down the list the less significant.

base class parameter pass through

background

Kotlin, like python, allows for optional parameters, a feature not found in Java.  A consequence of this feature is that libraries with heavily used classes can over time evolve to have a large number of optional parameters in the most heavily used methods.  A larger number of parameters should only occur in code in a language that has optional parameters, as without the parameters being optional, every extra parameter  would introduce overhead on every call.  These methods with a large number of parameters are often frequently used methods, and the goal is that well chosen defaults will mean that most of these optional parameters are only provided in exceptional cases.  While a typical instance of the class may require only 2 or 3 parameters, while there may be as many as 20 to chose from.

To find examples, I simply thought: “what are the most widely used python packages I can think of?”.  The first three that occurred to me to check were sqlalchemy,  attr, and Django, and for each of these I looked for the most basic usage cases I could find. The first examples I found are:

  • sqlalchemy:   Table Class  20 optional parameters (see ‘parameters’)
  • attr: atrr.s() function – 9 optional parameters (see attr.s function)
  • Django: Field class – 22 optional parameters (see __init__)

While such a large number of parameters should not normally occur in Java due to lack of optional parameters, I think these example establish that the pattern is common in well regarded python packages.

The Problem.

Consider the Django Fieldclass . This class serves as a base class for several other classes, such as  TextField, TimeField, BinaryField etc.  To code examples like these cleanly, some type of language mechanism is needed which effectively achieves:  “include all fields from the base class constructor (or other method from the base class) as also parameters to this class unless specifically declared in this class“.

Python uses the *args,**kwargs system, which for several reasons is not an appropriate solution for kotlin, but is at least a solution.  There are some more elegant solutions possible for kotlin, and perhaps one will be added at a later time.

In Python the code for defining the BinaryField Class is as follows (not exact code for simplicity):

class BinaryField(Field):

    def __init__(self, *args, **kwargs):
        kwargs['editable'] = False
        super().__init__(*args, **kwargs)
        // other init code goes here

while in kotlin (slightly changed for name conventions and simplicity) the code becomes:

class BinaryField(
        verboseName:String?=null, name:String?=null, primaryKey:Boolean=false,
        maxLength:Int?=null, unique:Boolean=false, blank:Boolean=false,
        nulled:Boolean=false, dbIndex:Boolean=false, rel:String?=null,
        default:Any?=null, //editable:Boolean=true, - force 'false' for editable
        serialize:Boolean=true,
        uniqueForYear:Int?=null, choices:String?=null, helpText:String="",
        dbColumn:Int?=null, dbTablespace:Int?=null, autoCreated:Boolean=false,
        validators:List<Any>?=null, errorMessages:String?=null
 ):Field(verboseName=verboseName, name=name, primaryKey=primaryKey,
        maxLength=maxLength, unique=unique, blank=blank, nulled=nulled,
        dbIndex=dbIndex, rel=rel, default=default, editable=false,
        serialize=serialize, uniqueForYear=uniqueForYear, choices=choices,
        helpText=helpText, dbColumn=dbColumn,
        dbTablespace=dbTablespace, autoCreated=autoCreated,
        validators=validators, errorMessages=errorMessages) {
  // class code here
}

Clearly, the call to the base constructor will be much shorter if not using named parameters, which is a choice, but in a list this long I would use named parameters.

The code (or almost identical code) will be repeated TextField, TimeField and the over 12 other fields that inherit from Field. Any update to the parameter list for the base Field class is tedious to say the least.

This is a case where kotlin requires boilerplate that is not required in python. What is needed is some way to say “accept all parameters to the base default constructor not specifically named in the base call, and pass these parameters through“. Given this problem will not occur in Java libraries which have no default parameters, it may be some time before kotlin designers consider this, if ever.  In the mean time, messy.

Constructor Calls

(to be added: by Nov 13)

Intricate, preset order, class data initialisation

Kotlin has what can appear a rather strange way of implementing the overriding of properties when extending classes.  The result is that when extending classes, the behaviour of initcode can be unexpected.  The way to avoid this is to use lazy properties in place of initialising during init.

open class Base{
    val simple = 3
    open val open = 3
    open val openGet = 3
    open val getter get()= 3
    open val getOpen get()= 3
    open val getDelg by GetDelg(3)
    init {
        println("base simple $simple open $open openG $openGet "+
                "getOpen $getOpen getter $getter "+
                " getDelg $getDelg")
    }
    //open fun add(a:Int,b:Int) = a + b
}

class SubClass:Base(){
    override val open = 4
    override val openGet get()= 4
    override val getter get() = 4
    override val getOpen = 4
    //override val getDelg by GetDelg(4)  //uncomment for null pointer
    init {
        println("sub simple $simple open $open openG $openGet "+
                "getOpen $getOpen getter $getter "+
                " getDelg $getDelg")

    }
}
class GetDelg(val value:Int){
    operator fun getValue(thisRef: Any?, property: KProperty<*>): Int {
        return value
    }
}

The print from instancing a SubClass object is:

base simple 3 open 0 openG 4 getOpen 0 getter 4 getDelg 3
sub simple 3 open 4 openG 4 getOpen 4 getter 4 getDelg 3

 

Open is 0 in the base class init because the code to set the value has not yet been run, but not it is not 3 as you would expect.

openG, a value overridden by a get() method, perhaps unexpected returns the override value in both init() methods

getOpen, a get() method in the base overridden by a simple initialisation, behaves as a simple initialised value overridden by a new initialise, which is to be unitialised in the base init() method

getter() , a get() method overidden by another get() method returns the override value as does openG

getDelg() actually generates a null pointer exception if called during  the base init() method, as the overridden value has not been calculated

Note: part of this behaviour is base on the fact that overridden properties are actually new distinct properties, so the do not inherit values from base class property which is still accessible via super.<name>.  This means, counterintuitively, that open in the base init() method, returns 0while super.open in the subclass init()will return 3

I will update with more on the use of lazy to avoid this issue, but the main point is to think carefully before initialising values in a base class that are open.

*args, **kwargs (to be added)

A specific use of *args and **kwargs has already been covered in base class parameter pass through.

Outside of that specific use, kotlin does have effective equivalents to most use cases, but may depend on the reflections library, which is not available on all platforms at this time.

vararg parameters capture the equivalent to a *args list, and allows for using the list for calling in a manner very similar to python.

callBy provide most of the functionality of **kwargs when used for calling functions, but some code is needed to map parameter names to KParameters.  A link to such code may be added to this page is someone asks 🙂

For cases where it is desired to capture parameters in map for, using a map actually makes better sense in every case I have found, but I will update this further if I find a case where this is not true.

Advertisements

Kotlin beyond Android: A risk?

Support for a language is dependant on the popularity of that language.  The only possible risk in choosing kotlin is the project becoming an orphan as kotlin never gains sufficient traction in the relevant space.  Choosing any language on the assumption that popularity will increase is a gamble,  so understanding the popularity of kotlin now and are the prospects for future popularity is essential in understanding the risk of future kotlin support.

Depending on the language popularity indicator you reference, kotlin is either climbing the rankings or already very near to top.  Kotlin on Android is widely tipped to overtake Java as the number 1 language, and choosing the number 1 language cannot really be considered a risk.  Could choosing Kotlin for development other than Android still be considered a risk, given the positive signs for kotlin uptake?

Continue reading “Kotlin beyond Android: A risk?”

Kotlin: The newest cross platform mobile solution

present-boxSearch ‘cross platform mobile development tools’ and you will find a list of approximately 10 solutions, with one on the list (Codename One) allowing coding in Kotlin, although with limitations on the use of kotlin libraries.

Kotlin, new choice for cross platform native?

Now there is a new choice, Kotlin/Kotlin Native.  A combination of the officially supported Kotlin for Android together with Kotlin Native for iOS.

Kotlin is the main choice for native android development, but now developing an Android app with Kotlin opens the possibility of reusing code from the Android app in an iOS App.

Note: This is not a solution for building a completely reusable app, but for sharing all ‘business’ logic or other logic not directly calling the OS.

Further, while Android apps are developed in Intellij or Android Studio, Kotlin Native requires CLion, a different IDE.

This probably makes a single build process for both Android and iOS out of reach for now.

Why all the IDEs from Jetbrains are not also available in plugin form or standalone IDE  (the same licence price for either would be fine).  Allowing a single master IDE to run each development could even boost revenues for Jetbrains as well a convenience for developers.  Add the paid premium plugin for both Java and Python for example, rather than settling for the free Python module in Intellij to avoid needing two different IDEs?

Kotlin on CLion, the original reason for Kotlin Native?

So why have Kotlin Native with the CLion IDE, when Intellij would be a better base for Android developers looking to also use Kotlin for iOS?

Perhaps because the originally identified target market for Kotlin Native was C/C++ developers?  Consider, that placing Kotlin Native within the IDE for C developers, effectively promotes Kotlin Native to these developers.

The first target for Kotlin was Java developers, that is clear, as is the fact that it was Java Android developers who adopted the language far more readily than other Java developers, despite kotlin not being specifically marketed to this sub group of Java developers initially.

So perhaps the biggest group of Java developers who could move to Kotlin, have already made the move to Kotlin.  So what next for Kotlin to keep the momentum going? Well on Tiobe, the next major language is C/C++, logically target that developer base.   (Although I suggest it would be more relevant to consider the leading edge data which is better provided by pypl).

 

BCSE vs TPSE: Two software industries, two development philosophies

twins-1It is my observation that there is a clear division of software into two very different software industries, which I am defining as Data Processing/Information Technology (DP/IT) and Software Product Engineering (SPE).   While I feel most software can be categorised to almost entirely one or the other of these fields, there does also exist also some developments that incorporate elements of both.

I also suggest there are two very different philosophies of software development, and there is a different relationship between each industry and each software development philosophy.

The Two software industries

Why new definitions?

Software has evolved into quite different directions, and Technology Product Software has seen the most significant evolution.  The Constant Evolution of the industry overall also means an evolution of terminology, and almost every existing term has a variety of interpretations.  So rather than try and appropriate existing terms, I have created new acronyms.

I believe the software industry overall has now become sufficiently large to look at the components, and that the once completely dominant sector of Bespoke Corporate Software Engineering has seen the Technology Product Software Engineering branch of in evolution to the extent that this now could be seen as a different industry.

I am defining terms which I find most useful for 2017. Hopefully these definitions will maintain some relevance for at least a few years, and hopefully they are useful for at least the discussions on these pages.

Bespoke Corporate Software Engineering (BCPE)

Frequently run by IT departments within companies with perhaps a CIO (Chief Information Officer) as the head of the activity, this type of computing has also fits reasonably well with the label data processing.

If you consider a large bank or government department with its own software development team, it could seem that software in this sense is not really an industry, but rather an activity within banking, government or whatever. What has arisen as an industry in its own right is the professional services industry providing software development services as an outsource alternative to banks, government etc, having their own internal software teams. ‘In house’ BCPE can be seen as vertical integration and in reality part of this same industry.  This software services industry has large accounting firms, and a number of computer companies who historically sold mainframe computers (Unisys, IBM etc.)  as some of the largest players within the industry.

At one time virtually the entire software industry fit in this category, and I suspect this category is still the largest employer.  Key characteristics are as follows:

  • Software Development within a large non-technology corporation
  • Software assists corporation with internal functions
  • The software developed is not sold/provided by the corporation as a product
  • The IT Crowd as satirical example
  • Bespoke business software development for use within a single organisation
  • A single customer (sometimes by way of a committee) to accept software
  • Also includes outsourced software development for large non-technology companies
  • Industry revenue is professional service fees to external providers and salaries for corporate staff

Technology Product Software Engineering (TPSE)

While Bespoke Corporate Software is developed specifically for use within the corporation sponsoring the development, Technology Product Software is produced in order to sell copies of the software developed, or provide a product or service based on the software.  This Software is about Technology companies where the software is the product, and the main business, of the company.  Classic examples would be a software game to run on a phone, or Adobe Photoshop, newer web examples would be Google and Facebook. The software is produced to used by an entire marketplace, rather than delivered to the specifications agreed with one specific customer.  The financial success of Product software projects than projects developing bespoke software, but the potential rewards are far greater.  This is a newer industry than bespoke software, but growing rapidly and now of greater economic value than bespoke software, although almost certainly still a smaller employer in total numbers than bespoke software.  Characteristics:

The decision to develop the software is made by a business with software as  core activity of the business

  • Software development by a technology company:  the core business of company initiating the software development is software technology
  • Any assistance with internal business function becomes secondary to provision of the software to a number of customers
  • The software developed is sold/provided by the corporation as a product
  • The Internship as comedic example
  • The customer is a market segment, not a single business
  • Customers can independently select to use the software product or not
  • The overall software team is usually also responsible for product design
  • Industry is directly a result of the software produced by the company

The Core test: Is development initiated by a Tech company where software is a core business function?

For a bank, the core business is being a bank. Generally, software to assist the bank be a bank can be considered a ‘non-core’ business activity. This means that generally a bank can outsource software development, because it is non-core, while a software technology company would cease to be a software technology company if that company outsourced software development. Companies such as those the bank outsources to are also part of the bespoke software industry under this definition unless they initiate the development themselves.  Technology Product software under this definition requires it is the technology company that identifies the need, and funds building the product in order to then promote the product to the chosen market.

When worlds collide: BCSE/TPSE Crossover

Not all software can be categorised as 100% bespoke or 100% product.  Consider an internet banking application within a bank.  Such an application follows product software engineering in that the customer for the software cannot ‘approve’ the software in the same manner as is possible for internal bespoke software engineering.  Does the ‘customer’ of the software have a choice, or does being a customer of the bank remove the customer choice?  Certainly the bank customers can elect not the use the app, but these same customers are unlikely to be able to chose an alternative application.  The end result is to some degree a mix of bespoke and product software engineering.

Now consider the software inside a toaster which uses a microprocessor.  Does the customer choose the application? Or is it the non-software features which fully determine who chooses the product?  Perhaps a mix of the two?  Again there is some mix of both product and bespoke development to embedded software.  Could the company retain its own identify if the software development is outsourced?  If yes, then the software is non core to company identity so the bespoke software model will be the best fit.

Who Cares? What does it matter?

I suggest it is important for correct analysis to determine if there are two or more distinct sub-groups.  Consider the following:

In my fish tank, the average fish has 1.5 legs.

This perhaps confusing result comes from failing to consider goldfish and starfish are actually two separate groups which deserve their own statistics.

So, as these pages analyse data, recognising there are two different types of software development becomes significant, if the two are actually different.  It is my experience these two types of development are different.

I suggest that different skill sets, is part of the reason that India such a significant role in Bespoke Corporate Software Engineering while Silicon Valley USA plays such a significant role in Technology Product Software Engineering.

I have heard statements such as: “the trend is that all software development is being outsourced”.  As discussed elsewhere, if software is your core business, you cannot really outsource so this statement specifically applies to bespoke (BCSE), not to the technology product industry.  Assuming a trend will apply across all software can be misguided if different factors apply for two different sectors.

Understanding this difference could also be significant those seeking to work in software, who may need to understand what trends actually apply in their chosen sector.  Similarly colleges offering courses may need to understand which for which sector their potential graduates should acquire skills.

Two Competing Development Philosophies

Background.

There is a famous quote the on the world market being limited to about 5 computers, that at least sounds like at one time it was believable, even if not very astute. For those first few computers, even operating systems were bespoke software, so the software industry began with only bespoke software and all else came later.  No only was the first software bespoke, but with the only reference building physical constructions, for ‘building’ software was building a bespoke physical construction.

Construction project development model

Before any construction (for example, the construction of a bridge or building) commences, you need the design and the plans.  A construction has steps, similar to these:

  • collect design ideas
  • budget the project
  • finalise a design agreed by stakeholders
  • finalise budget of actual design
  • construct to match the design
  • ensure compliance of the construction to the design and regulatory requirements
  • open the construction for use
  • now only maintenance if performed unless a specific separate upgrade project is launched

The project can only be ‘open to the public’, after construction is finished.  Software can follow this same process, and with software is called ‘waterfall’.  Waterfall does have certain advantages:

  • it is very easy for non-software people to relate to
  • it easily adopts to the bespoke software projects
  • the closed nature of the project works with a single budget allocation
  • separation of steps may enable use of specialists for each step
    • more narrowly skilled and widely available ‘construction workers’ for the construction phase
  • can enable outsourcing of specific steps while still retaining design control

This construction project process can be particularly suited to bespoke development, but has also been used for some notable product developments, and sometimes adapted  with each product version or major release being considered an separate development with its own set of the ‘waterfall’ steps.

The final test for ”design and regulatory requirements”,  is generally know as UAT (User Acceptance Testing) when applied to software.

Although ‘waterfall’ is out of favour as not ‘agile’, if the design requirements are sufficiently concrete from the outset, then ‘agile’ may not be as necessary as for projects which are exploring new and less tried ideas.  In the end, all development is either single waterfall, or an series of iterations of the waterfall process.

What does finished mean? With the single waterfall type process, the software is finished at the end of the cycle when development has stopped.  But consider in todays world the reaction if a major product software vendor stops development on a software product.  For example, if Canonical announced that development of Ubuntu will stop as the product is complete.  For a product, stopping development does not signal the product is complete, but rather that the project should be considered dead.  For the Ubuntu example, people would look for an alternative product once it has been announced that at some future date work will stop.  This may be the opposite of bespoke software waterfall development where managers look forward to the day the development is ‘complete’.  Finished with software, can mean something entirely different than it does with construction projects.

Software is all design. A program is essentially a fully detailed set of instructions, specifying exactly the steps that must be performed.  This means there is no exact fully detailed specification other than the completed software. While a system design document can be created, an accurate design document is in a sense an overview of the program itself, just lacking detail.  The design documents of a construction project do not lack any details, and the ‘build’ phase is about turning the instructions into something physical.  The instructions of a computer program can be ‘built’ into the actual program by the computer itself in seconds,  the challenge is specifying all the details of what must be built.  This fundamentally changes the project to effectively all design, and design is what must be agreed before a construction project can commence.  So in essence, the full design cannot be pre-agreed and this leads to problems in construction project model.

Living organism project development

In some cases, software can be considered as more similar to a living organism than to  than to a construction project.  The steps become:

  • The project is conceived but is not really functional at that time
  • The project goes through an embryonic phase where it is incapable of any significant functionality
  • The project is then ‘born’ and has first contact with the outside world but that contact is best limited as the young project is nurtured
  • The project gains capabilities as it matures over time (like the ability to walk or talk) and learns from the environment
  • Development continues with milestones reached although exact milestones may seem somewhat artificial (birthdays vs releases)
  • It is at some time it is declared the project is mature, although the exact point of maturity may in some ways be arbitrary
  • The project continues to develop
  • At some point development slows the project gets old
  • The project may eventually die although its place may now be filled by its own offspring

Idioms can only run so far, but there is much overlap with software products.

The reality is that waterfall never goes away, it just becomes very short iterations of waterfall, and these short iterations mean every stage of the waterfall must be able to be run very quickly.  This has the following consequences:

  • Design and development become intertwined, so designer and builder must either be the same person or a very small team with both skills
  • A single comprehensive UAT style testing is impractical for all but ‘last’ waterfall cycle, and the ‘last’ cycle may never really exist.  A new concept in testing that can occur over and over in a short time is required.
  • The living organism allows the project to adapt in an ‘agile’ manner which may see the project differ from the original brief, which can be problematic for a bespoke development contract

Agile / Living organism model and product software

While the living organism model may better match what is required from development, the result can be a challenge for monetising software. While a series of major new product versions can provide points at which customers pay fees, continual product evolution requires either customers pay monthly/annual fees or another revenue model is required.  Distinct releases provide for points at which companies can charge for new versions.  Despite the need for new business models, product software has changed.  This is why Windows 10 is scheduled to continually evolve with updates, rather than be replaced by a new version of windows as has happened in the past.  Development tools from Jetbrains all follow monthly or annual licence fees, and Adobe creative cloud invites you to choose a plan rather than purchase a version of a product as in the past.

Agile / Living organism model and bespoke software

Again, the living organism model is a problem for the way projects are run financially. With bespoke software, the challenge of fitting the business model can be even greater than with products.  The result is that the while almost all bespoke developments declare they are agile, this ranges from “because we hotdesk” through to actually having some iterations during development, but very rarely to actually adopting a full living organism view of software.  There are government departments, with annual IT budgets with little variation and teams of permanently employed outsourced or internal staff building projects, but they still view every IT project as a separate construction project and do not see a these as part of an overall picture where they are funding resources to deliver the continued evolution a set of software continuing to their needs.

Conclusion

There are currently two software industries and two software methodologies.

However, the two industries are in place because of mindset and the pressure of business models, not different business needs.  There is a mindset that building software must be like building houses, and a failure to grasp that in essence software is just a detailed design.  Yes, for large projects, separating design and then translating that design to code is a task that can be done with lower cost programmers in place of designer/developers, but still evolution will see this decrease.  In the same way typists were lower cost than managers, but eventually the combination of technology and education automated typing to a level where managers now type,  programmers who are not also designing the code will likely go the way of the specialist typist.  The bespoke industry will decline, but t will still take some time.  This data from github suggests that the product software industry could be a small as 22% of the overall software industry. The decline is currently more in the wages for these roles and the pressure from low cost off shore developers than the slow move to increase seeing all projects like products.

The two design methodologies follow more from business models that have not adapted, rather than simply being genuine alternatives.  As the industry matures further expect software project to continue to move to the living organism model.  All programs need to continue to evolve to remain relevant, and the living organism model is the most appropriate development model.

Software is in fact like a living organism in that is needs evolve to continue to reproduce.  Virus software may reproduce through its own code, but other software is symbiotic and relies on being chosen by humans to multiply. If users are not happy with the software they do not install copies, so evolution means only liked software reproduces.

When will your project ‘grow up’ and require typesafe code?

originalThere is common belief that after an initial very agile development period,  “grown up projects should switch to a static typed language”.  This also raises the question, “are type-safe languages less suitable for early stage development?”  This page considers the evidence for these beliefs, plus considers if there is a benefit to starting dynamic and switching to static, what is the crossover point?

TL;DR? As usual, read the headings and choose only read beyond the heading when you choose.  But in summary, modern statically typed language, and specifically kotlin, are bringing forward the point where adoption is logical, to the point it can now be argued that any project that justifies its own repository, would benefit from starting out as a kotlin project. Continue reading “When will your project ‘grow up’ and require typesafe code?”