Multi Platform

Kotlin Supports three overall platforms, with Native itself effectively being multi platform, producing even more options:

  • JVM
  • Javascript
  • Kotlin Native
    • ios
    • android
    • windows etc

Some code can is platform independent, and can be run across all platforms.  There is also code which is platform dependant, so strategies have to deal with a mix of both cases.  These pages describe how:

There is no equivalent to this for Python, and either the entire Python platform is ported or is not available.  While R-Python can produce native code, use outside the pypy project is actually discouraged.

Generally, this entire section is for cases where Python is not even an option, so these pages are not specific to moving from kotlin to python.

 

 

Advertisements

DSL Methodology: A key software concept.

With Kotlin the term ‘DSL’ has taken on a specific meaning, and that more specific meaning is explored in another page on Kotlin DSLs like kotlinx.html and how to write them.  DSL methodology is a key reason that the capabilities of a language to write domain specific extensions to the language becomes important.

This page concentrates on the concept of DSL Methodology.  DSL methodology is to consider software development as the task of creating the component tools that allow the expression of what the program does within a single concise function, and in a manner the not only executes correctly but is also easy for a person to read, understand and when necessary, modify.

  • Introduction to DSLs
    • DLS: the general DSL definition
    • DSL: independent language or and language extension?
    • When does a program create a DSL?
  • Human Language and DSLs
    • Why consider human languages?
    • The Dictionary is an insufficient reference.
    • Jargon as a form of DSL
    • Situation Specific Language extensions
    • Specific Human language: Independent Language vs Extensions.
    • Conclusion
  • DSL Methodology: The Basics
    • Simple language extension
    • DSL Methodology: Building Blocks
    • More Layers
    • The Core concept
    • When to apply DSL Methodology
  • Implementing DSL Methodology
    • The Core concept
    • software modules: group extensions together
    • Leveraging existing Language Extensions
  • Conclusion

Introduction

DSL: The general definition

A DSL is an acronym for Domain Specific Language.  A ‘Domain’ being effectively a specific application. Contrasting with ‘General purpose languages’ which attempt to allow for solving any programming problem, a DSL can be purpose designed for a specific ‘domain’ or a specific type of problem.

DSLs can sound like using Kotlin (or Python) to build an entirely new language, and this can seem true, but the reality is simpler. All kotlin DSLs are extensions of Kotlin, and although sometimes use cases can focus on the extension and make little use of the underlying language, that underlying language is always still there.

Two Types of DSL: Independent DSLs and Language (extensions) DSLs

Independent DSLs

There are DSLs like SQL, which are stand-alone languages to tackle a specific field or domain.  The first problem with independent DSLs is that a general purpose language is almost always also required.  This problem can be solved by combining a DSL with a general language, like using SQL from Python, although then you have two different languages, it does work.  The second problem with independent DSLs is that the features of the general purpose language are not accessible from within the DSL, so the DSL has to duplicate features of general purpose languages, and the duplicated features are generally inferior to those in general purpose languages.  E.g. numeric expressions in SQL are not as powerful as most general purpose languages, and there is often a syntax change from the general purpose language.

Language (extended) DSLs.

There is also a solution where a DSL is an extension to a general purpose language. For example, in place of developing in SQL with Python, SQLAlchemy brings the power of the SQL language to Python.

When we use the term ‘Kotlin DSL’ or even ‘Python DSL’, we mean a DSL made by extending Kotlin or Python with extra ‘vocabulary’ for domain specific features.  The DSL is a set of new language which extends an existing language.  This type of DSL is seen as the preferred solution. In programming, with an extension, there is less syntax to learn and greater consistency if, in place of a new independent language, extra ‘vocabulary’ for a language we already know can be used.

When does a program create a DSL?

When does adding new features to an existing language, create a new DSL? As with many things, the extremes are evident. The “hello world” program would be regarded not to create a DSL, and at the other extreme, some packages clearly implement a DSL. However, if you had half the features, would it still constitute a DSL? At what point does adding additional ‘vocabulary’ reach the definition of being a DSL?

The reality is that every program, including “hello world”, creates some new vocabulary somewhere, just in the case of “hello world” that vocabulary is not very useful. However, if the program is called “hello”, then the computer gains new syntax in ‘the shell’ such that typing ‘hello‘ now does something, and prints “Hello world”. The shell gains “hello” as one new word of vocabulary. While one word is not much of a language extension, the concept of extending a language is common throughout programming, so the principle of building a DSL always applies at some level.

Human language and DSLs

Why consider human languages?

A significant part of the human brain has evolved specifically to process language.  Since the first move from machine code to assembler, the goal has been for computer programs also to be processed by humans.  Just how do our brains handle domain specific language?  Don’t we take years to learn one single language and find learning another quite difficult?

Jargon as a DSL equivalent

One spoken language parallel to a computer extended DSL is ‘jargon’. Many ‘domains’ evolve their jargon or extensions to the language. Jargon more concisely and more specifically communicates the concepts needed in specific a domain than regular ‘non-jargon’ language does, but is generally used in combination with an underlying general-purpose language, such as English or French or Chinese. People who already speak a general purpose language can learn one or more jargon vocabularies in a much shorter time than learning the general-purpose language.

The dictionary is an insufficient model.

If you are reading this document, it can be assumed you can read English, and the reference for English is the dictionary.  But there may be words even on this page, where the dictionary is actually not that helpful, because the dictionary does cover what words part of the language, but not how they combine to provide meaning, or even a full understanding of the concepts behind the meaning.  Wikipedia can be a useful source of far more information behind the words, but there are also words that change with context.  All of this means the language becomes more like the syntax for expressing meaning, but it can take a lot of language to actually convey meaning.  Just as Python or Kotlin have a language syntax, but language extensions are built within that base syntax.  To understand the meaning of what is written, terms can become familiar to us, but until there are familiar we may have to go to the dictionary, the encyclopedia, or for context specific things like where something is, we may have to ask.  All of this is the paralleled in code but reading the definition of an object or function, but once we are familiar we should not need keep referring to that source.

Situation Specific Language extensions.

However humans also learn far more localised and situation specific language extensions.
Consider a random page from a novel. Most of the words can be found in the appropriate language dictionary, or an encyclopedia, because the are part of the general language (e.g. English). But there are words that are not sufficiently explained by either dictionary or encyclopedia, because they have  a specific meaning in the context of the novel, and the meanings are explained through the novel.  Names are one class of such words. Names can make reading a page at random a challenge. Read a random page from a novel and we skip the explanation of the specific meaning.  Just who is ‘Harry’ or ‘Sally’? Have they met?  What is their relation to the protagonist? Novels are designed to be read sequentially, so there is no index to easily find what has been defined, and usually no clear list of what is defined.  Depending on the novel, significant amounts can be specific to the novel.  Consider Lord of the Rings.  Not only are characters explored, but also types of creatures, new locations and imaginary world.  It can be described as a “Lord of the Rings Universe” being created.

So even to navigate an individual literary work, a new extended vocabulary can be required, varying from knowledge of just a few character names through to an entire altered universe.

Specific Human language: Independent Language vs Extension.

While jargon can seem impossible to understand for a ‘layman’ who only speaks regular ‘non-jargon’ language. The reality is, jargon normally does not create a replacement for regular language and communication in jargon alone is rarely sufficient. Even very domain specific communication requires a mixture of regular language and jargon together.

Imagine, for example, a French person and a Chinese person who are both from the same industry and use the same jargon, but other than that jargon are unable to communicate. Even adding words like ‘very’ to the jargon would be a problem. Like independent DSLs, jargon needs to coexist with a general purpose language.
Just like the French and Chinese colleagues, every independent DSL also needs some ‘normal words’ so they add their own limited set of ‘normal words’. This means Independent DSLs have to revisit many things already present in general programming languages, and the result is still restrictive.
The French and Chinese hypothetical colleagues would be far better placed if they both spoke a common regular language in addition to just the jargon.

Conclusion

Human communication in human language actually relies on language extension using a base set of rules.  This suggests that our thinking should be well adapted to the same approach within programs.

DSL Methodology: The basics

Simple Language extension.

Language extension is the core programming concept of defining things.  Even a variable definition is defining a new language element. For more significant language extensions in blocks, in python we have ‘import’ and to give increased scope of what can be imported there is ‘pip install’.

DSL methodology: building blocks

It is generally agreed that there is a maximum number of lines for a well defined function. Opinions on the actual limit vary,  but generally the recommendation range from that which can be seen on the screen at one time, through to as high as around 100 lines.

Now consider that, every program is described by a single function, usually called ‘main‘.  With a very simple program, all the code could be held in main, but as the code grows, that limit of  around 25 to 100 lines in one function will become a restraint.

How to describe the program in the main function, and keep main small enough to read and understand?

As the program grows, the developer can move some code to ‘other functions’ and in main simply call these functions containing the moved code This is one way the size of main can be controlled.

But simply moving ‘chunks’ of main into functions is not DSL methodology. DSL methodology is to create buildings blocks functions that allow writing the logic of main in a more concise way.  The logic of main stays in main, it is only logic to turn steps into extensions of the language that is moved to the functions.

Main stays readable if the concepts of the functions are clear.  Usually any  functions with ‘moved code’ will need to be generalised to convert them into building blocks, and the individual application specific nature come from parameters specified when those blocks are called. Then understanding what the program does can still be clear just by considering that main function.  A new developer may not need to read beyond main to learn, to understand what the program does, and may limit going beyond main to an area of functionality of particular interest, and infer the meaning of other new ‘vocabulary’.

More Levels

The concept is that main describes the program at the top level, but main will need the use of either program specific building blocks or ‘extended language’ and/or  language building blocks, know as packages, which are common to several applications.  In the case of “hello world”, the only other function is the print function, and that function is considered part of the language.  If you know the python language, you know how the python print works.  However if the other functions  beyond main are the ‘moved code’ described above, then these other functions will most likely be unique.  They are an extension to the language for the use of this main function, and unique to this main function.   While the function names can convey what is done at a high level, to know exactly what these functions do, a person reading the program will need to go and then read these functions.  To read main, this is the equivalent to looking up a word in the dictionary.

As the solution grows in detail, in turn these functions become complex and will also require their own extensions to stay within size constraints. As the program system grows the number of levels of extension to the original language grows.

The Core concept.

The core concept, is that the end result is the main is written using an extended language, and those extensions are build on other extensions. Each level  should be readable without looking up what each component of the new language means in full detail.   Each level is written in terms of an underlying extended language, and the program should be broken up into components that define new language blocks or ‘jargons’, which are separate from layers build using those jargon language blocks.

The role of each level or block is to provide the language extensions to make the level above simple to understand. The lowest level of the program is the only level of the program written in python or kotlin or whatever itself, as all other levels are built on the extensions.

A web server will rarely be built in ‘raw’ python alone, but will normally be built on a software stack of template engines, routing engines, database engines etc.  The language of the project becomes not just python, but python plus all those extensions.  Then the project may add its own extensions.

But to work in the project, you have to learn the language of each of the extensions, or at least the language of the extensions being used in the area of the project you are working.

Every project of any scale is not just built on a language, but on the language plus the set of extensions to that language.

When to apply DSL methodology?

It follows from the concept of ‘the main function describes the program’ that a very simple program such as hello world already achieves the goal of main being conveying what the program does. In fact any project simple enough to be contained in a single file, or  unlikely to require changes beyond a month from when the program is written, is too small scale or time frame to benefit significantly from DSL methodology,

The main relevance of DSL methodology is for long term projects with continued updates, developed by a team producing several releases over a time scale of more than one year.

It is with this type of software that reading what the code does can become a challenge even to the author of that block of code over time.

Implementing DSL methodology

the goals

DSL methodology is simply a slightly different way to view the normal principles of sound software development. The steps to implementing are all striving to achieve these goals.

The goals DSL methodology are:

  • allow each part of the system to be expressed in the simplest language possible and with the smallest possible language extension
  • keep system specific functionality at the highest layer possible, and avoid buried functionality
  • building blocks should be as generic as possible and able to be understood without considering the overall application

software modules:  group extensions together

Building blocks should where possible be grouped in to logical modules that together provide a specific type of new functionality.  In fact these modules should be considered to have functionality independent of the central application, and be able to have their own documentation, and perhaps own repository.

In fact, given that such extensions should not contain the logic of the main application, it may be possible to open source these extensions, even where the main application itself would not be open source.

Leveraging existing Language Extensions

Most often best described as packages, there are readymade language extensions which can give great capabilities.  Selection of these packages becomes very important as each selection becomes additional ‘extended language’ that the team must become familiar with.

The real world is, that beyond well established quite generic packages, there are many packages that takes steps towards keeping your code simple, but in the end in your usage still leave a program to big to maintain.  There choices then are:

  • extend an existing package
  • build a new package that uses the existing package internally but exposes only a new api
  • use the existing package as inspiration and effectively fork
  • build an alternative package

Conclusion

The end goal is to have a set of smaller packages, some internal modules to the application, some as separate packages perhaps even with their own lifecycle, together with the smallest possible core application.

Methodologies

Team members come in with greatly varied experiences.  This section provides some background to discussions that ‘old hands’ at coding may find all too familiar.  But for some team members this may be useful.  Please add comments if things added or changed here would make it useful to you.

Kotlin multi-platform projects structure

Previously, we talked about how to share code across platforms in kotlin world. But as your code base grows, you will encounter a scenario where some code in the projects is cross, but some other is platform specific, so will not be shared and there may be a version of this code for any or all platforms. Things like separating the platform specific implementation to another repo in order to reuse them in a future project. Well, lucky you, this page covers that topic.

1. Overall structure

  • Project platforms
    • This repo contains the platform implementation. The repo provides you with a universal API across platforms as well as the platform implementation.
    • This is the code base where you need to implement each API twice or thirds depends on how many platforms you want to support.
  • Project lib
    • This is the `common` code which you want to share across all platforms without any change. Such that, you can invoke `UserService.findAll()` to retrieve all the users no matter which platform you are working on.
    • This code will rely on `project platforms`. For instance, an API like `UserService.findAll()`, it needs to do an HTTP call where the actual implementation of sending an HTTP request is different across platforms. While using `project platforms`, this `project lib` can purely focus on the core business logic.
  • Project app
    • This is could be a native app which will call `project lib` for preparing all the data, while it only needs to focus on the UI and UX. Or maybe some native platform-specific feature.
This is a reasonable approach, at least to me. Where every `project` could focus on their own business.

2. Setup the project platforms

This is the easiest one. Because it’s just a typical kotlin multi-platform project setup. You can read the official document for more detail.
It should contain the platform implementation and the related tests.
The folder structure is straightforward:
  • common: for `common` code which contains all the `expect class`.
  • platforms:
    • jvm: contains `actual class` implementation for a jvm
    • js: contains `actual class` implementation for a js

3. Setup the project lib

The folder structure is like:
  • common: business logic that we want to share across platforms
  • bundle
    • jvm: Where we actually test and build the `common` for the `jvm` platform.
    • js: Where we actually test and build the `common` for the `js` platform.
This is one is a little bit tricky because we want to make it `common`. To things needs to take care here are:
  1. We need to add the `common` part from `project platforms` as dependencies.
  2. We want to test this `common` business logic.
  3. When we compile, we need to combine the platform implementations from `project platforms` along with the `common` to yield a platform-specific package.
First, let’s solve them one by one:
1. Add `common` part as dependencies, I tried many ways, including adding the generated `jar` file or `sources-jar` file. Always something wrong here, either the auto-completion stops works, or the internal member can’t be recognized while the namespace could be recognized. Only 2 ways work:
  • Just embed the source code from `project platforms :common` in you `sourceSets`: `main.kotlin.srcDirs += “$kotlin_idiom_path/interfaces/main”`
  • Add that common module as a subproject and add it to the dependencies.
  • I prefer the 1st one because now your gradle settings won’t contain any noises. It will benefit the side-panel of gradle in IDEA as well.
2. In terms of the test, you just write the tests in this `common` folder. But you can’t run the tests, you need to run them against the certain platform, otherwise, they are `common` code, which platform for them to run?
3. This is where the `bundle` coming into play. For building and testing. Let’s take `jvm` for example.
  • First of all, you don’t need to add any code here, this folder, as its name indicates, just for bundling code together. So, under in `:bundle:jvm` subproject, it only contains a `build.gradle` file.
  • You need to use `kotlin-platform-jvm` to compile this module
  • In the `sourceSets` setting: you need to add the source code from all the 3 places, both `common` modules from `project platforms` and this `project lib`, the source code of platform implementation from `project platforms`.
  • You need to add the tests only from `common` module of `project lib`, such that you can run the tests. And it won’t run the tests from `project platforms`, because they all gonna take care there. You don’t need to worry about that. Now run the `gradle :bundle:jvm test` will run the tests.
  • Why I add the source code rather than use the `jar` file? Well, hard lessons learned, this is the only way currently.

4. The end

Now run the `:bundle:jvm build`, it will build a lib to that platform. Try to consume it, it works really well. 🙂

Kotlin code sharing among platforms

When you try to deal with the across platform codes, you need to solve 2 things, one is what to share, the other is the architecture of how to share the code.

Different languages have different techniques to the second problem, but in Kotlin, you can use multiplatform projects to share the code.  With kotlin native, you can even expand the support to natively to iOS and Android.

1. Goals

  • share code among JVM, JS, iOS, and Android.
  • enable the use of XCode and Android Studio to edit the native app, while using another IDE to edit the sharing code part. (But you can use Android studio for editing the KN code as well because they use the same language and the setup will be ready.)
    • I saw some setups. They use root folder as a whole gradle project which will affect the Android project structure been displayed in the android studio because it will add some noise there.
  • share more code between iOS and Android.
    • The main hurdle is the platform-specific API. We can make them as `interface` and still shared the logic around them. But here we will use `kotlin multi-platform` approach to make the code more elegant.
    • Think of an API like this `UserService.getAllorNull()`, then your native side just need to bind the result to the UI. Even though the `HTTP` call is different across platform. But all the exception handling and data processing around the http call are the same and could be shared.
  • a clean structure where everything related should reside together.
    • I mean for the gradle phase as well. Such that, `android` folder is only for Android, `ios` folder is only for iOS. The root folder is for all projects. And the gradle settings for `KN-android` should be in the `android` folder as well as the `iOS` side.
  • enable testing the Kotlin native code as well.
    • It’s native project so anything native can be tested via the strategy on that platform. But what about kotlin native part, what about the `common` code? What about the kotlin native platform specific code for `android` and `ios`? Yes, we have that part covered as well here in the example.

1.1 What if there is no need to care about the platform-specific implementation?

If all you need is the API from kotlin-stdlib, and no platform specific code, then you don’t need to use this setup. You can simply remove the platform folder. And let the common just output an iOS framework for the XCode project to consume. For Android, embed the source code in Android studio to make it as dependency, then it’s fine. Same case for JVM and JS as well. And the setup for JVM and JS is as easy as following the official document.

2. Folder structure

  • ios: XCode project
  • android: Android Studio project
  • common: Common code that is about to share across platforms without any change.
  • platforms:
    • ios: platform specific API for iOS
    • js: platform specific API for js
    • jvm: a dependency for both `android app` and `jvmApp`

Very easy to understand. One interesting thing here is in the platforms folder, besides the common folder. What’s the platforms/android and platforms/ios folder for? Well, they are for platform-specific API.

2.1 How to consume the lib

  • JVM and JS just as easy as including the lib as the dependencies in build.gradle.
  • Android related code will be shared via simply including the according module in the android project.
  • iOS related code will be shared as an iOS framework because Kotlin native will generate that. We will then add it to the XCode building phase. You can see more details on my another [blog](http://www.albertgao.xyz/2018/01/14/how-to-create-kotlin-native-ios-project/).

2.2 Background knowledge for Kotlin multi-platform project

Let’s say you want to have an HTTP kotlin class where you can do the normal GET, POST thing, but the HTTP implementation is different across platforms. If you use the old school interface way, you need to declare an IHTTP interface, then implement it twice in android and ios. Then inject it at runtime to make it work.

But via using the Kotlin multi-platform project, you don’t need to inject, the compiler will only pick up the piece of codes depends on your building target. If you build for Android, it will only grab the Android implementation.

It currently works for jvm, js and kotlin native, oh, that’s a nearly-all-platform code sharing tech right? 😀

And it’s very simple.

You need to expect an implementation of your common code.

expect class Platform {
    fun get():String
}

Then implement the actual code in somewhere else, like android.

actual class Platform {
    actual fun get():String {
        return"android"
    }
}

Or in ios:

actual class Platform {
    actual fun get():String {
        return"ios"
    }
}

Then with some proper setup in gradle, it will work flawlessly.

2.3 How to code the `platforms-ios`

Oh well, you may think that old school interface may better in terms of iOS because you need to implement the iOS API specific thing in swift or obj-c then pass it back to kotlin native. But that is not true. Because there is a 1-to-1 API match in kotlin native such that you can implement the iOS specific API in kotlin as well. And it has the same signature which means you can reuse all the existing knowledge. And kotlin and swift are very similar. And from v6, the building phase will link some of the iOS API (you need to build the others) for you if you are on a Mac.
Which means in an ideal world, your swift side just needs to care about the UI part while all other logic will be handled in the Kotlin native side, shared across platform and tested well.

3. About the example

All the code for setup can be found in this repo.

  • `Sample` class is for code that is sharing across platforms (Which means you have to use API from `kotlin-stdlib-common`).
  • `Platform` class is a class which has been implemented twice for different platforms for showing the platform API case.
  • Open `android` folder from the root in Android Studio, run the app, it will show a string from the `:shared-android` project.
  • Open `ios` folder from the root in XCode, run the app, it will show a string from the `:platforms-ios` project.

In fact, all the apps (jvmApp, jsApp, android and iOS) retrieve the string by invoking the method from the Sample class. And the Sample class invokes the method from Platform class to get the string. When you build an iOS framework, the KN compiler will use :platforms-ios to build along with :common. And when you consume it in android project, the setup will use :platforms-android along with :common. No injection, no affection on the code structure, just that simple. Thanks to the Multi-platform project setup from kotlin.
It is a perfect example of showing how to share the code when you have to deal with the platform API.

Hope it helps. Thanks.