TulipeMoutarde.be

About

Table of Contents

Build mobile apps with Bazel. Part 1

Written on June 2 2018.

This blog post was written a long time ago and may not reflect my current opinion or might be technically out of date. Read with a grain of salt.

tl;dr;  This blog post series focus on using [Bazel](https://bazel.build/) to build polyglot mobile apps (iOS/android). We discuss why a monorepo is a good choice when dealing with shared code using Swift, Java, C and others. This specific post starts with Android.

(Update: The second part on building the iOS app is now available))

These days, I work a lot on software architecture and code reuse among projects and among platforms. In this blog post I will explore a way to oragnize source code to deal with the complexity of cross platform development.

We will build a small sample project on Android and iOS with Bazel and a monorepo. You can get the code related to this blog post on Github. Browse the tags to see the different steps.

Multi-platform mobile apps

One of the fallacies of mobile app development is the schism between Android and iOS. To go the native route, an app need two different codebases: one in Java/Kotlin for Android and one in Objective-C/Swift for iOS. Both require different knowledge and different tools resulting in two different teams where you could/should have one. Bugs are also duplicated and business logic is not canonical; there might be subtle differences in behavior between the apps.

Cross platform frameworks

To solve this problem, cross-platform frameworks have appeared. They have matured the past five years and they probably fill 80% of the common needs. Some completely abstracts the underlying platform (e.g., React Native, Flutter) while others still provide a first-class access to the native SDKs (e.g., Xamarin). They usually still sit on top of the native languages of the platforms. They provide hook to enter in the native world of Android and iOS.

They are no silver bullet though. Those frameworks have their quirks and sometimes you have to dig in and work with the native SDK. Debugging can also be a problem: is the bug in your code, in the cross platform framework or in the native one? It is a good idea to still have people who understand each platform and their associated tools.

They can also lag behind the official SDK. How fast do they integrate changes from Apple and Google in the official distribution?

This blog post won't use any of those framework. Expect a blog post in the future to know how and when they can shine and how/when they suck.

Good old C to the rescue

Another trend is to use the lingua franca of the programming languages: the C ABI. Most platforms can load and call libraries coded in C/C++. Mobile developers can leverage that situation and build cross-platform code that will work everywhere. And by everywhere, iOS and Android are not the only target; the gates are open to Windows, Linux, macOS and with the recent surge of WebAssembly, web browsers are also in the visor.

Slack, Dropbox, and Microsoft uses this approach for their cross platform apps.

Tools exist to help. For example Djinni (developed by Dropbox) is a tool that will generate the glue between C++ code and Objective-C/Java code from an IDL (Interface Description Language) file that describes your data structures and functions. You can call one side from the other, djinni handles the conversion.

When you think about it, C and C++ are not the only languages that can expose a C ABI-compatible library. Rust and Chicken Scheme can do that as well and might be well suited to your needs.

One of the biggest limitation of this approach is that you usually still need the native language of the platform to interact with platform-specific APIs: User Interface, Notifications, etc. You can share all the business logic code but you still need some entry points for the platform APIs.

A real-world example

Imagine the (not so) hypothetical case where we have two apps that we ship on the app store. If we release them on the two most common platforms, iOS and Android, we end up with 4 different codebases.

Four different codebase

As some of the features are similar between the two apps, we develop libraries to host the common code (some business logic, some UI/grapics elements):

Four apps with common lib per platform

It's already better. Especially if we had a third or fourth app. We can now add a third layer that will be common to all the apps:

Four apps with common lib per platform

The last layer could contain a lot of stuff: database access, network requests, business logic, etc. With more effort we can even imagine that our ViewModels would be shared between platforms. The small platform-specific layer between the lib and the Apps could be bindings automatically generated.

For this blog post, we'll stop at the second stage: different libs for each platform.

Multiple git repos

It is common among developers and dev shops to have separate repos for different apps/libs. In our case, that gives 6 git repos:

Lib-iOS and Lib-Android versioning

A solution for the libs is to version them and add them as a dependency to our apps.

For iOS, CocoaPod is probably the most common way to generate a package. This means we need a private PodSpec repo to keep track of the versions. That makes 7 git repos. The same goes for the Lib-Android, we might want to have a git repo acting as a Maven repository (e.g., with GitWagon). We now have 8 git repos.

The iOS and Android libs don't share any code yet, they are pure-Swift and pure-Kotlin code implementing the same set of feature. It's still better than not having a common lib for AppA and AppB as we have two duplicates instead of four.

Lib is used by both AppA and AppB. What if we make a breaking change into the Lib? This is where semantic versioning is important and you should have enough discipline not to break it. Unfortunately, it is quite hard to enforce automatically.

When working on the libs, we probably want to test changes in the context of AppA and AppB. We don't want to publish a new version until we know that the changes are correct. So we need a way to tell our build system to use the local (i.e., on the developer disk) version of the lib or the published version for building AppA and AppB. We can do that with buildVariant on Android and by fiddling with the Podfile/target on iOS but it can also be error prone.

Now if we work on a feature that requires changes in both the Lib and an App, the branches for both projects should be created and checkouted in sync. What happens if a developer starts to work on a feature on AppA and forget to change branch in the Lib?

By experience, it is quite common to make a change in the Lib, update AppA, forget about AppB and let it in a broken state. The CI (if we're lucky to have one) will catch the error but the harm is done: wasted energy.

Build tools

By default, iOS apps are built with xcodebuild, Android apps with Gradle. If we have libraries written in Rust, we'll need Cargo and if we have C libraries we'll need CMake (or anything that can build C). Tying all those build tools together can be quite challenging, especially when expressing dependencies.

Overall, working on that many different git repositories is a mess. It requires a lot of discipline and developer attention. If you develop a small app or does not require a lot a reuse this model works perfectly fine. I've used it for many years while developing Ruby On Rails app which tended to be self contained. But once you reach a certain threshold on certain kind of projects, it is simply painful.

Bazel and the monorepo

That's where a monorepo and Bazel come in. With those, we'll throw our packaging systems and use a unified build system.

Actually we might still need them, the ecosystems around CMake/Gradle/Cocoapod/xcodebuild are too pervasive to ignore. When developing an app, it is quite common to rely on 3rd party libraries that are packaged with the native tools of the platform.

We'll go the full route and use a monorepo. Our previous git repositories will be merged into one. Each previous repo will be a top level directory in the new one. This might sound crazy (or event heretic) but stick with me here.

Bazel

Blaze is a build system initially developed internally at Google. They open sourced it in 2015 under the name Bazel. Developers that left Google before Blaze was open source were so fond of it that they re-developed it to build stuff at the companies they were joining. That's how Buck (by Facebook) and Pants (by Twitter) were born.

Today, Bazel is widely used and can build a lot of different languages Uber, Dropbox, Etsy, Huawei, Stripe, SpaceX, Pinterest and many others are building software with it (links at the bottom of the post).

Conceptually Bazel is quite simple. You have a WORKSPACE file at the top of your project. This is where you'll load and configure all the rules you'll need to build your project. There are official rules for a lot of different stacks (e.g., C/C++, Java, Android, Go, Kotlin).

In every directory containing a buildable artifact you'll have a BUILD file. It describes where the files to compile are, the dependencies you need and other configuration options like the visibility of the build output.

And that's basically it. Bazel will build the dependency graph and aggressively cache the artifacts that have already been built. That's why you want to have a lot of small modules instead of a big one containing everything. Bazel encourages you to keep everything small and compose your code from smaller parts.

A side effect of this layout is that someone modifying a library must ensure that all the parts that depend on it are not broken. Bazel has a query system to let you see the impact of modifying one unit of code. As everything is in the same repo, the developer has everything she needs to ensure that the build is not broken. She doesn't need the discipline to create a branch for each repo affected by the feature she's working on as everything is in the same place.

When the provided rules are not enough, it is possible to code your own using Skylark, a language derived from Python.

All of this is well in theory but in practice, how does it work?

Initial setup

We will create a small Android App that displays a string from a Java lib. In future post, we will add a C library that will be called with JNI. We will then add more complex data structure and iOS to the mix.

You can follow the whole project on this Github repository.

Create a single top-level directory for all our code. This directory will be versioned in a unique Git repo. Feel free to use any other VCS. It does not really matter at this point.

Create a WORKSPACE file. This file will inform Bazel of all the tools we need to build the code.

Create AppA-Android

Add the Android SDK to the WORKSPACE file:

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:maven_rules.bzl", "maven_aar")

android_sdk_repository(
    name = "androidsdk",
    api_level = 27,
    build_tools_version = "27.0.3",
)

This first instruction makes the maven_aar rules available. It is useful to handle android libraries in the .aar format. The second one tells bazel the configuration of the Android SDK that we want to use.

Android usage is fully documented on the official website. All the possible options are listed under the Android Rules.Ensure that you have your $ANDROID_HOME environment variable correctly setup so Bazel can pick the android tools (see the android doc)

We can now create the android app. Beware that we don't use the typical Gradle layout but a very barebone one:

$ tree -L 7 App-A-Android/
App-A-Android/
├── BUILD
└── src
    └── main
        ├── AndroidManifest.xml
        ├── java
        │   └── be
        │       └── tulipemoutarde
        │           └── appa
        │               └── MainActivity.java
        └── res
            ├── drawable
            │   └── ic_launcher_background.xml
            ├── layout
            │   └── activity_main.xml
            ├── all-the-mipmap-folders
            └── values
                ├── colors.xml
                ├── strings.xml
                └── styles.xml

You'll notice the BUILD file right under the App-A-Android folder. It describes how to build this particular artifact:

App-A-Android/BUILD

android_binary(
    name = "AppA-Android",
    custom_package = "be.tulipemoutarde.appa",
    manifest = "src/main/AndroidManifest.xml",
    srcs = glob(["src/main/java/**"]),
    resource_files = glob(["src/main/res/**"]),
    visibility = ["//visibility:public"],
)

Nothing really fancy here, it should be readable. Note that the name property (AppA-Android in our case) will be used to identify the entity when building it or when referencing it. Once again, the full documentation for android_binary is available.

We can now check if we can build the project:

$ bazel build //App-A-Android:AppA-Android

If you have a device plugged in or a simulator running, you can directly launch the app on it:

$ bazel mobile-install //App-A-Android:AppA-Android

mobile-install has a myriad of ptions that you can use to customize the launch of the app (e.g., cold start, incremental updates).

Add external library

Almost all Android apps use external libraries to perform various actions. Developers can express the dependencies in the gradle build file. Dependencies are usually shipped as regular jar archive or as aar if they ship with resources. Bazel supports both of them natively.

We will add Timber, a small log library to our project. We first need to add the library to our WORKSPACE. Doing so will make it available for all the BUILD files:

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:maven_rules.bzl", "maven_aar")

maven_aar(
    name = "timber",
    artifact= "com.jakewharton.timber:timber:4.7.0"
)

The first instruction will load the Bazel maven rules and the second will actually point to the external dependency.

App-A-Android/BUILD: {{<highlight python "hl_lines=8-10">}} android_binary( name = "AppA-Android", custom_package = "be.tulipemoutarde.appa", manifest = "src/main/AndroidManifest.xml", srcs = glob(["src/main/java/"]), resource_files = glob(["src/main/res/"]), visibility = ["//visibility:public"], deps = [ "@timber//aar", ], ) {{< / highlight >}}

Notice how the deps param got a reference to Timber. In this case, the dependency is an aar package but it could have been a java lib that is in your monorepo. The monorepo and Bazel are well suited to code that does not depend on a lot of external dependencies. Google probably has a lot of internal code to perform various tasks while smaller companies probably depends more on external libraries.

We can now import Timber in our code and check that everything builds fine:

MainActivity.java:

// .../...
import timber.log.Timber;
import static timber.log.Timber.DebugTree;

// .../...
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    Timber.plant(new DebugTree());
    setContentView(R.layout.activity_main);
}

Note that we've imported the external dependency in the WORKSPACE file and refer to it in the BUILD file.

The resulting code is visible as a tag in the sample project.

Add Google Maven repository

Since Android Studio 3.0, Google distributes android libraries through a dedicated Maven repo. Unfortunately it does not to work natively with Bazel. The gmaven rules is a temporary (?) workaround to alleviate this issue. It feels like a ugly hack but it works.

gmaven_rules manually hardcodes all the libraries provided by Google in its maven repository in a single file. This means the code has to be updated if you need the latest version of the libs.

Add the following in WORKSPACE: {{}} GMAVEN_TAG = "20180513-1"

http_archive( name = "gmaven_rules", strip_prefix = "gmaven_rules-%s" % GMAVEN_TAG, url = "https://github.com/bazelbuild/gmaven_rules/archive/%s.tar.gz" % GMAVEN_TAG, )

load("@gmaven_rules//:gmaven.bzl", "gmaven_rules")

gmaven_rules() {{}}

Check the gmaven_rules releases page to find the right GMAVEN_TAG to use when you read this. Once the WORKSPACE has the gmaven_rules, we can use them in the BUILD file of our android app:

App-A-Android/BUILD: {{<highlight python "hl_lines=1 12">}} load("@gmaven_rules//:defs.bzl", "gmaven_artifact")

android_binary( name = "AppA-Android", custom_package = "be.tulipemoutarde.appa", manifest = "src/main/AndroidManifest.xml", srcs = glob(["src/main/java/"]), resource_files = glob(["src/main/res/"]), visibility = ["//visibility:public"], deps = [ "@timber//aar", gmaven_artifact("com.android.support:appcompat-v7:aar:27.1.1"), ], ) {{}}

We can now extends AppCompatActivity instead of the Activity:

{{}} package be.tulipemoutarde.appa;

import android.os.Bundle; import android.support.v7.app.AppCompatActivity; import timber.log.Timber; import static timber.log.Timber.DebugTree;

public class MainActivity extends AppCompatActivity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); Timber.plant(new DebugTree()); setContentView(R.layout.activity_main); } } {{}}

We can now build an Android app with dependencies on both Google and external libraries. Check the git tag on example repo if needed.

Create Lib-Android

This is getting interesting. We will add a library inside our monorepo. This library will be available for any BUILD file in the whole repo.

We create a new folder that will contain all our code: $ROOT/Lib-Android. Once again, we need to tell Bazel how to build the library in the BUILD file.

Lib-Android/BUILD:

android_library(
    name = "Lib-Android",
    srcs = glob(["src/**"]),
    visibility = ["//visibility:public"],
)

The visibility parameter tells Bazel that we want the artifact to be available everywhere. As we want to reuse this library in the main Android app, it should be visible.

Our Java code is pretty simple. Lib-Android/src/be/tulipemoutarde/lib/StringProvider.java:

package be.tulipemoutarde.lib;

public class StringProvider {
    private String baseString;

    public StringProvider(String baseString) {
        this.baseString = baseString;
    }

    public String getAugmentedString() {
        return "Processed " + baseString;
    }
}

We can now build the lib in isolation:

$ bazel build //Lib-Android

Note that this time we didn't specify the name of the artifact to build. This is because the identifier given to the lib matches the folder name. We couldn't do this with the android app because the folder is named App-A-Android while the app identifier is AppA-Android (notice the extra -).

We can now add the library as a dependency to the main app:

{{<highlight python "hl_lines=11">}} load("@gmaven_rules//:defs.bzl", "gmaven_artifact")

android_binary( name = "AppA-Android", custom_package = "be.tulipemoutarde.appa", manifest = "src/main/AndroidManifest.xml", srcs = glob(["src/main/java/"]), resource_files = glob(["src/main/res/"]), visibility = ["//visibility:public"], deps = [ "//Lib-Android", "@timber//aar", gmaven_artifact("com.android.support:appcompat-v7:aar:27.1.1"), ], ) {{}}

Ugh. All the deps are written differently.

Finally we can use the code in our Activity:

{{<highlight java "hl_lines=16-17 3">}} package be.tulipemoutarde.appa;

import be.tulipemoutarde.lib.StringProvider; import android.os.Bundle; import android.support.v7.app.AppCompatActivity; import timber.log.Timber; import static timber.log.Timber.DebugTree;

public class MainActivity extends AppCompatActivity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); Timber.plant(new DebugTree()); setContentView(R.layout.activity_main);

    StringProvider provider = new StringProvider("Bazel");
    Timber.d("Got an augmented string %s", provider.getAugmentedString());
}

} {{}}

We can now re-build the app:

$ bazel build //App-A-Android:AppA-Android

Bazel is smart enough to know when to recompile the lib. As usual, you'll find the sample project at this stage under a git tag.

Testing

Android Instrumentation Tests are still experimental in Bazel. They were introduced in 0.12 and are maybe still a moving target. The documentation has a step by step tutorial, we won't go through it at the moment.

The only downside is that we need to create a dummy android app if we want to test the library in isolation of the main app in an instrumentation test. This is also true when using Gradle.

General feeling and conclusion

It is the first time I use Bazel and my impressions are mixed. It feels really robust and forces the developers to think in small modules that you can mix and match together.

Bazel assumes that all your dependencies are available in-house. This is certainly true for companies like Google but not for smaller ones. Dealing with external dependencies feels alien. There are projects to generate rules from native tools but they feel clunky (I'm looking at you gmaven_rules()). If your language of choice has a standard tool for dependency management (e.g., NPM, Bundler, CocoaPod, Cargo), you will need some duct tape to integrate it into Bazel. It is not that bad but certainly something to check if you plan to switch to Bazel.

The monorepo setup is quite frankly a game changer for at least one of my current clients. Not all the projects I work on would benefit from it but it's a new tool that I'm happy to have in my toolbox.

I believe that a shop developing for a single platform (e.g., iOS) could benefit from Bazel. If you have 4 or 5 apps in the App Store, they are good chances that you have some code that you reuse among all of them. Some people uses symlinks and/or git submodules to deal with the situation. If it is your case, you might want to investigate Bazel to see if it could handle your workflow.

In the following blog posts we will:

If I find the time and the motivation, I would like to cover some other cases:

I'm exploring Bazel as I write this series so I probably missed a few things. Please ping me if you have any remark or question.

([Update] The second part on building the iOS app is now available))

References