Modernizing our Android build system

See the original post on Dropbox’s blog

One of the biggest challenges of the mobile developer community at Dropbox in 2018 was our custom build system. Our build system was slow, hard to use, and didn’t support some use cases which were out of scope of the original design. After 4 months of work by our Mobile Platform team, we were able to remove our unicorn implementation for something much more modern and easy to maintain.

In our new build system, we wanted to improve on a couple of things that our current build system was hindering:

• Make it easy to create new modules
• Allow developers to easily modify the build files
• Improve on build times for local development
• Industry standard approaches and tooling, so an engineer can easily Google their way out of problems
• Low barrier of entry, familiar to new hires
• Integration with Android Studio

  History

At Dropbox, we have a repository for all mobile development, called Xplat. One of the benefits is to easily share source code between our different mobile applications and across platforms. For a while, Dropbox invested heavily in cross-platform development via C++ that worked well for the apps that were developed at that time. We even open-sourced Djinni in 2014 to interface cross-platform C++ library code with platform-specific Java and Objective-C on Android and iOS. Most recently in early 2019, we made the decision to move away from C++ development, read more about why here. However, some of our mission critical libraries will remain on C++ e.g. DocScanner which uses OpenCV.

Since December 2016, Dropbox used a meta-build system to build our two mobile apps: Dropbox and Paper. It was a meta-build system in the sense that, for most of our modules, we didn’t write the Gradle build files by hand. These build files were autogenerated using an in-house system called BMBF. Additionally, BMBF would generate Java source code for our analytics, feature gating, and other common libraries written in C++.

What is BMBF? BMBF (Buildy McBuildface Basic Modular Build Format) was a tool written in Python to help modularize our mobile code base.

BMBF provided guaranteed layered dependency order and reduced boilerplate in build files.

BMBF used a structured and opinionated file system layout. Then it was able to generate build.gradle and wire in those modules into settings.gradle. This made it ‘easy’ to create new modules or re-use an existing module without sacrificing the benefits that come from using the official tools for each platform.

The following module config file:

device.bmbf.yaml

dependencies:
  - dbx/base/async

java:
  src_maven_dependencies:
    - dagger
    - dagger_compiler
  jvm_test_maven_dependencies:
    - junit

Link to gist

Would get parsed by BMBF’s build system and output a build.gradle file

apply plugin: 'com.android.library'
apply plugin: 'kotlin-kapt'

android {
    sourceSets {
        main {
            manifest.srcFile 'AndroidManifest.xml'
            java.srcDirs = ['java/src']
            test.srcDirs = ['jvm_test/src']
            androidTest.srcDirs = ['android_test/src']
        }
    }
}
dependencies {
    implementation project(':dbx:base:async')
    implementation project(':dbx:base:oxygen')
    implementation commonlibs.dagger2
    kapt annotationprocessors.dagger2_compiler
    api commonlibs.kotlinstdlib
    testImplementation testlibs.junit
}

Link to gist

How did BMBF not meet our needs? BMBF was opinionated, it made it difficult to add functionality to our Gradle scripts that were not built into BMBF. If BMBF didn’t support a workflow that a product engineer required, they would either file a ticket on Mobile Platform, or tried to add the functionality themselves.

BMBF had numerous gotchas and a steep learning curve. BMBF required a very specific file and folder structure. BMBF was not compatible with Gradle incremental builds because build.gradle files were being re-created every time a developer built the app. It was not uncommon for engineers who had been with the company for 6+ months to still had no idea how to create a new module using BMBF. Our Slack channels and help forums were bombarded with questions on how to resolve errors that BMBF presents.

BMBF was initially built to help facilitate modularization and sharing code between iOS and Android. Over the years, the original maintainers that created BMBF moved on to other projects or left the company. Our current engineers were not eager to maintain a legacy meta build system and were more in favor of leveraging a standardized build system. Since we started using BMBF, we stopped writing cross-platform modules and even wanted to rewrite C++ modules into platform-specific iOS and Android code. The time had come to revamp our build system.

---

Part I: The planning

Our team worked for several weeks on evaluating and analyzing our options to review:

• Gradle + BMBF (status quo)
• Gradle only
• Bazel
• Buck

Our team created a sandbox environment to accurately compare different build systems.

Gradle Only No work was needed for Gradle, as it was our baseline.

Bazel While Bazel was widely used at Dropbox, its use didn’t propagate to our mobile teams. An engineer from Developer Infrastructure worked on generating all the BUILD.bzl files required for all our modules in order to successfully build an APK.

Buck We did not evaluate Buck because it was not well supported by the community at the time and did not support Kotlin. Also, we did not have any in-house expertise in Buck at the time. We would have needed to dedicate 1-2 mobile engineers full time to work on Buck.

---

Build times

Using the prototypes, we compared the three different build systems’ build times and developer experience.

Build times	Bazel Only	Gradle + BMBF Build	Gradle only
Clean Build no cache	638 s	252 s (with buck cache)	~ 258 seconds
Clean Build w/cache	81.459s	N/A	N/A
NoOp Build	1.334s	20-30 seconds	~ 28.6 seconds
[Incremental] Main module modified	~36 sec	~181 seconds	~ 124 seconds
[Incremental] Shared library module modified	~29 sec	~200 seconds	~ 108 seconds

---

What are the major decisions, their options, and their tradeoffs?

Bazel Build System:

• Decision: Invest heavily in Bazel now and move C++ and Java/Kotlin development to Bazel.
• Options: This will require upfront investment in a Bazel MVP, as well as continued support to make up for new/missing features that are only released in Gradle by Google. As of December 2018, Google is still working on open sourcing some missing critical pieces of Bazel Android from the internal version, Blaze. For example, App Bundles, which was released on May 2018, is still not available on Bazel as of Dec 2018.
• Pros
  • Tool chain managed by Dropbox Developer Platform team
  • Bazel is widely used in Dropbox
  • Tool built and maintained externally
  • Unified build system for C++ and Java/Kotlin
  • Scales to larger code bases in terms of performance
• Cons
  • Not an industry standard for mobile
  • Latest and greatest features and libraries are not available
  • Requires a bigger upfront investment than migrating to Gradle
  • Requires continuous long term support from Dropbox Developer Platform team until Bazel becomes mobile industry standard (Currently we know only of Google as a user of Blaze for Android)
  • The Android Studio team at Google is focusing on Gradle, at the expense of Bazel
  • The Bazel team at Google is working on adding support for Android and open to feedback but will always be trailing Gradle by 1-2 quarters

Gradle Build System:

• Decision: Make a smaller, strategic, investment to move Java/Kotlin development to Gradle by checking in the project files and removing BMBF from Gradle model management.
• Options: Code generation and C++ development will continue to be done by BMBF. Invest in some guardrails (lint, Herald, templating) to make working with Gradle easy for developers.
• Pros
  • Industry standard for mobile
  • Latest and greatest features and libraries are available
  • Tool built and maintained externally
  • Low migration cost
  • Low maintenance cost
• Cons
  • Poor support for cross platform C++ development. Building xplat C++ code is delegated to BUCK via BMBF
  • Will require some support from Mobile platform over time (e.g version bumps, guardrails, maintenance)

BMBF Build System

• Decision: Keep and maintain BMBF as the mobile build tool
• Pros
  • Good support for cross platform C++ code
  • Harder to break Gradle configurations through developer error
  • New features can be made available by prioritizing work on them in-house
• Cons
  • (Unenthusiastically) managed and maintained by our team, rather than a separate build team
  • Tool is not built and maintained externally
  • Not an industry standard for mobile
  • Will require continuous long term support. New features will require in-house investment

Why not Buck as a build tool

• Decision: Buck is an optional build tool however we didn’t evaluate it as a serious option do to its clear lack of community support and drawbacks.
• Options: Buck is designed to address massively modularized code bases (100+ modules). It also would require significant expertise to maintain and support. As an example Uber has a 3 person team dedicated to Buck support, one of whom is the author of OkBuck which allows using Buck with Gradle projects.
• Pros
  • Good support for cross platform C++ code and caching
• Cons
  • Managed by mobile engineers
  • Not well supported by the community and Facebook (at least for external users)
  • Not an industry standard for mobile

Full Trade-offs Table

---

Decision

We decided to move forward last year with the Gradle Build System, and we will soon be revisiting Bazel. The cheap migration cost from BMBF to the underlying Gradle lead to the decision to first deprecate BMBF.

Although Bazel was extremely fast with regard to the the build time, we were concerned about deteriorating the local developer experience. At the time of our evaluation, Bazel was not very mature. Gradle is the industry standard for building Android apps. Tooling and libraries available for Gradle will take time for it to become available for Bazel.

---

Part II: The execution

What does an engineer do after planning and decision on an approach? More planning! This time, the planning was laser-focused on what features to build into the new Android build system. Our team created a doc which outlined the milestones of the project and shared it to our internal customers for feedback. Subsequently, we agreed on the project scope and the key ideas for the migration.

Foundation for migration

Our goal in implementing a Gradle-only solution was to introduce flexibility, but also keep some of the great features BMBF had: guaranteed layered dependency order and reduced boilerplate in build files. We had to lay down the foundation before we could remove BMBF…

Layered Dependencies At Dropbox, we add our modules into different layers. This is different from a MVx layer that one might be familiar with.

Each module is in a layer, determined by the direct subdirectory of dbx/ that it’s in. Layers are used to give a quick broad sense of the scope of a module and to enforce high-level dependency constraints.

Layers are ordered from “top” to “bottom,” which gives some high-level structure to the app and its dependencies. The layers also indicate the scope of the modules in them. The higher layers tend to be more specific and narrow in scope, while the lower layers are more general and broader.

The 4 valid layers are listed below, in order from top to bottom, along with descriptions of the scope that modules in each layer should have.

Layer Name	Layer Description
product	Modules relating to a single Product (eg Paper or Dropbox). Modules in this layer will typically be under a subdirectory specifying which product they’re part of.
core	Dropbox-related modules that are shared between multiple products. For example, Stormcrow (our gating library) lives in this layer.
base	Non-Dropbox-specific modules that are common utilities. For instance, our HTTP libraries are in this layer.
external	Code not written at Dropbox which cannot be pulled in as a library binary.

To give a concrete example, the module at dbx/core/stormcrow is in the core layer (because that is the immediate subdirectory of dbx/ that it’s in). It’s in core because Stormcrow is a Dropbox-specific concept, but is used by both DBApp and Paper. Since it is in core, this module cannot depend on a module in product, but can depend on other core modules, as well as base and external modules.

In BMBF, the layered verifier was written in Python (since BMBF was written in Python) and there was a Gradle task that made a call to this Python script every time the app was built. This was a waste of resource because there was no good way to cache the task. We wanted to make the code maintainable by any mobile engineer, so we decided to add a layered verifier in buildSrc written in Kotlin. In addition, we got Gradle UP-TO-DATE checks for free.

Reducing Boilerplate We wanted to allow an engineer to create a new module and not have to copy and paste a block of Gradle boilerplate from another module. We were able to achieve this by having a common Gradle file that defined what to apply for subprojects. In the end, most engineers can now create a new module and only need to focus on adding dependencies they require to their projects. Since the build.gradle files are no longer autogenerated by BMBF, engineers can now also define logic in their scripts that are not accounted for by the build system.

common.gradle

    subprojects { Project project ->
        project.apply from: xplatRoot.absolutePath + "/tools/gradle/test_results_formatter.gradle"
        project.plugins.withId('com.android.library') {
            project.apply plugin: 'kotlin-android'
    
            if (project.hasProperty('apply_jacoco_plugin') {
                // Runs in CI or when a local dev enables this property
                project.apply from: xplatRoot.absolutePath + "/tools/gradle/jacoco_test_coverage.gradle"
            }
    
            project.android {
                compileSdkVersion androidCompileSdkVersion
                buildToolsVersion androidBuildToolsVersion
    
                lintOptions {
                    ignore 'MissingTranslation'
                }
    
                defaultConfig {
                    minSdkVersion androidMinSdkVersion
                    targetSdkVersion androidTargetSdkVersion
                    testInstrumentationRunner 'com.dropbox.base.test.runner.DbxBaseTestRunner'
                    // This is needed for when we build this as a standalone target,
                    // which will typically be in tests.
                    multiDexEnabled true
                }
    
                compileOptions {
                    sourceCompatibility androidSourceCompatibility
                    targetCompatibility androidTargetCompatibility
                }
            }
        }

Link to gist

With the common Gradle code in place, an engineer can write simple build.gradle files and we don’t need to copy and paste boilerplate around. Also, if we decide to make changes to the common code, it’s very simple since we can do it in one place.

build.gradle

    apply plugin: 'com.android.library'
    
    dependencies {
        api project(':dbx:base:analytics_gen')
        implementation project(':dbx:base:error')
        implementation project(':dbx:base:json')
        implementation project(':dbx:base:oxygen')
        implementation commonlibs.guava
    }

Link to gist

---

Migration

The transition was pretty smooth. Looking back, we realize that overcommunication was the key to our success.

After laying down the foundation, we were ready to remove BMBF from autogenerating Gradle build files. This meant we needed to check-in all the autogenerated Gradle files. We made changes to the generators so that they would create the simplified versions of build.gradle based on the foundational work we did on the common Gradle code.

This was a high-impact change, so we decided to do it immediately after a release went out, and we forbid changes to the repository until this change was landed. As one could imagine, some engineers had created some new modules while we were migrating. For those engineers, we had a back-channel to allow them to force migrate their BMBF config files to build.gradle. The transition was pretty smooth. Looking back, we realize that overcommunication was the key to our success. At every stage of our project, we communicated with our customers (mobile engineers) about our goals and what we planned to do. For high-impact changes like this, we sent out emails, Slack messages, and mentioned it in our weekly cross-functional mobile meetings.

We planned out our work so that for the week after we made this transition we were sure to reserved some capacity to deal with issues that engineers might face.

---

Post-migration

At this point, you might think, “You already migrated all the code to Gradle and got rid of BMBF build.gradle generation… all done!”

The migration work unlocked the potential to clean up our directory structure—which was also the biggest time sink in our build times—so it allowed us to find other ways to improve build times.

BMBF used a non-standard directory structure for Android. Also the BMBF modules were not in the same root as our Android project, which meant additional boilerplate to our settings.gradle files.

/xplat/dbx/… → BMBF modules /xplat/android/… → Android projects

BMBF structure (Before)	Gradle default structure (After)
java/src	src/main/java
android/src/AndroidManifest.xml	src/main/AndroidManifest.xml
jvm_test/src	test/main/java
android_test/src	androidTest/main/java

There were 75 modules written in BMBF and migrating them by hand would not have been very efficient. Mobile Platform could have required all product teams to migrate their modules to standard Gradle. Instead, we chose to tackle this problem by writing a script that would move all of the code into the right directory. In the end, this migration script was a 500 line Python file that accurately determined which BMBF modules needed to be migrated, updated the imports in the source files, and updated the project dependencies in the build.gradle files.

BMBF-lite We made the decision to not completely remove BMBF because it was still responsible for code generation for Djinni, gating (Stormcrow) ADL files, and analytics ADL files. Previous to our project, the code generation was called every…single…time an engineer triggered the build, even though no changes were made to the source files. Another optimization we made was to introduce Watchman to watch the source directory and use it as an @Input and @Ouptut for our Gradle task. Watchman tells Gradle when to re-run the task and when the task is up to date.

This along with a few other minor improvements reduced our P50 local build times by 20%.

watchman.gradle

/*
 * Copyright (c) 2019, Dropbox, Inc. All rights reserved.
 */

// Watchman generates a json file to depict the xplat file structure filtered on "bmbf source" files
task watchmanCheckIfCodegenNeeded(type:Exec) {
    File outputJson = new File(project.buildDir, "changed-files.json")
    File watchmanJson = new File(xplatRoot, "tools/watchman/watchman-bmbf.json")

    workingDir xplatRoot
    commandLine "bash", "-c", "watchman watch-project $xplatRoot"
    commandLine "bash", "-c", "watchman -j < $watchmanJson.absolutePath"

    doFirst {
        standardOutput new ByteArrayOutputStream()
    }

    doLast {
        // Remove this piece of data that changes on every run (even with no modifications to the files)
        def filteredText = standardOutput.toString().replaceFirst(".*\"clock\".*\n", "")
        if (outputJson.exists()) {
            outputJson.delete()
        }
        outputJson << filteredText
        logger.lifecycle("Watchman query for BMBF files done: " + outputJson)
    }

    // Save the json as the output so other tasks can reference it easily
    outputs.files { outputJson }
    // Always run this task
    outputs.upToDateWhen { false }
}

Link to gist

---

Conclusion

Initially, we had a custom build system that solved our use cases across both platforms. Over time the customizations and costs outweighed the gains. Rather than continuing with BMBF we decided to split iOS and Android build systems to better take advantages of platform specific needs and tools. While this meant we lost some shared patterns and code we gained flexibility in not having a shared build system. Therefore, we made a decision to closer align with industry standards by going to Gradle and leaving the door open to re-evaluate Bazel in the future. As the mobile world continues to move at a blinding pace we want to be ready to adapt our architecture to support it for years to come. Finally, we were able to do this all with minimal interruption to Android engineers and without any additional boilerplate.

We’re Hiring! We hope to have you on board to help us make such dramatic changes in the future.
If you are an Android or iOS engineer who gets excited about solving problems at scale and sharing your findings with the community we’d love for you to come join the team!

Share on: