Mastering mobile app quality: Rippling's CI/CD journey with Bitrise

Rippling, a pioneer in the HR tech industry, is committed to delivering a seamless user experience through its mobile app. With a growing user base, ensuring the reliability and performance of the app at scale has been a top priority for the engineering team. This blog post takes you through the journey of how Rippling scaled its mobile infrastructure from eight MacBooks in the office to a fully cloud-based CI/CD setup. We’ll cover the challenges we faced in building a best-in-class mobile CI/CD with scalable cloud-based testing, automated notifications, HTML test reports, and daily health checks.

Rippling’s mobile tech stack

There are four technologies that comprise the tech stack of the Rippling mobile app:

React Native

• Rippling’s commitment to cross-platform development is evident through its use of React Native. This framework enables efficient code-sharing between iOS and Android platforms, fostering a streamlined development process.

Detox testing framework

• Detox is a robust, end-to-end gray-box testing framework for React Native applications. With its support for parallel test execution and a rich set of testing utilities, Detox ensures comprehensive test coverage for the Rippling mobile app.

Android SDK + Android Emulator

• Leveraging the Android Software Development Kit (SDK) and the Android Emulator, Rippling ensures that its app functions seamlessly across a range of Android devices.

MacOS + Xcode + iOS Simulator

• For iOS development, Rippling relies on the MacOS environment, Xcode, and the iOS Simulator. This combination provides the necessary tools for building and testing the app on various iOS devices.

Setting up a mobile testing environment for such a tech stack can be challenging and expensive due to various factors, including CPU performance, high memory requirements, GPU dependencies, virtualization support, scalability, and OS compatibility.

Failed attempts

As described in the previous section, setting up a mobile testing environment is not straightforward and requires a sophisticated infrastructure and skillset to operate and maintain the testing pipelines of the mobile app. To accomplish this feat, Rippling’s mobile CI/CD pipeline has gone through multiple iterations. A couple of the major tools we experimented with include:

• GitHub Actions: While GitHub Actions allowed us to successfully build our Android and iOS apps on their runners, running end-to-end tests on emulators and simulators became a challenge. The runners provided by GitHub were not powerful enough to smoothly run Android emulators and iOS simulators. We faced issues with slow app execution and emulator hangs, which created a lot of flakiness in the CI pipelines and made the test suite unreliable.

• AWS MacOS instances: AWS M1 Mac instances provided us with powerful machines to run our CI/CD pipelines. Initial experiments worked well, and we set up a pipeline to run end-to-end tests on these instances. But as time went on, we experienced slowness on the Mac servers, and simulators and emulators started to stutter and hang while executing tests. We realized this was happening because the Mac machines had been running for a long time without restarting. Moreover, restarting these machines took a good 30-45 mins before we could use them again, making it impractical for us to use this setup.

Note: GitHub large runners were not yet released when we were working on our CI/CD setup. With the introduction of Linux and macOS large runners, they may now be a viable option for a scalable CI/CD environment.

Stop-gap solution

At Rippling, we embrace our leadership principle of “Go to Western Union,” which means we do what’s needed, even if it’s out of our lane and unglamorous.

While we were developing our automated CI/CD testing pipeline, the product contributions to the mobile app were scaling quickly as well. To accommodate these daily contributions and avoid any major incidents due to releasing untested code, we devised a stop-gap solution to ensure the reliability of the mobile app on production. We accomplished this by setting up eight MacBooks in the office and configuring them as self-hosted GitHub runners. This helped keep our testing infra from falling apart while we continued our research for a reliable, cloud-based mobile CI/CD setup.

Bitrise: A mobile-first cloud-based solution

Recognizing the need for a scalable and efficient mobile CI/CD solution, Rippling turned to Bitrise.

Bitrise specializes in mobile-first CI/CD and provides all the necessary tooling required to build a scalable pipeline for our mobile requirements.

We chose Bitrise because it was able to meet our main technical requirements, including:

• Ephemeral Mac instances: The Mac instances provided by Bitrise are ephemeral, and their boot time on average is <10s.
• Seamless vertical scaling: All instance types of various capacities are readily available for use on Bitrise, and we can seamlessly switch between them.
• Horizontal scaling: We run 50 VMs in parallel on Bitrise to run our jobs, which is more than enough to fulfill our current scale.
• Support for common dev tools: Bitrise supports common mobile dev tools like Xcode, Android SDK, emulator/simulator, etc. Switching between different versions of these dev tools via the Bitrise dashboard is trivial, thereby reducing maintenance overhead.

Creating the mobile CI pipeline on Bitrise

With Bitrise, we finally had servers optimized for the mobile ecosystem and didn't have to worry about maintenance and other DevOps challenges we faced earlier. This gave Rippling the freedom to create helpful tooling to improve the testing, developer experience, and release process for the mobile app.

The mobile CI pipeline is the first in a series of tools built by the Rippling mobile team, intended to run quality checks on every pull request created by developers for the mobile app. The following are the primary stages of the mobile CI pipeline:

• Static code checks: Ensure that the code adheres to coding standards and best practices by running checks like Lint, Typescript, circular dependency detection, etc.
• Unit tests: Validate the functionality of individual units of code by running unit tests to catch any early errors.
• Seed test territories: Set up initial test conditions to ensure consistent test results for end-to-end tests.
• Build mobile apps: Build the Android and iOS apps in parallel to ensure there are no compile-time errors in the code. These builds are also used to run end-to-end tests in the next stage.
• Test mobile apps: Run end-to-end tests using the Detox testing framework on Android emulators and iOS simulators in parallel. This ensures that all the user flows are tested end-to-end on the mobile app and catches any errors/crashes introduced by the new changes even before they’re merged.
• Publish results: Once the whole pipeline has been executed, we close the feedback loop by generating test reports and publishing the results of the CI pipeline run via GitHub comments and Slack messages.

Closing the feedback loop with CI notifications

At Rippling, we aspire to have the best developer experience in the industry and hold great regard for developer productivity. Therefore, prompt notifications are a key aspect in improving developer experience and time. To achieve this, we generate detailed reports that provide insights into the success or failure of each stage in the CI pipeline.

Automated GitHub comments on PR

Automated comments on pull requests keep the developer informed about the status of the tests, fostering collaboration and quick issue resolution.

Slack notifications

Real-time notifications via Slack bots ensure that the PR author is promptly informed about the test results, facilitating swift action in case of failures.

Dynamic HTML test reports using Allure

Allure is an open-source framework designed for test report generation. We integrated it with our Detox testing framework to generate dynamic test reports for every test run. End-to-end test reports generated by Allure provide a visually appealing and interactive way to consume the test results, making it easier for teams to analyze and understand the outcome of their test suites.

Daily health checks

We run our mobile CI pipeline on the master branch multiple times a day. This keeps our master branch ready to deploy to production whenever required. The results of these health checks are published as Slack alerts in a common Slack channel, allowing us to monitor the health of the master branch.

Improving the mobile release (CD) pipeline

Rippling had initially created a mobile release pipeline using GitHub Actions. While the pipeline did its job, building and submitting the app to the respective Apple and Google stores took a lot of time. Any failures during the release process further delayed the release as the issue needed to be fixed and the job re-triggered.

Once we successfully implemented our mobile CI on Bitrise, we turned our attention to optimizing our CD pipeline with Bitrise's help as well. As part of this effort, we redesigned the complete CD pipeline and leveraged powerful Bitrise Linux and macOS servers to greatly improve our release times in the new CD pipeline. The new release pipeline gave us immense performance gains, allowing us to roll out a release 75% faster than our previous pipeline.

Today, we’re able to do an over-the-air (OTA) release in less than 15 minutes from the moment it’s triggered.

Release notifications

Just as with mobile CI, developer experience is always at the heart of Rippling’s efforts. To provide the best devX in the industry, we created a notification system for our CD pipeline as well.

Auto-generate release notes

To further enhance our devX and maintain thorough documentation of every mobile app release, we generate automated release notes based on the commit messages and PR titles of the developers that have contributed to the release. We achieve this by following the Conventional Commits specification to create human and machine-readable, meaningful commit messages.

The road ahead

We’ve come a long way in our mobile CI/CD journey and have been able to set up a reliable and efficient end-to-end pipeline for both Android and iOS platforms. By embracing Bitrise as a mobile-first CI/CD solution, Rippling has successfully overcome the challenges of testing its mobile app at scale. The robust testing pipeline ensures that every commit to a pull request undergoes a comprehensive set of checks, guaranteeing a high-quality user experience for Rippling’s growing user base.

Despite this work, there are still some gaps in our current testing framework. While we are able to reliably test most of our use cases using Android emulators and iOS simulators, there are still some native functionalities of the app that can only be tested reliably using real Android/iOS devices. We also haven’t introduced any form of quality checking during the app release process, which can increase the chances of a bug being introduced after multiple changes are merged into the master branch around the same time.

To find out how we solved the above-mentioned problems and other mobile infrastructure challenges, stay tuned for the next couple of blogs in this series:

• Real device testing with AWS device farm
• Smoke tests and canary testing pre/post mobile app release

Disclaimer: Rippling and its affiliates do not provide tax, accounting, or legal advice. This material has been prepared for informational purposes only, and is not intended to provide or be relied on for tax, accounting, or legal advice. You should consult your own tax, accounting, and legal advisors before engaging in any related activities or transactions.