Real Device CI/CD pipelines with Appium

Tutorials/Documentation
1

I am not seeing many useful resources on working CI with Appium, either with an emulator or real device. At the moment, I am solely targeting Android, but plan on doing iOS in the future as well.

As some or most of you have noticed, it is incredibly hard to set up good CI with Android Emulators. You need hardware acceleration, and a myriad other things to get the Android environment running. That in itself is a challenge. So let me share my experience.

My case is as follows. I have an app that connects to a staging environment. I need to be very careful on what data I change because I can’t just reset the database or seed it. It’s a limitation I can work around for now. But even so, most apps nowadays connect to an external server, so the constraint for CI I had was:
A. It must have hardware acceleration for the emulator to work.
B. It must not have network throttled so I can log in, send messages, etc… basically communicate with the BE server.

I am using WebDriver.io with the Appium service because the app I am testing is React Native and I wanted to not diverge from the developer’s stack.

Let’s go through each environment and my conclusions.

  1. Local execution. Generally speaking, local execution on my machine (i7 7700k proc + GTX 1070 on Ubuntu 22.04) execution is stable. Tests are passing either on an Android emulator (using Pixel 5 for now) or on my Android device connected to my machine. Developers tried it out too. With minor issues, they are able to run the suite on their local machines. Great success!

  2. Github Actions. Thankfully, they enabled hardware acceleration (CPU) on Github CI, so the emulator can actually run on a Linux machine. Previously, only macOS provided runners were able to run the emulator, but they cost 10x. After getting an 8-core runner, I was able to run the tests, but they are very flaky. I’ve yet to find out why. At some point, execution just hangs, emulator dies, something bad happens in the middle. I’ve been using https://github.com/reactivecircus/android-emulator-runner/tree/v2/ as a base action.

  3. Pricing is $250/mo for 1 real device for both Browserstack and SauceLabs. I haven’t tried SauceLabs yet, but tried Browserstack. SauceLabs has a limited amount of devices unless you get a private cloud with dedicated devices, Browserstack doesn’t. On the other hand, Browserstack doesn’t support the latest version of Appium (2.1.3). Latest it supports is 2.0.1. I have no idea about SauceLabs.

In terms of CI/CD it feels like I’m missing something. Is there really no good and easy solution for functional automated testing? I’m looking to learn from everyone else’s experience as well.

2

This is the pipeline we use to run end-to-end tests on Android Emulator: https://github.com/appium/appium-uiautomator2-driver/blob/master/.github/workflows/functional-test.yml

We use free-teer macOS instances. They could sometimes be slow, but the pipeline itself is pretty stable.

According to my exp it makes sense to setup a local env to run emulator or simulator-based tests. It is enough to validate most of the app functionality and it would fast enough to parallelize your test suites in comparison to (fast=expensive) cloud CI. Think about centralized test runners config administration (e.g using Ansible) - this might make the life much easier in the future.

As for real devices lab I think it is a pain to have such in-house. Although, there are some device APIs that are not presented on emulators/simulators or work differently. That is why for me it makes sense to have a very carefully selected/small set of tests specific for real devices. Then run these tests on devices that are most popular across your user base. Also, cloud-based real devices are good to verify device-model-specific issues manually, because a cloud provider usually has a big choice of different models and updates them fast.

3

Thanks for the advice! I will try to check out the action you’re using. I also wanted to go with emulators first, since there’s not a huge difference between emulators and actual devices nowadays, even for native apps, rather than web apps. As I said, I am testing a React Native app, which basically means native app. That’s where all the flakyness comes from. The screen just turns black at some point for no easily identifiable reason.