A/B testing strategy

Tutorials/Documentation
1

Not related to Appium , but as this is mobile automation discussion board I’ll give a try :slight_smile:

Recently we started to run A/B test on our app , and I was wonder what are the strategy to write automation which run in parallel to the process of A/B test.
Is there a way to omit specific versions from A/B test or run A/B test on specific version ?

2

@igal_epshtein we made 2 approches:

  • we have both variants in tests whatever met
  • we added ability with client to read start arguments and set a/b flag whaever requested on open driver. Appium has capability to send arguments on driver start. Valid for both ios and android.

Thus we cover some random flow and generate needed of a/b testing.

3

Thanks for the answer

Option A is a valid one , but then you need to handle two flows.

About option B , as far as I know , the decision of which a/b test flag you get being controlled by firebase , isnt it ?

4

we have feature flags in apps. and we can control them in 3 ways:

  • by backend
  • manually in debug menu of app overwrite any server flag sent
  • on app start by sending flags in parameters (Appium automation way)
5

Interesting

Our FF controlled by firebase

How do you send the flags in parameters ?(third option)

6

In driver capabilities. E.g. for ios:

processArguments Arguments to pass to the AUT using instruments e.g., -myflag

7

and for Android ?

And how do you utilise this flags from the app ?

8

i guess this should work:

intentAction	Intent action which will be used to start activity (default android.intent.action.MAIN)	e.g.android.intent.action.MAIN, android.intent.action.VIEW
intentCategory	Intent category which will be used to start activity (default android.intent.category.LAUNCHER)	e.g. android.intent.category.LAUNCHER, android.intent.category.APP_CONTACTS
intentFlags	Flags that will be used to start activity (default 0x10200000)	e.g. 0x10200000
optionalIntentArguments	Additional intent arguments that will be used to start activity. See Intent arguments	e.g. --esn <EXTRA_KEY>, --ez <EXTRA_KEY> <EXTRA_BOOLEAN_VALUE>, etc.