I wonder if someone in this forum uses AppThwack service with Appium. It would be good if you share your experiences here. As far as I know, AppThwack is the only service that provides possibility of Appium Cloud testing on real-devices, Sauce Labs only has possibility to test on emulators. We recently started to run our tests in AppThwack but found that the testruns are not stable and very slow compared to testruns on local Appium server. For example we often get sporadic error 502 (bad gateway) responses from their server when running tests. Unfortunately their support is also not very responsive. So would be really interesting to know if other people have better experience and can provide some tips/advice.
I had a similar experience with AppThwack. There are a bunch of other providers that support appium on real devices. I liked appurify out of all the ones I tried.
I also had a similar experience. I was only able to run tests sequentially (seems like they limit the number of requests, so running in parallel led to many server errors) and it was very slow. I’ve heard good things about Appurify, but because they got acquired by Google, I’m not sure if they’re accepting customers until next year.
Have you guys tried Appium on Testdroid yet? http://www.testdroid.com/
More info and step-by-step guide here:
http://testdroid.com/testdroid/6993/appium-on-real-devices-appium-faq
https://testobject.com/
much cheaper than Saucelabs as well
Another option is to build an internal device cloud. Saves a bunch of $$.
1 Like
Hi everyone. Trent from AppThwack here.
First, I’d like to apologize for delayed support. We built our company upon user experience and customer support, so to hear we’ve fallen short on either of those fronts is extremely troubling. As we’ve become more and more popular, support has become inconsistent. Sometimes we reply in minutes and sometimes it can be a day or two. We’re also guilty of investigating a ticket but not updating the ticket to reflect that there is activity going on behind the scenes. To address these issues we’re introducing monitoring on our side to ensure tickets are responded to promptly, even if just to acknowledge, as well as hiring an engineer to help with customer support.
Now, to the topic at hand: Appium. Appium is unique for us (and our competitors who’ve graciously joined this conversation) in that rather than executing the scripts for the user, we provide a direct session with the devices. This is challenging for a number of reasons.
- Timing is a challenge as the initial request for a session varies by how busy our lab is. We’re addressing this by scaling up on the number of available duplicate devices, but it’s still not 100%.
- Delays between calls from the user to the device are atomic and, given the fact the devices are remote, can introduce timing issues within the test scripts themselves.
- Appium client stability is also an issue. Depending on the client, some errors get swallowed or covered up by the client. For instance, the java client routinely throws a null ref error that covers up the actual server response. Furthermore, generic 502 errors are generally caused by unexpected application or Appium errors. Many of them are related to device specific application crashes that the user might not be aware of.
- With client-side Appium, parallel execution is the responsibility of the user. We believe in making the parallelism in execution an automated action, and even opening multiple local sessions for you via a command line tool leaves a bad taste in my mouth. It may work for a few devices, but what about hundreds? We can handle that scale easily, but for a user? Not so much.
- Since the tests are completed on the client side, we do not know if the tests have passed or failed, meaning our reports are simply high-level logs of overall execution. To make matters worse, unlike our server-side frameworks, with Appium we don’t know whether we should re-run the tests for you if execution encounters infrastructure problems like device popups, connectivity issues, and so on until after all of your tests have run or errored out completely.
We have a server-side Appium implementation in private beta where you upload your tests to us and we take care of parallelizing the execution. The challenge is supporting the myriad languages and test environments Appium users have built. If you’d like to learn more and check it out, please email me at trent-at-appthwack-dot-com.
I’d love to hear more about your experiences with both us and our competitors. At the end of the day we’re trying to provide services to make the lives of developers and QA folks easier. If not, we’re failing at our sole mission. I apologize for the experience you’ve had, and I’d be grateful to get a chance to make it right.
4 Likes
Sauce Labs has real devices in a private beta currently, and will release it to the public soon ^.^
1 Like