Touch Actions and Mobile Web

[vibro]

Hi all,

I’m working on testing a mobile website for both iOS and android. I knew the methods that tapped or clicked on a specific WebElement on a web browser didn’t work, so I made a workaround where I just tapped the location of the element, adjusted a bit for the screen. This solution worked in most cases.

However, I recently updated the java-client to 2.0.0 and the appium server to 1.2.3. I’m still using Selenium version 2.42.2. On iOS now, no touch actions work for me, I get a Not Yet Implemented error. I’ve also tried driver.tap(fingers, x, y, duration); but that also throws a Not Yet Implemented error. Has the way TouchActions are handled changed in the most recent release? What is the best way to do mobile web tapping, on Android or iOS?

Selenium 2.42.2
Java-client 2.0.0
appium 1.2.3

Mac OS 10.9.4
iOS Emulator 6.1 (because of the issues with scrolling in 7.x)

Thanks

[Hassan_Radi]

Did you change how you define your driver???

I mean did you know that Java Client has a design change:

There is an abstract AppiumDriver class which inherits from the Selenium Java Client. The AppiumDriver class contains all methods shared by iOS and Android. IOSDriver and AndroidDriver both extend AppiumDriver and provide more methods, and specific implementations for some methods.

[vibro]

Yes, I changed that! It’s creating an iOSDriver when I launch my tests.

[jonahss]

If you’re automating a webview, the commands all get sent to the webdriver, through the appium driver. So if safariDriver doesn’t implement the new touchAction endpoints, it’ll spit back a NotYetImplemented error.

What you need to do is switch contexts to the NATIVE_CONTEXT (using the contexts() methods). Run your touch actions, then switch back to the webview context.

Nothing should have changed with how it worked from before, but a couple people have run into this. I’ll take a closer look.

Tell me if the above works for you

[jonahss]

Yeah touch actions on a webview work for me, if I stay in the native_webview context…

[Sarfaraz_Momin]

hi,
i have switch as you said but its getting hanged when control come on perform() method…

[vibro]

So if I’m automating a browser (not a webview in a hybrid app) I need to change the context to NATIVE_CONTEXT to allow touch actions?

[jonahss]

That is correct! ccc

[Dan_Frei]

Hi,

I’m also facing this issue.
I’m running the tests on a real android device (galaxy tab3) and whenever I’m trying to use Touch Actions (e.g. driver.swipe() or driver.pinch()) I’m getting this:Not Yet Implemented error.
I’m using:
Selenium 2.42.2
Java-client 2.0.0
Appium 1.2.4

Device:Galaxy Tab3
OS: Android 4.4.2
Browser: Chrome 38

BTW:
I’ve tried switching the context to “NATIVE_APP” and yet it still doesn’t work (I get a different error:An unknown server-side error occurred while processing the command).

Any Ideas?

Thanks in advance,
Daniel.

[jonahss]

Definitely switch to the NATIVE_APP context, and look at the appium logs there. The unknown error could be something like trying to swipe out of bounds or something.

[Dan_Frei]

Hi Jonahss,

Thank you for the assistance!
Eventually I’ve been able to successfully execute TouchActions (swipe, tap etc…) by switching the driver to NATIVE_APP as you’ve suggested.

I now understand, that currently, appium supports Touch Actions only in native context (otherwise, I got a "Not implemented yet…error).
Unless I’m wrong, this means that if one desires to utilize the the benefits of TouchActions while testing a Mobile Web, he should get the coordinates of the WebElement he wants to manipulate via a TouchAction, switch to Native context, execute the touch command via an api that accepts X,Y coordinates and then, switch the driver back to the previous context like in the following pseudocode:

WebElement element = driver.findElement(By.id(“testElement”));
Point point = element.getLocation();
Dimension size = element.getSize();

int elementCenterX = point.getX() + Math.round(size.getWidth() / 2);
int elementCenterY = point.getY() + Math.round(size.getHeight() / 2);

String originalContext = driver.getContext();
driver.context(“NATIVE_APP”);

driver.tap(1, elementCenterX , elementCenterY , 1000);

driver.context(originalContext );

Unfortunately, I still face problems trying to utilize TouchActions for mobile-web testing purposes.
The major problem, is the difference between the X,Y reference point that is used in “NATIVE-APP” context and the one that is used in “WEBIEW”.
It looks like in “NATIVE-APP” mode, the driver regards the X,Y coordinates relatively to the physical screen in device pixels (screenXY), whilst in “WEBVIEW” mode, the driver regards the X,Y coordinates relatively to the to the top left of the fully rendered content area in the browser in CSS pixels (pageXY).

This means, that in-order to make things work, we need to somehow transform the aforementioned WebElement’s PageXY coordinate to a screenXY coordinate.

I’ve tried several approaches to solve this issue - unfortunately without success.

Approach #1(use org.openqa.selenium.internal.Locatable to get the “onScreen” coordinate):
Point onScreen = ((Locatable)element).getCoordinates().onScreen();
Result: Fails - I get an error saying that AppiumDriver doesn’t support the method: onScreen()

Approach #2 (Calculate the offset between the two reference points)
In order to do so, one of the basic things we need to do is getting the window size:
driver.manage().window().getSize();
Result: Fails -I get an error saying that AppiumDriver doesn’t support window().

Would appreciate your kind advice!

Thanks again and Best Regards,
Daniel.

[jonahss]

Aha, that is a good point. We should find a way to transform the coordinates. Can you create an issue on the githug repo please?

[Dan_Frei]

Thanks jonahss!

I’ve created an issue in the github as requested:

Support Touch Actions in web/webview context.

Thanks & Best Regards,
Daniel.

[vibro]

I’ve been getting around this by hardcoding an offset, but it’s not the most reliable method.

[Sarfaraz_Momin]

hi,
actually, I have faced this kind of error when i was using like e.g. tap(button).perform()
then I used (X,Y) co-ord but Y co-ord is not correctly located on screen while X co-ord is correct so i add some offset (in my case i add 100 i.e. Y+100) then its worked for me.
But i dont think this is the reliable way to do this job.

[maxski]

Hi guys!
Has anybody got it resolved? Please share your results as hardcoded Y offset couldn’t be worked for different mobiles as they have different resolutions.

[maxski]

Eric knows another workaround, how to get this resolved.
But it also works unstable:-(

Get screen and browser size

Hello, I’m using Appium in Ruby on Android real device Galaxy S4 I’m looking to get the screen and browser size but right now I’m using, after opening my webpage on Chrome: @browser.execute_script(“return window.innerWidth”).to_i @browser.execute_script(“return window.innerHeight”).to_i @browser.execute_script(“return window.outerWidth”).to_i @browser.execute_script(“return window.outerHeight”).to_i @browser.execute_script(“return screen.width”).to_i @browser.execute_script(“return scree…

[pess]

public static void tap(WebElement we) {
    Point p = we.getLocation();
    Dimension size = we.getSize();
    try {
      nativeContext();
      Point appLoc = driver.findElement(By.xpath("//UIAWebView[1]")).getLocation();
      driver.tap(1, appLoc.x + p.x + (size.width / 2), appLoc.y + p.y + (size.height / 2), 1);
    } finally {
      webContext();
    }
  }

[maxski]

Hi,

This doesn’t work for me:

Point appLoc = driver.findElement(By.xpath(“//UIAWebView[1]”)).getLocation();

org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command.

[maxski]

Smth that worked on one phone. failed on another…
Is there any ideas how to get correct coordinates for tap?

@jonahss
Looks, this can be fixed in 1.3.2. When are you going to release 1.3.2?