How can we identify any image on mobile app using Appium?

Support
1

Hi All,

I am using a Gaming app written in cocos2dx. I cannot identify the element by its ID. Can we identify the element by image using appium? If so how?

I tried using Coordinates taken by appium UI inspector but it does not always clicks on the correct button. Please help I am stuck.

I have used something like this-
js.executeScript(“mobile: tap”, new HashMap<String, Double>() {{ put(“tapCount”, (double) 1); put(“touchCount”, (double) 1); put(“duration”, 1.3); put(“x”, (double) 91); put(“y”, (double) 75); }});

2

Hi Komal,

Are there any other properties for the element other than ID, you can try using Xpath or CSS ?

Thanks,
Sujata

3

Hi Sujata,

when using appium UI Inspector it identify the entire app not one single object. Refer this screenshot - http://screencast.com/t/8AAeFgpwX

It shows the xpath but it remains same for all the screen in my app.

4

@Komal_Gupta You can take screenshots with Appium, but it does not contain logic to compare them. Easiest approach is to compare an Appium screenshot to the reference screenshot using a 3rd party library.

5

Hi.
we successfully integrated Appium and Sikulli (image recognition library). Some code examples to start i mentioned here: https://github.com/appium/appium/issues/943

6

How to take screenshot using Appium? Also suggest any 3rd party library for comparison

7

How did you take screenshot of a mobile app using sikuli?

8

i have not worked on sikuli…

Anyhow to take screen shot u can simply copy and paste the below code…

WebDriver driver1 = new Augmenter().augment(driver);
File file = ((TakesScreenshot) driver1)
.getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(file, new File(“Screenshot.jpg”));

Thanks,
Sudhanva

9

And the screen shot will be present in root of your source code…

thanks,
Sudhanva.M

10

sikuli does not taking screenshot but comparing images only. use takeShot() which i mentioned here FindByImage Support · Issue #943 · appium/appium · GitHub as pluss it does rotation when image in landscape.

11

@Komal_Gupta You can use Applitools Eyes for doing automatic image comparison with Appium. In fact Applitools automates not only the capturing of the screenshot, but also it’s validation, and has sophisticated algorithms to avoid false failures of the image comparison. Applitools’ Selenium SDK works out of the box with Appium as well as with any other WebDriver implementation, and is cross-platform and cross-device. Applitools have a free registration plan, and the SDKs are open source and available on GitHub.

Full disclosure: I’m part of Applitools core team.

12

Hi, how do you integrate Appium and Sikuli? Please give more details, thanks.

13

This just takes a screenshot of the whole screen which is not very useful for Appium as it’s not a browser running fullscreen.

14

Here is example of how You can do it
enter link description here

1 Like
15

This is really awesome. Using this I am able to Achieve image identification using appium.

@Test
public void identifyImageTest() throws InterruptedException {
String giftsAwaitImgLoc = “imagetest.png.PNG”;
Thread.sleep(7000);
OCR.clickByImage(giftsAwaitImgLoc);
}

16

Hi Tapan,

I have tried same code however getting below error. Could you resolve my error?

Environment: Win10
Appium 1.4.16
Device: Samsung Galaxy Note 5
Language: Java
Sikulix api 1.1.0

Code:
String giftsAwaitImgLoc = imgDir + “\LTM_menu.png”;
String acceptAllImgLoc = imgDir + “\LTM_spin.png”;

    //String giftsAwaitImgLoc = "imagetest.png.PNG";
    Thread.sleep(7000);
    OCR.clickByImage(giftsAwaitImgLoc);

Error:
[debug] RunTimeINIT: addToWindowsSystemPath: added to systempath:
C:\Users\ak49337\AppData\Roaming\Sikulix\SikulixLibs_201510051707
[debug] RunTimeINIT: checkJavaUsrPath: added to ClassLoader.usrPaths
***** show environment for API (build 201510051707)
user.home: C:\Users\ak49337
user.dir (work dir): D:\Image\AppiumFindByImage-master
user.name: AK49337
java.io.tmpdir: C:\Users\ak49337\AppData\Local\Temp
running 64Bit on Windows (10.0) from a jar
java 8-64 version 1.8.0_91-b15 vm 25.91-b15 class 52.0 arch amd64
app data folder: C:\Users\ak49337\AppData\Roaming\Sikulix
libs folder: C:\Users\ak49337\AppData\Roaming\Sikulix\SikulixLibs_201510051707
executing jar: C:\SikuliX\sikulixapi.jar
*** classpath dump sikulix
7: /C:/SikuliX/sikulixapi.jar
61: /C:/Users/ak49337/.m2/repository/com/sikulix/sikulixapi/1.1.0/sikulixapi-1.1.0.jar
68: /C:/Users/ak49337/.m2/repository/com/sikulix/sikulixlibswin/1.1.0/sikulixlibswin-1.1.0.jar
*** classpath dump end
***** show environment end
[debug] RunTimeAPI: global init: leaving
[debug] RunTimeAPI: initAPI: entering
[debug] RunTimeAPI: initAPI: leaving
[debug] RunTimeAPI: loadLib: VisionProxy.dll
[debug] Image: loaded: D:/Image/AppiumFindByImage-master/src/main/resources/LTM_menu.png (file:/D:/Image/AppiumFindByImage-master/src/main/resources/LTM_menu.PNG)
[debug] Image: cached: D:/Image/AppiumFindByImage-master/src/main/resources/LTM_menu.png (13 KB) (# 1 KB 13 – 0 % of 64 MB)
PASSED: AsetUp
PASSED: ZtearDown
FAILED: BgatherAllGiftsTest
org.openqa.selenium.ElementNotVisibleException: Element not found - D:\Image\AppiumFindByImage-master\src\main\resources\LTM_menu.png
Build info: version: ‘unknown’, revision: ‘86a5d70’, time: ‘2017-02-16 07:47:51 -0800’
System info: host: ‘DERIW50Z0119’, ip: ‘10.44.21.83’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_91’
Driver info: driver.version: unknown
at OCR.clickByImage(OCR.java:64)
at OCRNekoAtsumeTestSuite.BgatherAllGiftsTest(OCRNekoAtsumeTestSuite.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
at org.testng.TestRunner.privateRun(TestRunner.java:756)
at org.testng.TestRunner.run(TestRunner.java:610)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
at org.testng.TestNG.runSuites(TestNG.java:1133)
at org.testng.TestNG.run(TestNG.java:1104)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:236)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:81)

===============================================
Default test
Tests run: 3, Failures: 1, Skips: 0

===============================================
Default suite
Total tests run: 3, Failures: 1, Skips: 0

[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@2237bada: 20 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@1162410a: 6 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@28b576a9: 41 ms
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 4 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@37ff4054: 6 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@74cf8b28: 7 ms
[debug] RunTimeAPI: final cleanup
[debug] FileManager: deleteFileOrFolder:
C:\Users\ak49337\AppData\Local\Temp\Sikulix_9314725

17

I have implemented the SikuliX 1.1.1 jar with Appium 1.6.5 server and the java client version 5.0.0-BETA9 here - Appium SikuliX OCR