[Appium] How to find element only by image?

Abush · February 24, 2019, 1:00pm

Hi,

I am using appium latest (1.11.1)
Working on Win 10
IDE tool: Intellij
Writing in JAVA
java client - 7.0.0
Maven project

I am trying to find element by image only by using: driver.findElementByImage("path to png image)
Can someone simplified the how to make it work ?
The app is built with opengl, so no element exsist on the screen at all,
This is Android app, any help will be appriciate, tutorials are very general in this subject,

Thanks

Abush · February 26, 2019, 6:11am

Anybody?
Nobosy knows any tool that works with appium to catch element by recognize image?

Aleksei · February 26, 2019, 4:47pm

Write what you have tried and problem met. Hardly we can help without knowing your problem

Abush · February 26, 2019, 6:28pm

I tried the appium find by image but The appium never recognize the opencv in the npm, I have tried every method to Install opencv, it installed auccess but appium doctor keep. Saying that I need to install opencv.
I have tried sikuli but it always fails.
So I wanted to know if there is working method to fond element by image?

What I need:
Sample code that use exist screenshot to find it on the screen(as element).

Thanks

Aleksei · February 26, 2019, 9:30pm

there is guide https://appium.readthedocs.io/en/latest/en/writing-running-appium/image-comparison/#image-comparison-features
step ONE is have OpenCV 3+ native libraries. You need install it on your MAC/PC.

it is not quite obvious. but it is. follow guide from official site: e.g. with Linux: https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html

Abush · February 28, 2019, 12:41pm

Hi Aleksei,

I have already installed the opencv through npm,
When I run any driver.findByImage(“path.png”)
I get this error from appium server:

mime must be a string
[W3C] Encountered internal error running command: Error: opencv4nodejs module is required to use OpenCV features. Please install it first (npm i -g opencv4nodejs) and restart Appium. Read https://github.com/justadudewhohacks/opencv4nodejs#how-to-install for more details on this topic.
[W3C] at initOpenCV (C:\node_modules\appium\node_modules\appium-support\lib\image-util.js:100:11)
[W3C] at Object.getImageOccurrence$ (C:\node_modules\appium\node_modules\appium-support\lib\image-util.js:420:3)
[W3C] at tryCatch (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:67:40)
[W3C] at GeneratorFunctionPrototype.invoke [as _invoke] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:315:22)
[W3C] at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at invoke (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:136:37)
[W3C] at enqueueResult (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:185:17)
[W3C] at new Promise ()
[W3C] at new F (C:\node_modules\appium\node_modules\core-js\library\modules$.export.js:30:36)
[W3C] at AsyncIterator.enqueue (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:184:12)
[W3C] at AsyncIterator.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at Object.runtime.async (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:209:12)
[W3C] at Object.getImageOccurrence (C:\node_modules\appium\node_modules\appium-support\build\lib\image-util.js:632:30)
[W3C] at AndroidDriver.callee$0$0$ (C:\node_modules\appium\node_modules\appium-base-driver\lib\basedriver\commands\images.js:47:32)
[W3C] at tryCatch (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:67:40)
[W3C] at GeneratorFunctionPrototype.invoke [as _invoke] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:315:22)
[W3C] at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at invoke (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:136:37)
[W3C] at enqueueResult (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:185:17)
[W3C] at new Promise ()
[W3C] at new F (C:\node_modules\appium\node_modules\core-js\library\modules$.export.js:30:36)
[W3C] at AsyncIterator.enqueue (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:184:12)
[W3C] at AsyncIterator.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at Object.runtime.async (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:209:12)
[W3C] at AndroidDriver.callee$0$0 [as compareImages] (C:\node_modules\appium\node_modules\appium-base-driver\build\lib\basedriver\commands\images.js:55:30)
[W3C] at AndroidDriver.condition$ (C:\node_modules\appium\node_modules\appium-base-driver\lib\basedriver\commands\find.js:111:26)
[W3C] at tryCatch (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:67:40)
[W3C] at GeneratorFunctionPrototype.invoke [as _invoke] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:315:22)
[W3C] at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:100:21)
[W3C] at GeneratorFunctionPrototype.invoke (C:\node_modules\appium\node_modules\babel-runtime\regenerator\runtime.js:136:37)

I am mantion that I already use install opencv in various ways:
npm install --save opencv4nodejs
npm i -g opencv4nodejs

And also tried mannually, the appium doctor keep saying:
WARN AppiumDoctor opencv4nodejs cannot be found

But I got indication that the opencv is installed success when install from npm,
maybe Windows 10 path is wrong ? I add mannually all the pathes that I could,
I don’t know what else to do in order that the npm/Windows recognize the opencv and activate him.

Any help will be appriciate!

Aleksei · February 28, 2019, 4:39pm

@Abush once again. With npm you installing libraries which used to translate your java commands into commands that uses openCV. To install openCV follow guide. In general ot has 2 steps: download sources and compile then on your mac/pc.

PS use link in error to install openCV

Abush · February 28, 2019, 7:18pm

@Aleksei
I did it, I follow the guide…
I don’t understand why it fails as I attached the info,
I did exactly as the guide but it still fails,
Can you direct me more spesific?

Aleksei · February 28, 2019, 8:13pm

So you did “cmake” command and compiled openCV. Correct me if i am wrong.

Abush · February 28, 2019, 8:30pm

Cmake is installed and windows path works with cmake,
How to compile? Did I miss something?

Aleksei · February 28, 2019, 8:33pm

i do not use image any more. last time i was installing it - i used guide from https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html with macOS machine

try also: https://github.com/appium/appium/issues/11865

Abush · February 28, 2019, 8:44pm

So if not image then what are you using?

Aleksei · February 28, 2019, 8:51pm

just do not need search by image any more. another project

Abush · February 28, 2019, 8:58pm

Any other suggestions?
How can I found images?

Aleksei · February 28, 2019, 9:30pm

i suggest fix your case. other case harder.

check you installed appium also with “npm install -g appium”
check appium version up to date
i checked “npm i -g opencv4nodejs” and looks like it DOES ALL needed. it compiles also openCV libraries.

Abush · February 28, 2019, 9:43pm

I did all of this… Exactly as you write…

Aleksei · February 28, 2019, 10:22pm

do we have exact appium command that failing?

Abush · March 1, 2019, 3:43am

Every usage of opencv fails

mykola-mokhnach · March 1, 2019, 5:51am

Sometimes providing a correct value to NODE_ROOT environment variable solves module resolution issues.

Abush · March 1, 2019, 6:45am

How can I do it?
Appium is installed through npm global and works great, I use the appium server from java code, but every time I tried to use opencv it fails as I mention above.