How to do image comparison

Support
21

Thanks for the help. I have already shared above the images I am comparing. But note that the img1 is scaled smaller in the resulting screen shown in img2. I did also attempts with an extraction of the image seen on the screenshot. I will add it here in this message. But there is a conflict because the native element was not made with this image. It was made with the im1 above while using a down-scaling.

I have checked the base 64 encoding. And the result can be opened showing an image in an image editor. So it seems good. But I do not know if my javascript client can use a Buffer type instead of a string. I will try to convert buffer to string but I was not successful.

const likedImage = cv.imread(likedImagePath);
const likedImageBase64 = await cv.imencodeAsync(‘.png’, likedImage);
require(‘fs’).writeFile(‘likedImageBase64.txt’, likedImageBase64, (err) => {
// throws an error, you could also catch it here
if (err) throw err;
// success case, the file was saved
console.log(‘Lyric saved!’);
});

here is an exact extraction from the screenshot. But in an older message you can find the original image that has transparent background, and that the natice element has while down-scaling it.
occurrence1

22

FindElementByImage, do you know if it requires the native element to be made using the image, or is it enough if it looks like in the image on the screen?

The native element may contain an image with transparent background, while the image I use may use or not a black background. And the image seen on the screen has a black background.

Using openCV seems to answer more concrete questions, if an image is part of another one and without ratio difference errors. Maybe I am more productive with using raw openCV.

23

In your example above Cesarv, you use

driver.setSetting(Setting.DEFAULT_IMAGE_TEMPLATE_SCALE, 0.75f);

So maybe I am missing this when I just run the driver with selector type “-image” and a base64 encoded string (A Buffer in what I tried, not a string but I will change that).

I will look in the Java client where this setting is used.

24

If your template contains a black background, the image from the screenshot must have a black background as well. If you make the background of the template transparent, this will be ignored when comparing.

As for this:

driver.setSetting(Setting.DEFAULT_IMAGE_TEMPLATE_SCALE, 0.75f);

You need to check if you can use such a setting in JavaScript. You will probably need to provide it to the corresponding JavaScript function in Appium.

Check the findByImage() method in the lib/node_modules/appium/node_modules/appium-base-driver/build/lib/basedriver/commands/find.js directory of your node.js installation.

25

OK thanks.

I have tried with several images. An exact extraction with the right black background is one scenario. I have also tried with no scaling (take the screenshot and extract a square of exact size from it).

How can findElementByImage return an element if the image is much larger that the element and can contain several elements?

26

I have reduced the scope of the problem. I took an image of exact scale. I also tried to fix the encoding. I used the binary option. BUt I still get an element not found, and the error in the appium console about the aspect ratio. So I will continue with raw OpenCV.

const likedImagePath = './occurrence1.png';
const likedImage = cv.imread(likedImagePath);
const likedImageBase64 = await cv.imencodeAsync('.png', likedImage);
await driver.findElement('-image', likedImageBase64.toString('binary'));
27

I tend to think it would not work for Appium Desktop. At least for the server, which is located in its resources (without ugly hacks). Although you can always connect to another server running from CLI. This would be the best option.

Regarding images matching - there are two main things to mention there. If you need to check for exact occurrences then make sure the example image is exactly a part/subimage of the template image. The correct method to find the coordinates of such subimage occurrences would be the findImageOccurrence one. And in case the example image has different resolution or is not strictly a part of the template image (for example rotated or has the same shape but different color) then try the matchImagesFeatures function.

28

OK thanks without all your help I would be stuck.

Appium desktop is just to inspect the tree to find my ids and so on. I can live wihtout opencv in it. It was just for convienence so I don’t need the command line appium server.

Sure I will use find occurrence.

I was just focusing first on the getElementByImage. And all my last trials were wit han exact scale image. But yes using openCV should work, using the occurence algo.

The previous algos did not answer yes or not, it jsut gave a mazximum likeliwood with a wrong squar ein the middle.

29

I could make image detection work.

I have made a Typescript example that I would like to add on the doc page.

I will make a PR. I found the doc file.

30

here is the PR.

docs: example of image occurrence comparison in Typescript/Javascript by MagicPoulp · Pull Request #13833 · appium/appium · GitHub

And mykola, you should use the noreply private mode in github. Bots can find your email and spam you, unless you have an email you can take risks with.

Blocking command line pushes that expose your personal email address - GitHub Docs