Hello, I just discovered appium and think that it might be able to replace some code I’ve been writing, however I am not sure whether it can cover everything for me. I am automating a game that uses a niche game engine (cannot use the Unity plugin). For that reason, significant image matching is used.
I have read the docs and press release. Unfortunately since this plugin is new there is limited info in the wild.
- is there a
findElement**s**ByImage / find all?
- is screenshotting used (can be laggy), or is the device screen streamed? Inspired by my Selenium background, I often I use an explicit-style wait (‘click on ready’) with high-frequency polling, made possible by streaming (checking whether an image exists maybe 10 times a second). [See scrcpy]
- are there ways to control grayscale vs. color matching? In most cases I need grayscale (puts more emphasis on pixel correctness rather than slightly different areas with the same colors) but sometimes I need color (same buttons, different colors)
- is there a way to change the OpenCV match method (
cv.TM_SQDIFF_NORMED is good for me)?
- is there a way to specify in coordinates the screen area being searched, or the bounding box?
Assuming none of these are currently possible (please confirm), they could all be feature requests, but there might be some design limitations I am not aware of (please comment). For example already I am not a fan of having to send base64 images over HTTP en masse, but that is an acceptable design trade-off of webdriver. I know image matching is already a stretch for the scope of webdriver, and low priority, hence why it was not added until recently. The “webdriver way” would be to write integrations for every game engine, for example, however that is not realistically possible.
Bonus Unity question out of curiosity: for games that can be driven using the Unity plugin, is input control live (suitable for automation) or delayed (more suitable for testing)?
You can check appium/packages/images-plugin/lib at master · appium/appium · GitHub and appium/packages/opencv/lib/index.js at master · appium/appium · GitHub for more implementation details.
I believe some parts that are currently using WebDriver protocol might be reimplemented using BiDi websocket approach.
If you are really interested about contributing to this plugin and being serious about it then we could have an online conversation where I would be happy to explain the stuff I know about. It is also advisable to join our Slack, where we discuss development-related questions.
1 Like
Thanks. I have no problem contributing to things I use. As a hobbyist who is not currently an appium user, this is a bit of a hard sell, especially since it seems there are more than just 1-2 things I’d need to migrate. I have also been burned before working on open source, as it’s not my project and I cannot control the code base (merge), I would need strong support from at least one contributor. I guess I was hoping to understand the landscape a little more.
I guess the Slack would be a good place for that, but it has a strange email address domain name requirement so it seems I am not able to lurk.
I would have no problem rolling the easy features here - find all, grayscale, match method, and bounding box. I have 2 “major” issues: streaming the screen - especially getting a good scrcpy client library (in Node?), and actually really wanting to put template images on the server and refer to them by relative path instead of necessarily sending them b64 over HTTP. Since all of the issues are MVP for me I’d want to get the scrcpy one sorted out first.
appium is heavy for my use case - I don’t feel particularly obligated to the added complexity it requires (for me, mostly result of the uiautomator dep), I just prefer using Other People’s Code. I guess I will probably continue on my own for now. Cheers
No problem. I’ve invited you to Appium just in case you’d need to ask any dev related questions.
I would need strong support from at least one contributor
I would be happy to help. Appium development has also been my hobby for quite a while, I don’t contribute to it as part of my regular job, rather just for having some fun and technical challenge.
1 Like