Hello everyone,
I’m currently facing an issue with Appium while attempting to automate interactions on an iPhone that has screen mirroring enabled. Specifically, when I enable remote control(Apple watch mirroring) on my iPhone, and then try to automate interactions using Appium, I encounter difficulties in clicking elements on the remote screen.
Problem Statement:
- When I enable screen mirroring on my iPhone, I can see the mirrored screen on another device (e.g., Apple Watch) and can also view the mirrored screen in the Appium inspector.
- However, despite being able to see the elements in the Appium inspector, I’m unable to interact with them programmatically using the typical WebDriver commands (e.g.,
click()). - It seems that Appium is unable to recognize or interact with the elements on the mirrored screen properly.
Steps to Reproduce:
- Enable screen mirroring on an iPhone. (Settings → Accessibility → Apple Watch Mirroring → toggle on/off button)
- Apple Watch screen mirroring is enabled on iPhone
- Launch the Appium inspector and connect to iPhone.
- Attempt to interact with elements on the mirrored screen using Appium WebDriver commands (e.g.,
click()).
Expected Behavior:
- I expect Appium to recognize and interact with the elements on the mirrored screen as it does with the regular device screen.
- By xpath or cordinate should be able to click on elements on the mirrored screen on iPhone without any issues.
Actual Behavior:
- Despite being able to see the elements in the Appium inspector, the WebDriver commands fail to perfome click on element which we see on remote screen iPhone.
- The
click()command, for example, does not trigger any action on the mirrored screen. - The
click()command works only for “digital crown” or “side button”, but it doesn’t work for screen elements - I encounter the same issue when I try to perform a click using the Appium Inspector or execute the same action in a Python script
Here is example from Appiun inspector
Additional Information:
- I’ve tried using different Appium versions and configurations, but the issue persists.
- I’ve also experimented with various capabilities and settings, but none seem to resolve the problem.
- Is there a specific configuration or capability that needs to be set when dealing with screen mirroring scenarios in Appium?
Environment:
- Appium version: 2.5.2
- Appium insector: 2024.3.4
- iOS version: iOS 17.4.1
- WatchOS: 10.4
- Device: iPhone 15 and Watch Ultra 2
- Screen Mirroring Device: Remote control (screen mirroring) enabled on the iPhone, I control Apple Watch with iphone
I would appreciate any insights or suggestions on how to resolve this issue. Thank you in advance for your help!