Detect other app screen and get size of it

Support
1

I am using Appium Mac2 driver to automate an app that opens an intermediate screen after I click a button on the first screen, I want to get the size of the intermediate screen.

My intermediate screen looks something like this (the screen appears on top of the app):

image
image.png1200×761 109 KB

2

Since the window belongs to a different app you need to switch to it. Check https://github.com/appium/appium-mac2-driver#application-under-test-concept for more details.

Also, since it is a web browser, it probably makes sense to introduce the usage of the second driver (selenium), which would interact with the content inside the web browser

3

Hi, my app is actually a MacOS app that has an overlay(from the same app, not a web browser). I want to detect the other screen once I login.I’m using Mac2 driver since it’s a MacOS app, but I believe it does not support contexts or windows (not sure if the intermediate screen is which). Is there any workaround to detect the second screen?

login = driver.find_element(by=AppiumBy.XPATH, value='//XCUIElementTypeButton[@title="Login"]')
login.click()

# print(driver.window_handles)
# driver.switch_to.window(driver.window_handles[-1])
# driver._switch_to.context("NATIVE_APP")
# print(driver.contexts)
# UnknownCommandError: Unknown extension command "getContexts". Only setValue,click,scroll,swipe,rightClick,hover,doubleClick,clickAndDrag,clickAndDragAndHold,keys,tap,doubleTap,press,pressAndDrag,pressAndDragAndHold,source,launchApp,activateApp,terminateApp,queryAppState,appleScript,startRecordingScreen,stopRecordingScreen,screenshots commands are supported.
4

I believe it does not support contexts or windows

yes it does not

Is there any workaround to detect the second screen

it may only be detected by xctest if it is accessible. The web view itself though might still be debuggable (or might be made debuggable), so you can connect selenium to it and work like it was a normal web page.

One more possible solution that comes to my mind would be to try AppleScript to interact with the dialog, although it must still expose some recognisable controls for that.

5

Thanks for the idea. I could not find the second screen in the driver.page_source, and Appium Inspector could not detect the second screen. Can’t use selenium as both screens are actually part of a native MacOS app, not browser (I’m using Mac2 driver).

I tried your last suggestion of using AppleScript, and used XCode’s accessibility inspector which managed to detect the second screen as a dialog(which was highlighted in green, I hid app details with black boxes).

accessibilityInspector
accessibilityInspector.jpg2866×1324 285 KB
AXhierarchy
AXhierarchy.png938×610 80.1 KB

I am thinking of using the accessibility role and description shown in Hierarchy to check if the second screen exists in AppleScript, but I don’t know much about AppleScript. Will this approach work?

If anyone is good at apple script, could you advise me on the code below?
For reference, i did come up with a initial code that is throwing syntax error Expected end of line, etc. but found property.:

tell application "MyMacOSApp"
    activate
    delay 1.0
end tell

tell application "System Events"
    tell window of application process "MyMacOSApp"
        tell button "Login" of group "Login"
            click
        end tell
        set dialogWindow to first window whose accessibility role is "AXWindow" and accessibility description is "SecondDialogWindow"
        if dialogWindow exists then
            return true
        else
           return false
        end if
    end tell
end tell

After the apple script works, I can use isSecondScreenPresent = driver.execute_script("macos: appleScript", {"script": apple_script}) to check for the screen