Ios: Can't see Elements I can interact with

CAD_Codes · February 29, 2020, 12:54am

Up front: I’m a backend dev, who knows a bit of ruby, so apologies if this is stupidly simple.

I’m adding some appium love to our app. On the android side of things, it’s going pretty smooth.

On the IOS side, I’m not able to see any of the element IDs.

The screen before this is simple, and I can see all of the expected elements. The login screen however, gets a little weird. I can interact with the screen, but appium can’t find any of the accessibility labels.

There seems to be a giant “XCUIElementTypeOther” floating on top of everything:

And here is the XML of the screen, if that helps at all:

<?xml version="1.0" encoding="UTF-8"?><AppiumAUT><XCUIElementTypeApplication type="XCUIElementTypeApplication" name="Journi" label="Journi" enabled="true" visible="true" x="0" y="0" width="375" height="812">
  <XCUIElementTypeWindow type="XCUIElementTypeWindow" enabled="true" visible="true" x="0" y="0" width="375" height="812">
    <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="0" y="0" width="375" height="812">
      <XCUIElementTypeOther type="XCUIElementTypeOther" name="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" label="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" enabled="true" visible="true" x="0" y="0" width="375" height="812">
        <XCUIElementTypeOther type="XCUIElementTypeOther" name="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" label="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" enabled="true" visible="true" x="0" y="0" width="375" height="812">
          <XCUIElementTypeOther type="XCUIElementTypeOther" name="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" label="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" enabled="true" visible="true" x="0" y="0" width="375" height="812">
            <XCUIElementTypeOther type="XCUIElementTypeOther" name="push-notification-toast-container-view" enabled="true" visible="false" x="0" y="-110" width="375" height="110">
              <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="16" y="-30" width="343" height="22"/>
            </XCUIElementTypeOther>
            <XCUIElementTypeOther type="XCUIElementTypeOther" name="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" label="EMAIL ADDRESS PASSWORD Password Sign in Forgot your password? Don’t have an account yet? Sign up v1.1.16.680, staging Help, FAQs &amp; Privacy" enabled="true" visible="true" x="0" y="0" width="375" height="812"/>
          </XCUIElementTypeOther>
        </XCUIElementTypeOther>
      </XCUIElementTypeOther>
    </XCUIElementTypeOther>
  </XCUIElementTypeWindow>
  <XCUIElementTypeWindow type="XCUIElementTypeWindow" enabled="true" visible="false" x="0" y="0" width="375" height="812">
    <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="0" y="0" width="375" height="812">
      <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="0" y="812" width="375" height="54">
        <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="0" y="812" width="375" height="54"/>
      </XCUIElementTypeOther>
    </XCUIElementTypeOther>
  </XCUIElementTypeWindow>
  <XCUIElementTypeWindow type="XCUIElementTypeWindow" enabled="true" visible="true" x="0" y="0" width="375" height="812">
    <XCUIElementTypeStatusBar type="XCUIElementTypeStatusBar" enabled="true" visible="true" x="0" y="0" width="375" height="44">
      <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="0" y="0" width="375" height="44">
        <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="0" y="0" width="375" height="44">
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="6" y="-5" width="180" height="12"/>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="190" y="-5" width="180" height="12"/>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="6" y="17" width="180" height="12"/>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="13" y="14" width="67" height="15">
            <XCUIElementTypeStaticText type="XCUIElementTypeStaticText" value="4:51 PM" name="4:51 PM" label="4:51 PM" enabled="true" visible="false" x="31" y="14" width="32" height="18"/>
          </XCUIElementTypeOther>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="21" y="8" width="56" height="22"/>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="190" y="17" width="180" height="12"/>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="294" y="14" width="67" height="15">
            <XCUIElementTypeOther type="XCUIElementTypeOther" value="No signal" name="Cellular" label="Cellular" enabled="true" visible="false" x="294" y="17" width="17" height="12"/>
            <XCUIElementTypeOther type="XCUIElementTypeOther" value="SSID, 3 of 3 Wi-Fi bars" name="3 of 3 Wi-Fi bars" enabled="true" visible="false" x="316" y="17" width="16" height="12"/>
            <XCUIElementTypeOther type="XCUIElementTypeOther" value="Not charging" name="100% battery power" label="100% battery power" enabled="true" visible="false" x="336" y="17" width="25" height="12"/>
          </XCUIElementTypeOther>
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="294" y="14" width="67" height="15"/>
        </XCUIElementTypeOther>
        <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="8" y="24" width="82" height="14"/>
      </XCUIElementTypeOther>
    </XCUIElementTypeStatusBar>
  </XCUIElementTypeWindow>
  <XCUIElementTypeWindow type="XCUIElementTypeWindow" enabled="true" visible="false" x="0" y="0" width="375" height="812">
    <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="0" y="0" width="375" height="812"/>
  </XCUIElementTypeWindow>
  <XCUIElementTypeWindow type="XCUIElementTypeWindow" enabled="true" visible="false" x="-1" y="-1" width="377" height="814">
    <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="-1" width="377" height="814">
      <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
        <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
          <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
            <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
              <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234"/>
            </XCUIElementTypeOther>
          </XCUIElementTypeOther>
        </XCUIElementTypeOther>
        <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
          <XCUIElementTypeKeyboard type="XCUIElementTypeKeyboard" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
            <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
              <XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="false" x="-1" y="812" width="376" height="234">
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="q" label="q" enabled="true" visible="false" x="-1" y="819" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="w" label="w" enabled="true" visible="false" x="37" y="819" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="e" label="e" enabled="true" visible="false" x="74" y="819" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="r" label="r" enabled="true" visible="false" x="112" y="819" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="t" label="t" enabled="true" visible="false" x="149" y="819" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="y" label="y" enabled="true" visible="false" x="187" y="819" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="u" label="u" enabled="true" visible="false" x="224" y="819" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="i" label="i" enabled="true" visible="false" x="262" y="819" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="o" label="o" enabled="true" visible="false" x="299" y="819" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="p" label="p" enabled="true" visible="false" x="337" y="819" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="a" label="a" enabled="true" visible="false" x="18" y="873" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="s" label="s" enabled="true" visible="false" x="55" y="873" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="d" label="d" enabled="true" visible="false" x="93" y="873" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="f" label="f" enabled="true" visible="false" x="130" y="873" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="g" label="g" enabled="true" visible="false" x="168" y="873" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="h" label="h" enabled="true" visible="false" x="206" y="873" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="j" label="j" enabled="true" visible="false" x="243" y="873" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="k" label="k" enabled="true" visible="false" x="281" y="873" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="l" label="l" enabled="true" visible="false" x="318" y="873" width="39" height="47"/>
                <XCUIElementTypeButton type="XCUIElementTypeButton" name="shift" label="shift" enabled="true" visible="false" x="-1" y="927" width="49" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="z" label="z" enabled="true" visible="false" x="55" y="927" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="x" label="x" enabled="true" visible="false" x="93" y="927" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="c" label="c" enabled="true" visible="false" x="130" y="927" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="v" label="v" enabled="true" visible="false" x="168" y="927" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="b" label="b" enabled="true" visible="false" x="206" y="927" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="n" label="n" enabled="true" visible="false" x="243" y="927" width="39" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="m" label="m" enabled="true" visible="false" x="281" y="927" width="38" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="delete" label="delete" enabled="true" visible="false" x="326" y="927" width="49" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="more" label="more, numbers" enabled="true" visible="false" x="-1" y="981" width="95" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="space" label="space" enabled="true" visible="false" x="92" y="981" width="96" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="@" label="@" enabled="true" visible="false" x="187" y="981" width="48" height="47"/>
                <XCUIElementTypeKey type="XCUIElementTypeKey" name="." label="." enabled="true" visible="false" x="234" y="981" width="48" height="47"/>
                <XCUIElementTypeButton type="XCUIElementTypeButton" name="Next:" label="Next" enabled="true" visible="false" x="281" y="981" width="94" height="47"/>
              </XCUIElementTypeOther>
            </XCUIElementTypeOther>
          </XCUIElementTypeKeyboard>
        </XCUIElementTypeOther>
      </XCUIElementTypeOther>
    </XCUIElementTypeOther>
  </XCUIElementTypeWindow>
</XCUIElementTypeApplication></AppiumAUT>

Thanks in advance!

mykola-mokhnach · February 29, 2020, 7:39am

You probably need to focus on accessibility fixing for the app. Check https://github.com/appium/appium/issues/6517 for more details.

If the app is not native then consider switching to web view mode

CAD_Codes · March 3, 2020, 5:05am

This was totally it, I have learned a ton about accessibility today. Thanks for the help!

pole · August 31, 2020, 4:52pm

Hi! im having the same problem in IOS with an developed in Flutter.
How do you fix this problem?

I only see class names: XCUIElementTypeWindow and a lot of XCUIElementTypeOther like in your image.

In all the screens i see the same source… is like that source is blocking the real source or not showing at all.

Thanks!

mykola-mokhnach · August 31, 2020, 6:47pm

There is a separate driver for flutter-based apps: https://github.com/truongsinh/appium-flutter-driver