[iOS] SendKeys to custom keyboard

zarashima · June 7, 2020, 2:51am

Hi,

Our application has a custom keyboard used to set text in specific fields. On Android, there is no problem in using sendKeys to set text, but iOS is a different story. I’ve tried to used sendKeys or element.setValue(…) or Actions etc…but after these tries Appium said it can’t set text on the disabled element which is the native keyboard in this case(custom keyboard has hidden the native keyboard one).

2020-06-07 09:48:35:573 - [WD Proxy] Got an unexpected response with status 400: {“value”:{“error”:“invalid element state”,“message”:“An element command could not be completed because the element is in an invalid state (e.g. attempting to click a disabled element)”,“traceback”:“”},“sessionId”:“65DA3771-8826-49A8-A1B0-D09F8CE67B50”}
2020-06-07 09:48:36:492 - [XCUITest] Cannot type in the text field because of InvalidElementStateError: An element command could not be completed because the element is in an invalid state (e.g. attempting to click a disabled element).
2020-06-07 09:48:36:493 - [XCUITest] Trying to apply a workaround…
2020-06-07 09:48:36:878 - [WD Proxy] Got an unexpected response with status 404: {“value”:{“error”:“no such element”,“message”:“unable to find an element using ‘class name’, value ‘XCUIElementTypeKeyboard’”,“traceback”:"(\n\t0 WebDriverAgentLib 0x000000010172e804 FBNoSuchElementErrorResponseForRequest + 312\n\t1 WebDriverAgentLib 0x000000010172e5cc +[FBFindElementCommands handleFindElement:] + 460\n\t2 WebDriverAgentLib 0x000000010170c560 -[FBRoute_TargetAction mountRequest:intoResponse:] + 220\n\t3 WebDriverAgentLib 0x00000001016f99c8 __37-[FBWebServer registerRouteHandlers:]_block_invoke + 504\n\t4 RoutingHTTPServer 0x0000000104a6a208 -[RoutingHTTPServer handleRoute:withRequest:response:] + 144\n\t5 RoutingHTTPServer 0x0000000104a6a9c4 __72-[RoutingHTTPServer routeMethod:withPath:parameters:request:connection:]_block_invoke + 44\n\t6 libdispatch.dylib 0x00000001963797d4 + 16\n\t7 libdispatch.dylib 0x0000000196327ce8…

Is there any solution for iOS in this specific case?

Aleksei · June 7, 2020, 5:41am

Hi

Add all code examples you tried as code
Add pageSource of screen in a moment you try to enter text

zarashima · June 7, 2020, 7:02am

Hi,

My tries are as below:

Use sendKeys:
element.sendKeys(input)
Use setValue:
element.setValue(input)
Use Actions:
Actions action = new Actions(driver); element.click(); action.sendKeys(input).perform();

The page source is

 <XCUIElementTypeStaticText type="XCUIElementTypeStaticText" value="yyyyyy" name="yyyyyy" label="Token Serial" enabled="true" visible="true" x="28" y="142" width="94" height="21"/>
              <XCUIElementTypeTextField type="XCUIElementTypeTextField" name="xxxx" label="xxxxx" enabled="true" visible="true" x="137" y="138" width="257" height="30"/>
              <XCUIElementTypeStaticText type="XCUIElementTypeStaticText" value="yyyyyy" name="yyyyyy" label="yyyyy" enabled="true" visible="true" x="28" y="185" width="107" height="21"/>
              <XCUIElementTypeTextField type="XCUIElementTypeTextField" name="xxxxx" label="xxxxx" enabled="true" visible="true" x="137" y="180" width="257" height="31"/>
              <XCUIElementTypeButton type="XCUIElementTypeButton" name="xxxxx" label="xxxx" enabled="true" visible="true" x="20" y="225" width="374" height="46"/>

For security reasons, I censor locator value. The one with yyyyyy locator’s value above is the input field where the custom keyboard is displayed instead of the native keyboard.

This is the source that I think it’s custom keyboard element:
<XCUIElementTypeOther type="XCUIElementTypeOther" enabled="true" visible="true" x="-1" y="515" width="416" height="222"> <XCUIElementTypeButton type="XCUIElementTypeButton" name="q" label="q" enabled="true" visible="true" x="4" y="527" width="33" height="40"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="w" label="w" enabled="true" visible="true" x="46" y="527" width="32" height="40"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="e" label="e" enabled="true" visible="true" x="87" y="528" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="r" label="r" enabled="true" visible="true" x="129" y="528" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="t" label="t" enabled="true" visible="true" x="170" y="528" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="y" label="y" enabled="true" visible="true" x="211" y="528" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="u" label="u" enabled="true" visible="true" x="253" y="528" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="i" label="i" enabled="true" visible="true" x="294" y="528" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="o" label="o" enabled="true" visible="true" x="336" y="528" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="p" label="p" enabled="true" visible="true" x="377" y="528" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="a" label="a" enabled="true" visible="true" x="25" y="578" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="s" label="s" enabled="true" visible="true" x="67" y="578" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="d" label="d" enabled="true" visible="true" x="108" y="578" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="f" label="f" enabled="true" visible="true" x="149" y="578" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="g" label="g" enabled="true" visible="true" x="191" y="578" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="h" label="h" enabled="true" visible="true" x="232" y="578" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="j" label="j" enabled="true" visible="true" x="274" y="578" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="k" label="k" enabled="true" visible="true" x="315" y="578" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="l" label="l" enabled="true" visible="true" x="356" y="578" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="△" label="△" enabled="true" visible="true" x="4" y="628" width="54" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="z" label="z" enabled="true" visible="true" x="67" y="628" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="x" label="x" enabled="true" visible="true" x="108" y="628" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="c" label="c" enabled="true" visible="true" x="149" y="628" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="v" label="v" enabled="true" visible="true" x="191" y="628" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="b" label="b" enabled="true" visible="true" x="232" y="628" width="33" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="n" label="n" enabled="true" visible="true" x="274" y="628" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="m" label="m" enabled="true" visible="true" x="315" y="628" width="32" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="⌫" label="⌫" enabled="true" visible="true" x="356" y="628" width="53" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name=".?123" label=".?123" enabled="true" visible="true" x="4" y="678" width="74" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="space" label="space" enabled="true" visible="true" x="87" y="678" width="240" height="39"/> <XCUIElementTypeButton type="XCUIElementTypeButton" name="close" label="close" enabled="true" visible="true" x="336" y="678" width="74" height="39"/> </XCUIElementTypeOther>

Thanks

Aleksei · June 7, 2020, 9:11am

sendKeys uses keyboard! not element. Try:

driver.getKeyboard().sendKeys("my_test_text")

I do not see combination tap into inputField and send text using appeared keyboard.
As last solution we can tam on keyaboard using it locators char by char

zarashima · June 7, 2020, 1:04pm

Yeah but the displayed error message show that ‘element is disabled etc’. When I use the old build that does not use custom keyboard then the sendKeys functions properly, so at first I think the root cause due to custom keyboard displayed

Let me try with getKeyboard() then

mykola-mokhnach · June 7, 2020, 2:34pm

I assume this is a limitation of XCTest itself. You could try to create a native test and check if https://developer.apple.com/documentation/xctest/xcuielement/1500968-typetext works as expected.

I would copy the text and paste into the destination field as a workaround.

zarashima · June 7, 2020, 2:59pm

Unfortunately copy and paste is disabled on that on all input fields available in the app xD.

Aleksei · June 7, 2020, 3:28pm

@zarashima as far as your keyboard visible in pageSource you can tap it keys by char finding each element. slow but will 100% work as workaround.

mykola-mokhnach · June 7, 2020, 5:12pm

This is sad.

I’ve removed the keyboard presence verification in refactor: Do not check for keyboard presence before input by mykola-mokhnach · Pull Request #340 · appium/WebDriverAgent · GitHub, because it is actually not needed according to what apple documentation says. Install appium@beta and apply this patch to your WDA source. Let me know if it helps something

zarashima · June 8, 2020, 2:29am

Thanks. I will give it a try and let you know later