First of all, I’m running on OS X El Capitan, 10.11.3, with Xcode Version 7.2.1 (7C1002), Appium.app Version 1.4.13 (Draco), Robot Framework 3.0, robotframework-appium-library 1.3.5, the latter 2 running with ActiveState Python 2.7.5.2. My application under test is Texas Instruments TI-Nspire CAS for iPad for which I have full access to the source code and make my own .app builds with Xcode. I’m testing this app on an iOS 9.2 simulator.
The problem I’m having is that Instruments regularly crashes after “a while” of running tests, and Appium does not handle these crashes in a consistent and “friendly” manner. Here’s an example from last night’s test run:
After about 2 hours of running 50 Robot Framework test cases, most dealing with causing and detecting error dialogs in the TI-Nspire application, Instruments crashed when trying to recognize one of the error dialogs. Here’s the Appium.app log:
2016-03-03 00:37:28:048 - info: [debug] Pushing command to appium work queue: "au.getElementsByAccessibilityId('DLG')"
2016-03-03 00:37:28:048 - info: [debug] Sending command to instruments: au.getElementsByAccessibilityId('DLG')
2016-03-03 00:38:21:865 - info: --> GET /wd/hub/status {}
2016-03-03 00:38:21:865 - info: [debug] Responding to client with success: {"status":0,"value":{"build":{"version":"1.4.13","revision":"c75d8adcb66a75818a542fe1891a34260c21f76a"},"isShuttingDown":false},"sessionId":"f8d890a6-4a2d-4db6-80b1-7341bcc3ed11"}
2016-03-03 00:38:21:866 - info: <-- GET /wd/hub/status 200 0.528 ms - 179 {"status":0,"value":{"build":{"version":"1.4.13","revision":"c75d8adcb66a75818a542fe1891a34260c21f76a"},"isShuttingDown":false},"sessionId":"f8d890a6-4a2d-4db6-80b1-7341bcc3ed11"}
2016-03-03 00:39:18:474 - info: [debug] [INSTSERVER] Instruments exited with code null
2016-03-03 00:39:18:476 - info: [debug] Cleaning up after instruments exit
2016-03-03 00:39:18:482 - info: [debug] Instruments exited unexpectedly
2016-03-03 00:39:18:484 - info: [debug] Shutting down command proxy and ignoring any errors
2016-03-03 00:39:18:485 - info: [debug] Closing socket server.
2016-03-03 00:39:18:487 - info: [debug] Instruments socket server was closed
2016-03-03 00:39:18:493 - info: [debug] Stopping iOS log capture
2016-03-03 00:39:18:495 - info: [debug] Reset not set, not ending sim or cleaning up app state
2016-03-03 00:39:18:495 - info: [debug] We were in the middle of processing a command when instruments died; responding with a generic error
2016-03-03 00:39:18:498 - info: [debug] Cleaning up appium session
2016-03-03 00:39:18:502 - info: [debug] Id selector, 'DLG', not found in Localizable.strings.
2016-03-03 00:39:18:503 - info: [debug] Pushing command to appium work queue: "au.getElementsById('DLG')"
2016-03-03 00:39:18:504 - info: [debug] Condition unmet after 110456ms. Timing out.
2016-03-03 00:39:18:505 - info: [debug] Responding to client with error: {"status":7,"value":{"message":"An element could not be located on the page using the given search parameters."},"sessionId":null}
2016-03-03 00:39:18:515 - info: <-- POST /wd/hub/session/f8d890a6-4a2d-4db6-80b1-7341bcc3ed11/elements 500 110471.503 ms - 130
Notice that Appium recognizes that Instruments has crashed, yet chooses to return a simple “An element could not be located on the page using the given search parameters.” error message. This loss of information means that my Python code cannot detect the Instruments crash and try to re-launch Appium, Instruments and the app under test (which is what I plan to do if Appium cannot deal with these crashes for me). After such an Instruments crash, Appium’s next response to any API misleading, because it doesn’t deal properly with no response from Instruments:
TypeError: string indices must be integers
Traceback (most recent call last): File
...
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/NspireiOSLibrary/__init__.py", line 2303, in ElementExists el = self.AppLib.get_elements(locator, first_element_only=True, fail_on_error=False)
File "<decorator-gen-71>", line 2, in get_elements
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/robotframework_appiumlibrary-1.3.5-py2.7.egg/AppiumLibrary/keywords/keywordgroup.py", line 15, in _run_on_failure_decorator return method(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/robotframework_appiumlibrary-1.3.5-py2.7.egg/AppiumLibrary/keywords/_element.py", line 254, in get_elements return self._element_find(locator, first_element_only, fail_on_error)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/robotframework_appiumlibrary-1.3.5-py2.7.egg/AppiumLibrary/keywords/_element.py", line 429, in _element_find elements = self._element_finder.find(application, locator, tag)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/robotframework_appiumlibrary-1.3.5-py2.7.egg/AppiumLibrary/locators/elementfinder.py", line 32, in find return strategy(browser, criteria, tag, constraints)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/robotframework_appiumlibrary-1.3.5-py2.7.egg/AppiumLibrary/locators/elementfinder.py", line 43, in _find_by_id browser.find_elements_by_id(criteria),
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium-2.49.2-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 246, in find_elements_by_id return self.find_elements(by=By.ID, value=id_)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium-2.49.2-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 739, in find_elements {'using': by, 'value': value})['value']
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium-2.49.2-py2.7.egg/selenium/webdriver/remote/webdriver.py", line 201, in execute self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Appium_Python_Client-0.21-py2.7.egg/appium/webdriver/errorhandler.py", line 24, in check_response super(MobileErrorHandler, self).check_response(response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium-2.49.2-py2.7.egg/selenium/webdriver/remote/errorhandler.py", line 107, in check_response
message = value["value"]["message"]
So now some questions for the Appium community:
- Why does Instruments crash so often and how do others deal with this in long running suites of test cases?
- Why does Appium “hide” the information about an Instruments crash
- Why does Appium not remember and deal with the fact that Instruments crashed?
- Is there any way to get Appium to re-start Instruments, and the app under test if necessary, when Instruments crashes?