Appium random "Error 61 Connection Refused"

When attempting to connect to appium remotely, sometimes there is a Connection Refused Error (61)

Appium Version: 1.4 GUI, and 1.5.3 command line
OS running appium: Windows 10
Test Language: Python
OS running test: Mac OSX El Capitan
Occurs on both simulated and real devices