Appium 1.6.4 and C# .NET webdriver loosing communications

PC Side:
.Net driver is 3.0.0.2 with all the latest dependencies.

Mac:
All latest OS updates, etc; xCode 8.3.2, Appium 1.6.4 console app and desktop app both. All up-to-date.

iPhone: iOS 10.3.1 real device

The problem is since updating to Appium 1.6.4 is Appium and the C# driver seems to stop talking. The C# driver will give an exception that it cannot connect to the remote session.

OpenQA.Selenium.WebDriverException: Unexpected error. System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xxx.xx.xx.xx:4723

This will last for about 5-10min and then suddenly it starts working again. All the test cases during that time are failing. An observation looking at Appium during this time is nothing is happening. There is no spew in the logs scrolling by on the screen. Its just sitting there with nothing happening.

Happens with both the console version of Appium and the UI version of Appium. UI version seems to happen more often.