After a bit more testing I noticed that timeout did not do anything either, so I kept on searching.
The weird part was that the test more ofter passed after setting this capability, but it might be that a reboot of my local machine or the device just made it x faster just like irb that just worked fine. So it took me some time to figure out this was not the solution either.
At some point I bumped into a post that mentioned the open_timeout and read_timeout capabilities.
And read_timeout solved the problem for me.
If I put it on 10, my test will fail after 10 seconds.
If I leave it out or put it on 60, my test will fail after 1 minute.
When I put it on 120 I can connect to my phones.just fine.
I hope this helps people that encounter the same issue I had.