Appium / NodeJS - Memory Allocation Crash


I have a similar issue with Appium 1.6.4-beta.
We run our iOS tests nightly for almost 7 hours and at some point (during the last 1-2 hours) we kick off a new Appium instance on a different port to run the Android tests, and they overlap for almost 1 - 2 hours.

The issue is that the iOS tests crash some time after the Android tests start with the following stack trace:

[debug] [MJSONWP] Calling AppiumDriver.timeouts() with args: ["implicit",1000,"01227913-2361-48fb-a16b-98cbaa73266b"]
[debug] [XCUITest] Executing command 'timeouts'

<--- Last few GCs --->

[59843:0x102804600] 21441585 ms: Scavenge 1375.0 (1426FATAL ERROR: Scavenger: promoting marked
 Allocation failed - process out of memory
 1: .9) -> 1374.3 (1426.9) MB, 8.0 / 0.0 ms  (+ 1.8 ms in 509 steps since last GC) allocation failure 
[59843:0x102804600] 21441763 ms: Scavenge 1375.2 (1426.9) -> 1374.5 (1426.9) MB, 8.9 / 0.0 ms  (+ 1.0 ms in 523 steps since last GC) allocation failure 
[59843:0x102804600] 21441924 ms: Scavenge 1375.4 (1426.9) -> 1374.9 (1426.9) MB, 8.2 / 0.0 ms  (+ 1.9 ms in 536 steps since last GC) allocation failure 

<--- JS stacktrace --->
Cannot get stack trace in GC.
node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: void v8::internal::ScavengingVisitor<(v8::internal::MarksHandling)0, (v8::internal::PromotionMode)0, (v8::internal::LoggingAndProfiling)1>::EvacuateObject<(v8::internal::ScavengingVisitor<(v8::internal::MarksHandling)0, (v8::internal::PromotionMode)0, (v8::internal::LoggingAndProfiling)1>::ObjectContents)0, (v8::internal::AllocationAlignment)0>(v8::internal::Map*, v8::internal::HeapObject**, v8::internal::HeapObject*, int) [/usr/local/bin/node]
 6: v8::internal::Heap::IteratePromotedObjectPointers(v8::internal::HeapObject*, unsigned char*, unsigned char*, bool, void (*)(v8::internal::HeapObject**, v8::internal::HeapObject*)) [/usr/local/bin/node]
 7: void v8::internal::CallIterateBody::apply<v8::internal::FixedBodyDescriptor<24, 40, 40>, v8::internal::ObjectVisitor>(v8::internal::HeapObject*, int, v8::internal::ObjectVisitor*) [/usr/local/bin/node]
 8: v8::internal::Heap::IteratePromotedObject(v8::internal::HeapObject*, int, bool, void (*)(v8::internal::HeapObject**, v8::internal::HeapObject*)) [/usr/local/bin/node]
 9: v8::internal::Heap::DoScavenge(v8::internal::ObjectVisitor*, unsigned char*, v8::internal::PromotionMode) [/usr/local/bin/node]
10: v8::internal::Heap::Scavenge() [/usr/local/bin/node]
11: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
12: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) [/usr/local/bin/node]
13: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
14: v8::internal::OrderedHashTable<v8::internal::OrderedHashSet, v8::internal::JSSetIterator, 1>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [/usr/local/bin/node]
15: v8::internal::KeyAccumulator::CollectOwnPropertyNames(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::JSObject>) [/usr/local/bin/node]
16: v8::internal::KeyAccumulator::CollectOwnKeys(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::JSObject>) [/usr/local/bin/node]
17: v8::internal::KeyAccumulator::CollectKeys(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::JSReceiver>) [/usr/local/bin/node]
18: v8::internal::FastKeyAccumulator::GetKeys(v8::internal::GetKeysConversion) [/usr/local/bin/node]
19: v8::internal::KeyAccumulator::GetKeys(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::KeyCollectionMode, v8::internal::PropertyFilter, v8::internal::GetKeysConversion, bool) [/usr/local/bin/node]
20: v8::internal::(anonymous namespace)::GetOwnPropertyKeys(v8::internal::Isolate*, v8::internal::BuiltinArguments, v8::internal::PropertyFilter) [/usr/local/bin/node]
21: 0x17d320c06167
22: 0x17d321e75c11
23: 0x17d321e750e8
24: 0x17d323812d24
25: 0x17d320c07e55

If I run a subset of them locally, they run in parallel without any issues.

The interesting part is that on the Continuous Integration server, even though the iOS ones crash, the Android ones keep running just fine.

I have also replied to this GitHub issue but I didn’t see any activity on it:

Any ideas how I could fix this?
Thank you!

try run with ERROR log level or at least NOT DEBUG.

The issue reproduced also with logging level set to Error.

After updating to Appium 1.6.4 (stable version) the issue stopped reproducing.

Also it seems that it was reproducing even without having the Android build run in parallel. Just having the iOS build run for 5 hrs or so caused the crash to occur.

Just as a FWY - I’m using node v7.1.0 and npm v3.10.10 (not sure if it matters at all)

just wonder how many tests you are running for 7h :slight_smile: ?
we have 80 and run time 2h about.

Right now we have 351 and it takes around 9 hrs to run.

We’re using a lot of API calls to get our system under test, to set up our data faster rather than through the UI.
The average would be somewhere around 1m 30s per test.

The 9 hrs are in the context of running iOS and Android in parallel (if they would not run in parallel it would be less than 9 hrs), and they share some backend resources that can sometime causes some delays. We could improve that by making them not share any backend resources.

At this point we run them nightly and 9 hrs is still a decent period of time. We’ll keep writing some more so probably we’ll get close to 12 hrs. We’ll see then how we address this, coming in the office each morning and having the tests still run is not what we want.