io.js 3.0.0 memory leak

[novacrazy]

See Unitech/pm2#1500 for the bug report filed with PM2 before I realized it was being caused by io.js.

With io.js 2.4.0 and 2.5.0 there were no memory leaks for that module. Perhaps it could have something to do with the new Buffer implementation?

[trygve-lie]

I've also got an app I'm test running at 3.0.0 and after 4 hours it uses the double amount of memory as it used on 2.5.0. And it seems to be climbing on 3.0.0.

[jbergstroem]

Perhaps @ChALkeR, @shigeki or @jasisk could do some testing?

[shigeki]

@jbergstroem Sorry, I'm just getting on board to SFO now.

[ofrobots]

Is there a test-case that be used to reproduce the memory leak?

[novacrazy]

Not that I can think of, unless you already have a system running PM2. Up until the server machine crashes there are no error messages or logs to track it down with. It is most certainly an issue only on io.js 3.0.0, though. I reverted back to 2.5.0 and memory usage has been stable for almost 10 hours now.

[rvagg]

fwiw you can't assume that high memory usage and growing early memory usage are a memory leak, the way V8 handles memory and GC changes over time so you may possibly be dealing with a "normal" memory profile. It'd be good to have data for a longer run.

[ofrobots]

Based on the original report in Unitech/pm2#1500, it seems that the growth rate is 30MB/min and the app runs out of memory in a couple of hours. This does smell like a memory leak, but It would be very hard to debug this, or to know if the memory leak is actually in io.js, without a test case that shows the problem. At the least, can you grab and compare heap-snapshots and see what kinds of objects are growing?

[novacrazy]

Well, I changed nothing other than upgrading io.js, and tried out both old, current and pre-release versions of PM2. I even tried out bumping down my dependencies from the last time I upgraded those.

When viewed with htop, I saw 500+ MB (kept growing until the server crashed) for each process with 3.0.0, versus about 40-60MB stable per process with 2.5.0. I wouldn't call a 1000% memory increase with up to 2MB per second continuous growth at idle a "normal" curve.

Perhaps it is completely normal if the bug is in PM2 instead, and previous versions' GC still managed to collect it somehow while 3.0.0 doesn't. My actual application code and processes seems unaffected; only the PM2 processes have such high memory usage. The PM2 processes also handle all the cluster and load balancing stuff, which could be more affected by the new Buffer implementation than my code is. I honestly don't know, but I'm willing to give you any data I can possibly provide.

[jbergstroem]

To people experiencing the same issue: The best way to drive this issue forward is more metrics over time and a reduced proof of concept we all can test.

[novacrazy]

I'm sorry, but I honestly have no idea how to take a heap snapshot of the individual PM2 daemons.

[jbergstroem]

@novacrazy try importing heapdump and send it a unix signal.

[novacrazy]

Alright, I injected require('heapdump'); into Common.js so all the daemons should have it. Now I just need to wait a bit for things to accumulate and for my rural internet connection to download the files. I'll post the (probably massive) results in a little while.

[ofrobots]

Before you upload the heapdump, be aware that it contains the full JavaScript heap and may include private information (such as user data, keys, etc.) and the source code of your JavaScript functions.

[novacrazy]

@ofrobots Very good advice, thank you.

Also, @bnoordhuis, heapdump fails to compile with io.js 3.0.0: http://pastebin.com/RVnY5Mrs

Not sure if that is my fault or part of the last V8 upgrade.

[bnoordhuis]

I just closed out a similar issue 30 seconds ago. :-)

I don't think that's an issue with node-heapdump but with V8. It's a C++11 project and you'll need at least gcc 4.8 or clang 3.4. You could get away with older versions until now but no more.