On Maintaining a Native Node Module

As maintainer of node-serialport, a native node.js module, I constantly deal with a class of tickets focused on getting node-serialport to work with other native compiled environments. I decided to write up and share the experience of a native module maintainer for everyone. Hopefully this context and history helps others understand the special level of frustration that is maintaining a native node module.

Originally, node-serialport used a project called node-gyp directly to provide the instructions for compiling the module on various platforms (Windows, Mac, Linux, etc.). While it worked (assuming all necessary dependencies) it was a nightmare for doing any form of mass installation because undoubtably the necessary dependencies (build tools, compiler, etc) were never installed. The node-serialport project quickly became swamped with tickets about how to install on Windows, Mac, and Linux.

Due to this, we moved to using a project called node-pre-gyp, which allowed us to utilize pre-compiled binaries if they exist from a known location and fallback to the local build process if they do not. This was hooked into an elaborate build system using Travis-CI and AppVeyor to generate the necessary compiled versions for Windows, Mac, and Linux on each release published to npm.

Things were happy at this point since people using Windows, Mac, and Linux without the build components could easily use node-serialport without even being aware of its native module nature.

During this time period, the project formerly named node-webkit, now nw.js, started to evolve and gain traction. It utilized its own derivative fork/hack from node-gyp in order to build the necessary native componets which was named nw-gyp and this choice had ramifications that affected the compilation of other native modules. Needless to say, as people began trying to use it in combination with the node-serialport project things did not work smoothly. Until the NW.js team published the previously linked instructions for node-gyp and node-pre-gyp, this project was blamed for incompatibility and many issues were filed.

Then the Atom editor was released. The Electron team that maintains the application framework around the Atom Editor updated to the latest version of Chromium around September, 2014, which consequentially broke compatibility with any and all node-gyp compiled modules. Similar to NW.js, the Electron team posted details about how build Electron with other native modules, but those native modules mostly were not aware of this. As such, many issues were filed against this and other project concerned about the compatibility of that project when they try to integrate with Electron.

The general node.js leadership has not openly discussed this mainly because they do not maintain end-user facing native modules, honestly there aren't a tremendous number of them. I can't begin to tell you how many times I have been told that I should just use the built-in tty module, as if I hadn't tried that and many other options before. For those of us that do maintain a native module, the constant hamster wheel is enough to drive one insane. Until a maintainer can explore the other project, there is a very real sense that it could be the fault/issue/incompatibility of the project being maintained since Node.js, io.js and the ecosystem changes so quickly and its too much to keep up with. At this point, here is the general state of the world:

• node-gyp is the de-facto build standard for Node.js and native modules. All things should be compatible with node-gyp at a minimum or else it should be considered out of compliance.
• node-pre-gyp since it is an enhancement beyond node-gyp by adding pre-compilation and retrieval, it should be considered as part of the compliance suite for any native module.
• nw-gyp to my knowledge is only used by NW.js and as such should ensure that it or application that includes does not break compliance with node-gyp or node-pre-gyp. That is currently not the case however and inclusion of node-gyp and node-pre-gyp modules still require special handling.
• Atom-Shell/Electron is currently not compliant with node-pre-gyp due to the way that it identifies its node execution version and environment. That said, the node-pre-gyp maintainers are working on a way to make node-pre-gyp modules work with Electron. There is currently an outstanding pull request that will provide support for this variation handling. Until that pull request is merged in, continue to use the instructions provided by the Electron team.

Specifically for node-serialport, as the prime maintainer I will not be merging in modifications that take this project backwards (ala reverting to node-gyp). Also, I will not be integrating special cases to support new native environments that have broken compatibility without thorough testing in every execution environment we already support.

In summary, maintaining a native module in node.js is a constant battle. The node community needs to hold projects that break native compilation compatibility accountable by encouraging those projects to ensure node-gyp and node-pre-gyp modules can still be included without issue or failure.