Building Neutronself-contained node & npm sketchbook

Neutron is a self-contained node & npm application made for quick prototyping and teaching.

It borrows ideas from Processing and tries to create similar novice and artist friendly environment.

Neutron was born out of two main needs:

  1. sketching — I love npm ecosystem, almost every piece of code I would want to reuse is there, usually with a few different approaches. This, unfortunately, is not so nice for quick throwaway sketches. I have to set up a project, set up a build pipeline (budo helps here though), open browser, and remember to install dependencies by hand. I wanted to be able to do it quicker, with less steps.
  2. teaching — which I really enjoy, I believe that sharing knowledge is important, and that programming is easiest to understand through creative outlets: making graphics and sounds. I always go with Processing since it works everywhere, you just download binary, and run it. At some point, someone always asks about porting the thing they made to web, and even though Java applet is a possibility, usually JavaScript is a better approach — another language they will have to learn, and a new set of libraries to explore; this can be discouraging for newcomers. I wanted Processing environment, but with JavaScript instead of Java.

Neutron is based on Electron — basically a programmable Chrome window with require() support. This gives us best of both worlds: file system access with require('fs'), full node support, and a window which can render HTML. Neutron wraps this functionality in easy to use application.

On start, Neutron asks to load a JavaScript file, this can be done either by dragging it on the window, or clicking to open file select. This simple UI is build with React, probably overkill in this situation, but this allowed me to quickly move forward.

The most interesting part of this software is an automatic package installation. This consists of two things:

  1. finding out which packages are required
  2. installing packages with npm

For 1. I used recursive-deps package, after having tested few other solutions. This one was the most straightforward and reliable, and as the name suggests, finds dependencies recursively, which is helpful for sketches spread across multiple files.

Installing packages was trickier, since I knew from the beginning that npm should be a part of the application, and I couldn't just shell out to a globally installed executable.

npm itself doesn't have documented public API right now (it used to have for a while though), so this required some trial and error. Here's the most important gist (install npm as your project dependency first!):

const npm = require("npm");

const packagesToInstall = ["p5", "lodash"];

npm.load(
  {
    color: false,
    progress: true,
    save: true,
    unicode: false,
  },
  (err) => {
    if (err) {
      return console.error(err);
    }

    npm.commands.install(packagesToInstall, (err) => {
      if (err) {
        return console.error(err);
      }

      console.log("installation done");
    });
  }
);

There are some caveats though, from what I've found:

  • save: true to save packages in package.json doesn't work, I solve this by creating package.json containing only {} before installing any packages
  • there are issues with getting the log info and streaming to the client, I have hacked a semi-solution providing custom node stream that I then send to main process, this works with some issues on OSX, but I couldn't get it to work on Windows
  • to avoid re-installing already installed packages on every save, I first use npm.commands.ls to check what's already installed
  • I also had to filter out built-in packages from recursive-deps, so for example require('fs') doesn't try to install fs from npm

The full relevant code is here.

The biggest problem of this approach is using native packages. If this was based on node, then most packages published to npm are already compiled, and it would be much less problematic. Since Electron uses different V8 version, almost all of native packages have to be compiled after installation. Usually, if node and npm are set up correctly, this is not that big of a deal, at least on OSX (Windows is still a bit more problematic).

The aim of this tool is to allow users to use JavaScript quickly, without any complex setup. I've experimented a lot with bundling some build tools with the app, but sadly, I failed. It seems to be possible on OSX, but I lack Windows-specific knowledge to make this a reality, and I suspect that compilation would still require Visual Studio to be installed.

I've ended up bundling three native packages which I use most often when teaching workshops:

For building an executable version of the app, I started with local Windows builds on my OSX machine, using Wine. This doesn't work when trying to build native modules — if you need them, you have to compile on yor target platform.

Luckily, this is easily sorted with CI build artifacts! I use AppVeyor for Windows builds, and Travis for OSX ones. As long as the project is opensource, they are free, which is great.

They also work great with electron-builder that I use for building the app. To get this to work, I had to add two config files:

To have them publish automatically to GitHub, GH_TOKEN has to be set. It can be generated here: https://github.com/settings/tokens, with access right for public_repo. For Travis, it can be added as an environment variable in project settings, and for AppVeyor, it can be included straight into appveyor.yml if it's encrypted: https://ci.appveyor.com/tools/encrypt.

To make Neutron work a bit more like Processing, I've decided to expose a simple wrapper around Electron functions, this allows me not only to get the window size, but move it around the screen, get displays, access mouse position outside of a window, and more.

Detailed API info is up on Github.

Building Neutron was painful at times, especially the week I've spent trying to figure out native modules. I believe this tool will be useful for others, for both sketching and teaching.

I'm already using it often enough that I believe it was a good investment of my time. Lately, any idea I have, or anything I want to try, starts with running this bash function:

mksketch() {
  cd $(mktemp -d)    # make random temp directory
  touch index.js     # create empty JS file
  neutron index.js & # I'm using neutron through CLI
  vim index.js       # start playing!
}

I've also made a longer write-up about ideas and process behind building this: building neutron