Problems with Systemd, Ghost and Node

During an upgrade to Ghost 2.25.0 I encountered some errors. A quick search suggested I needed an upgrade to Node, but then I couldn't start Ghost. I'll detail the problem and its fix in this post. Many thanks to @jeff on the Ghost Forum for his assistance.

Spotting the problem

The symptom was easy to spot - my blog wouldn't load and Cloudflare presented me with a "bad gateway" message. Accessing the site directly by IP also gave me an Nginx "HTTP 502 bad gateway" response, so the problem definitely wasn't Cloudflare.

Attempting to start the blog via ghost start resulted in the process hanging:

+ sudo systemctl start ghost_localhost
  Starting Ghost

Yet running ghost run allowed the blog to start, so I knew the blog itself wasn't completely dead.

We can see that ghost start actually runs sudo systemctl start ghost_localhost so I tried to run that: the blog still didn't come up. Some diagnostics via sudo systemctl status ghost_localhost showed:

Main process exited, code=exited, status=203/EXEC

A "203/EXEC" error is "no such file or directory". Here's our problem.

Workaround

As a workaround, while finding the time to fix the problem, I left ghost run running in a screen session. To do this:

1. SSH into your server
2. Run screen
3. cd to the blog directory
4. Execute ghost run
5. Disconnect screen using Ctrl+A then D

Viewing the service file

Not being a systemd master, I appreciated @jeff's guidance here. My service definition file for the blog is at /etc/systemd/system/ghost_localhost.service and a quick look showed me this:

[Unit]
Description=Ghost systemd service for blog: localhost
Documentation=https://docs.ghost.org

[Service]
Type=simple
WorkingDirectory=/var/www/blog
User=999
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /usr/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target

Using ls I could see /usr/bin/node did not exist anymore. This is likely because I installed the latest Node without using my distribution's software repositories (they were behind). It's worth testing other paths using ls at this point too.

Finding Node

Sounds obvious, but in order to fix the problem we need to know where Node now lives. Fortunately there's a command for this in Linux: whereis

manager@server:~$ whereis node
node: /usr/local/bin/node

Editing the service file

Next we edit the service definition to update the path to Node. First we open a text editor and edit the file (in my case via sudo nano -w /etc/systemd/system/ghost_localhost.service) to correct the path:

[Unit]
Description=Ghost systemd service for blog: localhost
Documentation=https://docs.ghost.org

[Service]
Type=simple
WorkingDirectory=/var/www/blog
User=999
Environment="NODE_ENV=production"
ExecStart=/usr/local/bin/node /usr/bin/ghost run
Restart=always

[Install]
WantedBy=multi-user.target

Reloading to apply systemd changes

If you just attempt to start the ghost_localhost service at this point you'll be warned that the config has changed on disk, so has to be reloaded. Reloading is a simple case of running sudo systemctl daemon-reload.

All fixed!

The blog can now be started via sudo systemctl start ghost_localhost. The blog will also be running correctly after restarting the server.

Banner image, the Ghost logo, from https://ghost.org/design/.

The problem and solution are also documented in this forum post.