Running pm2 automatically on system boot

pm2 is a nodejs package designed to run and manage processes in the background. Much like a systemd, it deals with making sure service is up and running and gives you some basic tools to monitor it.

So where’s the difference between systemd and pm2?

systemd is mostly just for running existing services, like ntpd (although there is systemd implementation of it) or apache, and this is main focus point in my feeling.

pm2 on the other hand tries to be so much more. You can run it as a so-called cluster mode, in which it will make your processed use all the CPUs on the machine.  It acts as a logrotate (yes, I know) for the processes it runs. If you embed some metrics in your code, it will show it for you! Yeah, yeah, and much more. Just refer the documentation, I don’t want to point all possibilities here.

One of the options is to run it using systemd.

To be completely frank, my first thoght was: Great, we will run deamons using a deamon and run it via systemd… daemon.

Let’s set up systemd to run pm2 automatically as a custom user (www-data in our case) even after system reboot.

According to documentation, the only thing you ought to do is

$ pm2 startup systemd -u www-data --hp /var/www

Which is not really working as it’s intended. First of all, pm2 is not being started automatically, because systemd expects pm2 will be a forking (Type=forking), which means that service will run fork() as it’s startup.

We should start from installing NodeJS. Installation of latest version of NodeJS (on Debian in this case) is easy in it’s insecurity:

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

Now, install pm2 using npm:

npm install pm2 -g

And we’re done. Before we go further, let’s prepare a simple ecosystem (services ran by pm2) definition:

root@jessie:~# pm2 ecosystem
[PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
[PM2] PM2 Successfully daemonized
File /root/ecosystem.config.js generated

Generated js file looks like this:

module.exports = {
  /**
   * Application configuration section
   * http://pm2.keymetrics.io/docs/usage/application-declaration/
   */
  apps : [

    // First application
    {
      name      : 'API',
      script    : 'app.js',
      env: {
        COMMON_VARIABLE: 'true'
      },
      env_production : {
        NODE_ENV: 'production'
      }
    },

    // Second application
    {
      name      : 'WEB',
      script    : 'web.js'
    }
  ],

  /**
   * Deployment section
   * http://pm2.keymetrics.io/docs/usage/deployment/
   */
  deploy : {
    production : {
      user : 'node',
      host : '212.83.163.1',
      ref  : 'origin/master',
      repo : 'git@github.com:repo.git',
      path : '/var/www/production',
      'post-deploy' : 'npm install && pm2 reload ecosystem.config.js --env production'
    },
    dev : {
      user : 'node',
      host : '212.83.163.1',
      ref  : 'origin/master',
      repo : 'git@github.com:repo.git',
      path : '/var/www/development',
      'post-deploy' : 'npm install && pm2 reload ecosystem.config.js --env dev',
      env  : {
        NODE_ENV: 'dev'
      }
    }
  }
};

We don’t need deployment section for our example, let’s just cut this to something managable.

A most basic daemon will look like this (app.sh file).

#!/bin/bash

while true; do
    echo $THEVARIABLE
    sleep 5
done

Sorry, couldn’t think of anything more basic this will run forever. Sample output:

root@jessie:~# THEVARIABLE="YES" bash app.sh 
YES
YES
YES
^C
root@jessie:~#

After some adding/removing code in ecosystem.config.js file looks like that:

module.exports = {
  /**
   * Application configuration section
   * http://pm2.keymetrics.io/docs/usage/application-declaration/
   */
  apps : [

    // First application
    {
      name      : 'testapp',
      script    : 'app.sh',
      env: {
        THEVARIABLE: 'true'
      }
    }

};

Now, we can run our dumb app, just to see if all is set:

root@jessie:~# pm2 start ecosystem.config.js 
[PM2][WARN] Applications testapp not running, starting...
[PM2] App [testapp] launched (1 instances)

[table was removed for readability]

 Use `pm2 show <id|name>` to get more details about an app
root@jessie:~# pm2 show testapp
 Describing process with id 0 - name testapp 

[table was removed for readability]

 Add your own code metrics: http://bit.ly/code-metrics
 Use `pm2 logs testapp [--lines 1000]` to display logs
 Use `pm2 monit` to monitor CPU and Memory usage testapp

Yay! Ok, that works for us. So we can kill it and clean up:

root@jessie:~# pm2 kill
[PM2] Stopping PM2...
[PM2] Applying action deleteProcessId on app [all](ids: 0)
[PM2] [testapp](0) ✓
[PM2] All processes have been stopped and deleted
[PM2] PM2 stopped
root@jessie:~# rm -rf .pm2/

Now let’s proceed to setting up systemd configuration.

Da Configuration

So, we will first need to comment below line in systemd unit file created in the first place by invoking:

# pm2 startup systemd
root@jessie:~# pm2 startup systemd
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

[...]

>>> Executing systemctl status pm2-root
● pm2-root.service - PM2 process manager
   Loaded: loaded (/etc/systemd/system/pm2-root.service; enabled)
   Active: active (running) since Tue 2017-04-25 18:44:59 GMT; 26ms ago
     Docs: https://pm2.keymetrics.io/
 Main PID: 3384 (PM2 v2.4.5: God)
   CGroup: /system.slice/pm2-root.service
           └─3384 PM2 v2.4.5: God Daemon (/root/.pm2)         

[table was removed for readability]

Apr 25 18:44:59 jessie systemd[1]: Started PM2 process manager.
[DONE]

File in question is located here:

/etc/systemd/system/multi-user.target.wants/pm2-root.service
#Type=forking

We’ve created this service as root though, and it’s not what we want in this case. Let’s undo this now, create a home directory for www-data user and re-create systemd unit file:

pm2 unstartup
mkdir -p /opt/pm2_home
chown www-data:www-data /opt/pm2_home
pm2 systemd startup -u www-data --hp /opt/pm2_home

Here. Now the systemd unit file will have a slightly different name: /etc/systemd/system/multi-user.target.wants/pm2-www-data.service

Remember to change the forking setting as we did before.

su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 start ecosystem.config.js"

Let’s fix us a home direcory for pm2 on www-data user to have it all in one place:

Let’s load our services now:

su -s /bin/bash www-data -c "pm2 start ecosystem.config.js"

Now we will need to add HOME variable to systemd unit – pm2 seem to use it a lot instead utilizing PM2_HOME (although it’s most likely used in different context). After all modifications, complete system unit file will look like this:

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
#Type=forking
User=www-data
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/opt/pm2_home/.pm2
Environment=HOME=/opt/pm2_home
PIDFile=/opt/pm2_home/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Note that this file location should be here: /etc/systemd/system/multi-user.target.wants/pm2-www-data.service

And it’s usually symlinked to systemd main system directory. Let’s do this and reload systemd daemon.

ln -s /etc/systemd/system/multi-user.target.wants/pm2-www-data.service /etc/systemd/system/pm2-www-data.service
systemctl daemon-reload

pm2 will be able to start, but most likely won’t be able to restart added services because those are not saved in a dump file, which pm2 is loading at startup. So if you already loaded your ecosystem json file with tasks/services or deamons definitions (whatever you’d like to call them), you should save it by invoking

pm2 dump

But wait! We want to run pm2 as www-data user. Let’s do this right:

su -s /bin/bash www-data -c "pm2 dump"
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 kill"

Right. So we fixed systemd unit, loaded services, saved dump of a current configuration and killed pm2 daemon executed by hand. It’s time to check if everything works as intended.

service pm2-www-data start

pm2 now should be running. Now making changes to ecosystem will not be easy without configuring deployment as per pm2 documentation, so this setup is pretty much static. You can mitigate it by running following commands every time you chance ecosystem file:

service pm2 stop
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 start ecosystem.config.js"
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 dump"
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 kill"
service pm2 start

I will stop at that. Hope this little tutorial will help you run pm2 from systemd.

What it takes to bring agility in your organization?

The first time I’ve heard about “agile” in relation to IT project I thought about yet another buzzword that does nothing to what I did back then. It was 2008 and It was my second job in IT. Before I was working as an IT administrator which basically meant that I was doing everything from installing Windows XP on workstations, maintaining Novell Netware server as well as an awesome Windows NT machine with state of the art tax application installed on it (yep, that was sarcasm).

My work routine was like this: I got to work in a morning, reviewed if backups were there and were valid. Checking emails and then proceed with my personal to do list. This list involved improving stuff in our infrastructure, implementing Bacula as a primary solution (xcopy was cool, but it wasn’t enough). Continue reading What it takes to bring agility in your organization?

Moodle: Adding label to a course using code

Adding lablel to any course is a very simple operation. Label is the most basic element you can add to a course body in Moodle (next to course sections or descripions). Label is basically some text usually surrounded with some html to add formatting. And it does nothing more. You can add, move it or delete it. That’s it.

Altough it’s simple to click around and add an element, adding it automatically on the other hand can be a tricky business. If you do something wrong, you can break course (best case) or mess up whole Moodle database almost beyond repairing (worst case scenario). Continue reading Moodle: Adding label to a course using code

Moodle: modify form elements before displaying the form

Moodle forms are built on top of PEAR QuickForm lib with some non standard js code and other extensions and modifications. It’s dead simple to generate some standard html forms, but when you have to do some tweaks to it, you’ll have to break some sweat and time. Continue reading Moodle: modify form elements before displaying the form