Running pm2 automatically on system boot

pm2 is a nodejs package designed to run and manage processes in the background. Much like a systemd, it deals with making sure service is up and running and gives you some basic tools to monitor it.

So where’s the difference between systemd and pm2?

systemd is mostly just for running existing services, like ntpd (although there is systemd implementation of it) or apache, and this is main focus point in my feeling.

pm2 on the other hand tries to be so much more. You can run it as a so-called cluster mode, in which it will make your processed use all the CPUs on the machine.  It acts as a logrotate (yes, I know) for the processes it runs. If you embed some metrics in your code, it will show it for you! Yeah, yeah, and much more. Just refer the documentation, I don’t want to point all possibilities here.

One of the options is to run it using systemd.

To be completely frank, my first thoght was: Great, we will run deamons using a deamon and run it via systemd… daemon.

Let’s set up systemd to run pm2 automatically as a custom user (www-data in our case) even after system reboot.

According to documentation, the only thing you ought to do is

$ pm2 startup systemd -u www-data --hp /var/www

Which is not really working as it’s intended. First of all, pm2 is not being started automatically, because systemd expects pm2 will be a forking (Type=forking), which means that service will run fork() as it’s startup.

We should start from installing NodeJS. Installation of latest version of NodeJS (on Debian in this case) is easy in it’s insecurity:

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

Now, install pm2 using npm:

npm install pm2 -g

And we’re done. Before we go further, let’s prepare a simple ecosystem (services ran by pm2) definition:

root@jessie:~# pm2 ecosystem
[PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
[PM2] PM2 Successfully daemonized
File /root/ecosystem.config.js generated

Generated js file looks like this:

module.exports = {
  /**
   * Application configuration section
   * http://pm2.keymetrics.io/docs/usage/application-declaration/
   */
  apps : [

    // First application
    {
      name      : 'API',
      script    : 'app.js',
      env: {
        COMMON_VARIABLE: 'true'
      },
      env_production : {
        NODE_ENV: 'production'
      }
    },

    // Second application
    {
      name      : 'WEB',
      script    : 'web.js'
    }
  ],

  /**
   * Deployment section
   * http://pm2.keymetrics.io/docs/usage/deployment/
   */
  deploy : {
    production : {
      user : 'node',
      host : '212.83.163.1',
      ref  : 'origin/master',
      repo : 'git@github.com:repo.git',
      path : '/var/www/production',
      'post-deploy' : 'npm install && pm2 reload ecosystem.config.js --env production'
    },
    dev : {
      user : 'node',
      host : '212.83.163.1',
      ref  : 'origin/master',
      repo : 'git@github.com:repo.git',
      path : '/var/www/development',
      'post-deploy' : 'npm install && pm2 reload ecosystem.config.js --env dev',
      env  : {
        NODE_ENV: 'dev'
      }
    }
  }
};

We don’t need deployment section for our example, let’s just cut this to something managable.

A most basic daemon will look like this (app.sh file).

#!/bin/bash

while true; do
    echo $THEVARIABLE
    sleep 5
done

Sorry, couldn’t think of anything more basic this will run forever. Sample output:

root@jessie:~# THEVARIABLE="YES" bash app.sh 
YES
YES
YES
^C
root@jessie:~#

After some adding/removing code in ecosystem.config.js file looks like that:

module.exports = {
  /**
   * Application configuration section
   * http://pm2.keymetrics.io/docs/usage/application-declaration/
   */
  apps : [

    // First application
    {
      name      : 'testapp',
      script    : 'app.sh',
      env: {
        THEVARIABLE: 'true'
      }
    }

};

Now, we can run our dumb app, just to see if all is set:

root@jessie:~# pm2 start ecosystem.config.js 
[PM2][WARN] Applications testapp not running, starting...
[PM2] App [testapp] launched (1 instances)

[table was removed for readability]

 Use `pm2 show <id|name>` to get more details about an app
root@jessie:~# pm2 show testapp
 Describing process with id 0 - name testapp 

[table was removed for readability]

 Add your own code metrics: http://bit.ly/code-metrics
 Use `pm2 logs testapp [--lines 1000]` to display logs
 Use `pm2 monit` to monitor CPU and Memory usage testapp

Yay! Ok, that works for us. So we can kill it and clean up:

root@jessie:~# pm2 kill
[PM2] Stopping PM2...
[PM2] Applying action deleteProcessId on app [all](ids: 0)
[PM2] [testapp](0) ✓
[PM2] All processes have been stopped and deleted
[PM2] PM2 stopped
root@jessie:~# rm -rf .pm2/

Now let’s proceed to setting up systemd configuration.

Da Configuration

So, we will first need to comment below line in systemd unit file created in the first place by invoking:

# pm2 startup systemd
root@jessie:~# pm2 startup systemd
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

[...]

>>> Executing systemctl status pm2-root
● pm2-root.service - PM2 process manager
   Loaded: loaded (/etc/systemd/system/pm2-root.service; enabled)
   Active: active (running) since Tue 2017-04-25 18:44:59 GMT; 26ms ago
     Docs: https://pm2.keymetrics.io/
 Main PID: 3384 (PM2 v2.4.5: God)
   CGroup: /system.slice/pm2-root.service
           └─3384 PM2 v2.4.5: God Daemon (/root/.pm2)         

[table was removed for readability]

Apr 25 18:44:59 jessie systemd[1]: Started PM2 process manager.
[DONE]

File in question is located here:

/etc/systemd/system/multi-user.target.wants/pm2-root.service
#Type=forking

We’ve created this service as root though, and it’s not what we want in this case. Let’s undo this now, create a home directory for www-data user and re-create systemd unit file:

pm2 unstartup
mkdir -p /opt/pm2_home
chown www-data:www-data /opt/pm2_home
pm2 systemd startup -u www-data --hp /opt/pm2_home

Here. Now the systemd unit file will have a slightly different name: /etc/systemd/system/multi-user.target.wants/pm2-www-data.service

Remember to change the forking setting as we did before.

su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 start ecosystem.config.js"

Let’s fix us a home direcory for pm2 on www-data user to have it all in one place:

Let’s load our services now:

su -s /bin/bash www-data -c "pm2 start ecosystem.config.js"

Now we will need to add HOME variable to systemd unit – pm2 seem to use it a lot instead utilizing PM2_HOME (although it’s most likely used in different context). After all modifications, complete system unit file will look like this:

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
#Type=forking
User=www-data
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/opt/pm2_home/.pm2
Environment=HOME=/opt/pm2_home
PIDFile=/opt/pm2_home/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Note that this file location should be here: /etc/systemd/system/multi-user.target.wants/pm2-www-data.service

And it’s usually symlinked to systemd main system directory. Let’s do this and reload systemd daemon.

ln -s /etc/systemd/system/multi-user.target.wants/pm2-www-data.service /etc/systemd/system/pm2-www-data.service
systemctl daemon-reload

pm2 will be able to start, but most likely won’t be able to restart added services because those are not saved in a dump file, which pm2 is loading at startup. So if you already loaded your ecosystem json file with tasks/services or deamons definitions (whatever you’d like to call them), you should save it by invoking

pm2 dump

But wait! We want to run pm2 as www-data user. Let’s do this right:

su -s /bin/bash www-data -c "pm2 dump"
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 kill"

Right. So we fixed systemd unit, loaded services, saved dump of a current configuration and killed pm2 daemon executed by hand. It’s time to check if everything works as intended.

service pm2-www-data start

pm2 now should be running. Now making changes to ecosystem will not be easy without configuring deployment as per pm2 documentation, so this setup is pretty much static. You can mitigate it by running following commands every time you chance ecosystem file:

service pm2 stop
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 start ecosystem.config.js"
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 dump"
su -s /bin/bash www-data -c "HOME=/opt/pm2_home pm2 kill"
service pm2 start

I will stop at that. Hope this little tutorial will help you run pm2 from systemd.

Year 2016 retrospective

It’s always good to stop for a moment and look back at last sprint, month, quarter or, like this time, last year. It’s December, holiday season upon us, so let’s sit with a glass of good whisky and look what was going on in the last twelve months.

In 2016 I wrote astonishing number of five posts. Including this one. Obviously, my writing goals hasn’t been met at all. Not even close. Too bad for me and us all. But maybe it’s better not to write at all than write a shitty post every day.

So I’ve been busy. Up until March I had a lot of job interviews, in Berlin and in Poland. But I was aiming to get back to my home town and live full time with my family. I got tired of living in two different cities. Because of that I couldn’t get heads or tails of my life. I felt sick of constant commute, to be missing my close family and friends, and my private life which was basically nonexistent. Continue reading Year 2016 retrospective

How the professional crisis look like?

This year marked my ten years professional record of cool career in IT. During those years I did a lot of things with a lot of different technologies. I feel like I know a lot and I don’t know squat. I didn’t touched many technologies considered “modern”. I didn’t do that because most of the companies are still using stuff invented in 80s and 90s.

Today I’m questioning those 10 years work I’ve done. Today, I’m thinking of changing everything around.

When starting work I was a systems administrator dealing with Windows servers and some not too nice software. On top of that, I was fixing printers problems, fixing chairs, changing wires and solving network issues.

Then I got to know the insurance industry. It was great. Until I stopped learning and every day was just another day. I was smoking a lot, I was drinking a lot. I got fat. Then I woke up.

Continue reading How the professional crisis look like?

Linux Autumn in Poland

I rarely appear publicly. I don’t desire nor like it. Last time when I did similar thing was three years ago (in 2013) at the conference I dreamed of, planned and made it happen. Had some help however from friends and family. If it wasn’t for them, it wouldn’t happen.

This is one of the recordings:

In my opinion I was quite lousy leading speaker there. Unfortunately I wsn’t able to find a replacement to focus on more technical aspects of it.

Anyway, it happened, it was one of best conferences in Szczecin, Poland. At least in my opinion.

I hope I’ll be able to organize more of such events. So far, nothing since three years (2013).

So right now I decided, with the help of my friends, to submit a talk. And since I’m usually throwing myself against the biggest challenge, I’ve found a topic I could struggle with: Architecture of applications. Microservices.

Continue reading Linux Autumn in Poland

What it takes to bring agility in your organization?

The first time I’ve heard about “agile” in relation to IT project I thought about yet another buzzword that does nothing to what I did back then. It was 2008 and It was my second job in IT. Before I was working as an IT administrator which basically meant that I was doing everything from installing Windows XP on workstations, maintaining Novell Netware server as well as an awesome Windows NT machine with state of the art tax application installed on it (yep, that was sarcasm).

My work routine was like this: I got to work in a morning, reviewed if backups were there and were valid. Checking emails and then proceed with my personal to do list. This list involved improving stuff in our infrastructure, implementing Bacula as a primary solution (xcopy was cool, but it wasn’t enough). Continue reading What it takes to bring agility in your organization?

Job interviews, do you speak them?

It was middle of year 2011 and I decided enough is enough. I needed a new job. And this is how my long almost five years long quest began. It started slowly, with updating my CV, looking around in my area, checking companies I probably want to work for and figuring out what I’d like to do next.

Job interviews are hard. You try to prepare, dress up, go to the companys office, you talk a lot, you meet people, shake hands, talk some more and you get rejected a lot.

Essentially, seeking for a job is like looking for a publisher for your new shiny book. Or looking for a company that will create your awesome device. It’s hard, very hard. Continue reading Job interviews, do you speak them?

DevOps Days Warsaw

After years and years of struggling to join a bigger conference, I managed to see how conference looks like. Warsaw way.

That was a second edition of such conference in Poland, so I guess team is still learning. But, I must say, organization was well thought, didn’t notice bigger issues.

Conference lasted two full days of talks and ignite talks on the first day. On the first day, surprise surprise, there was also an after party. Continue reading DevOps Days Warsaw

Welcome back!

After a long, long, long, (seriously) long time not writing, I decided to write something daily. Yes, daily.

… but not necessarily on this blog though.

In the mean time: I haven’t decided yet what to do with this space. I know I still want to write in English, but I don’t know how much technical I want to stay.

Currently I’m working with one magazine in Poland and I’m writing mainly for them instead of blogging. I’ve been thinking on translating everything I write also to English and publish it right here. That may be one way to maintain publishing here.

Another idea which came into my mind is writing about recruiting. I have a bit of experience with HR and talking to various people from different countries (Germany, US, Ireland, UK, Poland) about job. Couple of weeks ago I came across recruiter from adult industry – that was interesting.

I guess I’ll have enough material with those two.

For now my intention was just saying “hi” and getting my all thoughts together in this post.

Couchbase check for Nagios/Icinga

Couchbase is a NoSQL document database designed for high scalability and availability. It’s very flexible and dead easy to configure once it’s installed. And it’s much more than simple NoSQL for documents, for example it can be used as a Memcached replacement. Continue reading Couchbase check for Nagios/Icinga

Moodle: Add (file) resources and upload files into it

File resource in Moodle is another course element that can be used for serving files for students. It’s assumed that there will be only one file per resource, but you can upload multiple files and set one of them as default to be downloaded when student click on the resource.

We’ll be adding multiple resources and multiple files in this example. Doing so using pure php coding would be complicated enough to mess things up very easily, so we’ll use Moodle APIs to do some things for us. And it’ll be additional plus as we will be more Moodle version independent as long as API will remain not changed. Continue reading Moodle: Add (file) resources and upload files into it