Node JS, PM2, and running apps in cluster mode - FIXED!

We’ve been increasingly receiving issues concerning cluster mode not running properly on servers with multiple CPU / Cores. An indication of this is high CPU usage and constant restarts if you check the NodeJS heartbeat in the services section.

This mainly impacts Strapi and Nuxt apps running on Node v16 and any other app types that start via the npm start command as opposed to having a specific file pointed to - ie, server.js, main.js, etc.

If you are using Node v16 and a server with 1 CPU, then you shouldn’t experience this issue.

Work-a-rounds

As we’re looking more into the issue and how we might resolve it, there are a couple of work-a-rounds you can try.

  1. From the services section, uninstall Node 16 and install Node 14; then you’ll need to click the ‘repair’ button on the ellipsis menu after Node 14 installs. This resets the apps and PM2 so they run correctly.

  2. If you need to keep node 16, then you can update your web app to point directly to the start file. However, this solution won’t work for using the GHA integration at the moment. In web app > settings > build:
    2.1 If you have anything listed in the ‘Artifact Path’ box, remove it and deploy out the app before moving to next step (this will make the site not render so please only do this during maintenance downtime)
    2.2 Replace Entry File from npm with ./node_modules/nuxt/bin/nuxt.js (note, if using Strapi, then replace nuxt with strapi)

2 Likes

We’ve resolved the PM2 issues around cluster-mode for Node v16. Here are the release notes.


As we’ve been continuing our efforts to harden Cleavr, some reports started to come in about Strapi and Nuxt apps not working properly when utilizing cluster-mode on Node v16. We’ve tackled that issue and added some new :candy: along the way!

Strapi

  • Split Strapi out to Strapi 3 and Strapi 4 app-types to provide improved support by version
  • Resolved cluster-mode issue on Node v16
  • Resolved persistent file storage issue during deployments

For existing Strapi apps

Unfortunately, we were not able to make the cluster-mode improvements backwards compatible for existing apps. But…! You can make some quick updates to take advantage of the new improvements.

Strapi v3 apps

Add the following file and it’s contents to the root of your project and push to your code repo:

.cleavr.runner.js

Add the following contents to the file:

const strapi = require("strapi");
strapi().start();

After pushing the above file to your code repo, see PM2 Ecosystem Updates section below to complete setup.

Strapi v4 apps

Add the following file and it’s contents to the root of your project and push to your code repo:

.cleavr.runner.js

Add the following contents to the file:

const strapi = require("@strapi/strapi");
strapi().start();

After pushing the above file to your code repo, see PM2 Ecosystem Updates section below to complete setup.

Nuxt

  • Split Nuxt SSR out to Nuxt SSR 2 and Nuxt SSR 3 app-types
  • Nitro server-engine support for Nuxt SSR 3 :zap::zap::zap:
  • Resolved cluster-mode issue on Node v16

For existing Nuxt apps

See PM2 Ecosystem Updates section below if you’d like to take advantage of new updates for cluster-mode with Node v16.

Directus

  • 1-click install! :classical_building:
  • Resolved cluster-mode issue on Node v16
  • Resolved persistent file storage issue during deployments

Now when you add a new Directus site, you can enter admin login credentials and install the Directus bootstrap with just 1-click! Need to deploy your Directus site from your code repository? No problem! A web app is still created when adding a new Directus site, so you can deploy from your code repo just as easy.

For existing Directus apps

See PM2 Ecosystem Updates section below if you’d like to take advantage of new updates for cluster-mode with Node v16.

PM2 Ecosystem Updates

Navigate to your web app > settings > build tab, and update script to ".cleavr.runner.js" and args to “”. Except for Nuxt SSR 2 apps, you’ll need to add start for args.

Also, make sure instances is set to max and exec_mode is set to cluster_mode.

After making the changes, deploy your project for the changes to take effect.

Note for monorepos

If you have a monorepo setup, we recommend you do not make these changes quite yet. We’ll be working on a solution to better handle monorepos.

Read more about the new updates in our Cleavr Slice blog.

Oh, no!

I’m running Quasar and i now also get

had too many unstable restarts (16). Stopped. "errored"

Please adwise, my customer is waiting

Hello @peterc,

I think there is another topic that is related to the issue that you’re facing. Please refer to this thread

Do let us know whether that works for you or not.

2 Likes

I was going to call my customer in 5 minutes, you save me from a lot of trouble @anish

Thank you so much!

3 Likes

Hello, I’m running NodeV16 with StrapiV4 at Vultr 2 CPU and added the .cleavr.runner.js file at the root. Then I deployed the app but the server still has a 100% CPU utilization.
I’m running a trial to test Cleavr. Any idea how to fix this?
Thanks

Hello @BureauBerg,

First of all welcome to Cleavr forum.

We’ll look into the issue but in the meantime, can you please check if the site is throwing any errors like 502 error? You can also check PM2 log from the deployments page by clicking on Load PM2 Logs button or by going to Server > Logs. We’ve also noticed that CPU utilization reach maximum while there are certain errors at the app level.

You can also view NodeJS logs from the services section and resolve issues if there are any.

You can follow these links to troubleshoot 502 errors for NodeJS based applications:

Hi Anish, thanks for your reply.

It indeed throws a 502 error so I checked the PM2 log which shows:
PM2 log: App name: xyz disconnected
PM2 log: App [xyz] exited with code [1] via signal [SIGINT]
PM2 log: App [xyz] starting in -cluster mode-
PM2 log: App [xyz] online

and the Nginx log is as follows:
The “connect() failed (111: Connection refused) while connecting to upstream, client”

I’m a newbie but it looks to me that Nginx doesn’t have access permission to the app. And some folder settings that might be not correct yet?
I created a new system user and added the app to that user when setting it up in Cleavr and I thought these permission settings would be added automatically.
Could anyone shed light on this?

Thanks a lot!
Jacco

Hello @BureauBerg,

Two possible case for your issue:

It looks like you’re building your app on GitHub. If your project requires database connection during build you need to provide database credentials in PM2 Ecosystem. To do so go to Webapp > Settings > Build > PM2 Ecosystem and on the env section add you credentials. It may look something like this:

  env: {
    "PORT": 3333,
    "CI": 1,
    "NUXT_TELEMETRY_DISABLED": 1,
    "DATABASE_HOST": "localhost",
    "DATABASE_PORT": "3306",
    "DATABASE_NAME": "database_name",
    "DATABASE_USERNAME": "database_username",
    "DATABASE_PASSWORD": "database_password"
  }

Another one, that you’ve not updated your environment variables yet. Strapi requires some secrets to run such as APP_KEY. If you’ve not updated your environment variables from Webapp > Environment make sure to check your .env file on the local project and update them accordingly.

Make sure to re-deploy after performing the above steps mentioned above.

I hope it helps. Let us know if that doesn’t resolve your issue.