Hey guys,
I’m having an issue with my API, I’ve done a bug report on Directus and they said it’s likely two instances are running of Directus, which shouldn’t happen:
If you go to https://cms.polinations.com/items/home_whats_on_featured
Each refresh will toggle between “FORBIDDEN” and showing the correct result.
Also when upgrading Directus, you need make sure you run npx directus bootstrap otherwise it can and does cause issues with deployments.
Thanks,
Tom
Hi @Tom,
I’m not seeing the forbidden screen after refreshing mult times. Were you able to locate the issue?
If it happens again, check the NGINX error logs and PM2 logs in server > services to see if there are any clues there.
Hi @amiedema,
It’s happening again, see here - https://cms.polinations.com/items/iconic_plant?fields[0]=*&fields[1]=*.*&fields[2]=story.*&fields[3]=story.image.*&fields[4]=story.audio.*&filter[slug][_neq]=fern&filter[status][_eq]=published
Each refresh will toggle between answers. I have been slowly moving away from Cleavr because of the amount of bugs. I now host my front-end with Vercel, but moving the back-end is a much bigger task. I hope these issues get ironed out because without the bugs Cleavr is amazing.
When pushing to GitHub you are not correctly clearing out PM2 and you are allowing multiple instances to run:
This is not only doubling memory usage but is also causing major issues with the API.
Hi Tom, this would be expected when running in cluster mode - which utilizes available cores. In this instance, I’d assume your server has 2 cores which is why you see two processes running with PM2. The number of cores used can be altered, but we run apps in cluster mode for multi-core servers by default as it’s more performant.
What issues are occurring with the API?
HI @amiedema
If you visit this URL: https://cms.polinations.com/items/iconic_plant?fields[0]=*&fields[1]=*.*&fields[2]=story.*&fields[3]=story.image.*&fields[4]=story.audio.*&filter[slug][_neq]=fern&filter[status][_eq]=published
And press refresh, it will toggle between a response and not found
Can you try again? I think this was related to the undefined processes - which I believe was the product of a domain renaming bug for node sites that we’ve since fixed. In any case, I cleared the unknown process and the url seems more stable now when refreshing.
Hi @amiedema
It seems to only temp fix it. The issue has now appeared again - https://cms.polinations.com/items/privacy_policy
EDIT:
Restarting Directus in PM2 fixes the issue, any new collections I create will have this problem until I restart the application.
The activate deployment step reloads/restarts pm2, are there any clues in the deployment logs (web app > deployments > deployment > activate deployment) for that step?
You might try adding the following custom deployment step at the end as well -
cd {{ releasePath }}
pm2 restart .cleavr.config.js
Hey @amiedema
I think you’re misunderstanding the issue. When I create a new collection in Directus and set it to public in the permissions, each refresh will give a different response between “forbidden” and the results. Each refresh will give one or the other;
For example refresh 1 will give “forbidden”, refresh 2 will give the results, refresh 3 will give “forbidden” and so on.
This will also break the CMS with newly created collections as sometimes it doesn’t think the collection exists and sometimes it does, so it starts to break adding fields to the collection or updating the page.
It isn’t until I restart Directus (PM2) or redeploy does this issue correct itself.
I have created a new collection so you can look into this:
https://cms.polinations.com/items/test
I suspect it’s something to do with Directus running in cluster mode in PM2.
Gotcha! Thanks for the clarification, I am able to reproduce this on a test site with NodeJS v16 and a server with 2 cores.
I can see an issue on Directus’ project here that is related: Permissions problems on root user · Issue #13661 · directus/directus · GitHub
Updating the web app > environments tab in either one of the following ways worked for me:
Add the following variables:
CACHE_SCHEMA=false
CACHE_PERMISSIONS=false
Above is disabling additional caching. Though, if your server is running Redis (check server > services > redis) then the following is probably better:
CACHE_STORE=redis
After syncing the env variables, redeploy the app.
We’ll likely add CACHE_STORE=redis as a default variable for new directus apps.