Feature Request: Persistent File Storage for all Web apps

Hi,

I saw this feature release for strapi and directus. But for example deploying a Bedrock WP site (PHP) results in the same thing. When deploying, all uploaded media files in “web/app/uploads” get lost. It is possible to fix that with a custom deployment hook. However, I think it would be better to add a general “persistent file storage” functionality for all type of Web apps, where users can specify one and/or multiple paths that should persist around deployments. This would make it straightforward to add single files as well as directories to be persistent.

Beyond that, this could also increase stability for highly interactive web apps. For example, a forum where people can upload media files. Rolling back to a previous deployment could at some places result in a “file not found”, as the media file where not uploaded on that previous deployment. Using a persistent path would not cause this problem on rolling back and would also save storage on the server, I guess.

What do you think about that? :v:

Hello @Jeazyee,

I think that’s a great idea!

We wanted to get the solution out first for the apps we know for sure have this issue and are definitely planning on making something similar available for other app types.

Thanks so much for asking the question and providing examples! We love when requests are paired with clear examples, it helps us picture it better and get excited!

1 Like

@amiedema Awesome!

Just for inspiration, I really love how another server management tool handled this persistent file directory stuff. Your way of managing .env is way better, though. :wink:

Persistent Files / Directories overview:

Add persistent File / Directory:

Functionality works like that:
Creates a symlink after deployment (directly after pulling the git repository) from the path of the persistent_storage folder in the {{ projectPath }} to the {{ releasePath }} of the file/folder that should persist

ln -s "{{ projectPath }}/persistent_storage/path/to/folder/persist" "{{ releasePath }}/path/to/folder/persist"
2 Likes

I am sharing my universal deployment hook script for all those who don’t want to wait until something official is out. My script creates the files and directories if they don’t exist, and links them recursively in the deployment repo. So in any case, the persistent files/directories are being used! Additionally, it copies files and directories from the previous deployment if there was no persistent storage in use. :v:

Usage:
Go to your web app → Deployment Hooks → Add Hook

Paste my script:

#!/bin/bash
# Purpose: Shell script that creates and links your persistent dirs and files recursively
# Usage: Run this in your deployment hooks after pulling the files from your repository
# --
# Version: 1.1
# Author: Jitendra Bodmann (@jeazyee)
# Last updated: 24/Mar/2022
# --------------------------------------------------------------------------------

### CONFIGS ###
# the persistent files
PERSISTENT_FILES=(
    "path/to/file.ending"
)
# the persistent dirs
PERSISTENT_DIRS=(
    "path/to/directory"
)
# the location and name of the persistent storage directory
PERSISTENT_ROOT_DIR="{{ projectPath }}/persistent_storage"
# the location of the current deployment
CURRENT_DEPLOYMENT_DIR="{{ releasePath }}"
# the location of the previous deployment
PREVIOUS_DEPLOYMENT_DIR="{{ projectPath }}/current"
### CONFIGS END ###

for P_FILE in "${PERSISTENT_FILES[@]}"
do
    if [[ ! -e "${PERSISTENT_ROOT_DIR}/${P_FILE}" ]]; then
        mkdir -p "${PERSISTENT_ROOT_DIR}/${P_FILE%/*}";
        if [[ ! -L "${PREVIOUS_DEPLOYMENT_DIR}/${P_FILE}" ]] && [ -e "${PREVIOUS_DEPLOYMENT_DIR}/${P_FILE}" ]; then
            cp -R "${PREVIOUS_DEPLOYMENT_DIR}/${P_FILE}" "${PERSISTENT_ROOT_DIR}/${P_FILE}";
            echo "copied file ${P_FILE} from previous deploy to be persistent";
        else
            touch "${PERSISTENT_ROOT_DIR}/${P_FILE}";
        fi
    fi
    rm -rf "${CURRENT_DEPLOYMENT_DIR}/${P_FILE}";
    mkdir -p "${CURRENT_DEPLOYMENT_DIR}/${P_FILE%/*}";
    ln -s "${PERSISTENT_ROOT_DIR}/${P_FILE}" "${CURRENT_DEPLOYMENT_DIR}/${P_FILE}";
    echo "created symlink for persistent file ${P_FILE}";
done

for P_DIR in "${PERSISTENT_DIRS[@]}"
do
    if [[ ! -e "${PERSISTENT_ROOT_DIR}/${P_DIR}" ]]; then
        if [[ ! -L "${PREVIOUS_DEPLOYMENT_DIR}/${P_DIR}" ]] && [ -e "${PREVIOUS_DEPLOYMENT_DIR}/${P_DIR}" ]; then
            mkdir -p "${PERSISTENT_ROOT_DIR}/${P_DIR%/*}";
            cp -R "${PREVIOUS_DEPLOYMENT_DIR}/${P_DIR}" "${PERSISTENT_ROOT_DIR}/${P_DIR}";
            echo "copied dir ${P_DIR} from previous deploy to be persistent";
        else
            mkdir -p "${PERSISTENT_ROOT_DIR}/${P_DIR}";
        fi
    fi
    rm -rf "${CURRENT_DEPLOYMENT_DIR}/${P_DIR}";
    mkdir -p "${CURRENT_DEPLOYMENT_DIR}/${P_DIR%/*}";
    ln -s "${PERSISTENT_ROOT_DIR}/${P_DIR}" "${CURRENT_DEPLOYMENT_DIR}/${P_DIR}";
    echo "created symlink for persistent directory ${P_DIR}";
done

Edit the PERSISTENT_FILES and PERSISTENT_DIRS with the ones you want to keep accross deployments. Multiple entries require new line and no separators.

Like this:

And add the hook.

Move the hook directly after “Copy Project”

Deploy your app and enjoy!

The script will create the dirs and files that should persist and links them into the release:

Now if you deploy again or roll back, all files and dirs that should persist are still there with all contents! :partying_face:

Edit: Now it will also copy files and directories from the previous deployment (only when there was no persistent storage in use). This makes it very convenient to move to the persistent storage without needing to copy anything to it manually.

3 Likes

Nice! Thanks for sharing!

This is great, thanks for sharing!