In this article:
- About Persistent Storage
- Storage Regions
- Working with Persistent Storage
- Accessing Persistent Storage from your VM
About Persistent Storage
Anything you store in /storage directory will be accessible across multiple runs of Jobs and Notebooks in a given storage region.
Persistent storage is a separate area where you can read and write files during a job or while running a notebook. You can also read and write data from your Workspace, but as the name indicates, persistent storage persists across job and notebook runs.You can access the persistent storage at /storage.
Persistent storage is accessible by jobs and notebooks within the same region. Paperspace Persistent storage is located two regions:
East Coast (Paperspace Cloud)
West Coast (TPUs: Google Cloud)
Working with Persistent Storage
NOTE: Concurrently running jobs all access the same persistent storage, so be careful to make sure that multiple jobs are not writing to the same file. If you need to use a large dataset over multiple job runs, be sure to save it in Persistent storage.
# This simple.sh script is in the current working directory on my
# laptop, and gets uploaded to the workspace when my paperspace job is
# created. You can put it in an empty directory and run:
# paperspace jobs create --container ufoym/deepo --machineType P5000 --command './simple.sh'
# Let's fetch a large file if it's not already there and put it in
# persistent storage:
if [ ! -f /storage/war-and-peace.txt ]; then
apt -y update && apt -y install curl
curl http://www.gutenberg.org/files/2600/2600-0.txt > /storage/war-and-peace.txt
# then we can operate on it and place the result in /artifacts:
wc /storage/war-and-peace.txt > /artifacts/war-and-peace-word-count.txt
# you can delete the files in /storage using another job like this:
# paperspace jobs create --container ufoym/deepo --machineType P5000 --command 'rm /storage/war-and-peace.txt'
You can access your Persistent Storage from a Paperspace Linux VM by accessing the /storage file. Learn more here.