In this tutorial we are going to learn to use Python SDK provided by Google Cloud Platform to upload, download and delete files in Google Cloud Storage (GCS).
For this tutorial, you must have a Google cloud account with proper credentials. To create an account on GCS, go to here.
Assuming you have created the account and also the first project using the dashboard, let’s move forward.
Requirements
For this tutorial you need to install gcloud Python SDK in the following manner:
pip install gcloud
Once done with installing gcloud, make sure you have added Google Cloud Storage access key as a JSON file to the operating system environment variables. Please refer to this site to learn how to add a GCP key as a JSON file to the local machine.
Buckets In GCS
When we create a project in GCS we have to create a bucket to keep data on it. So we create a bucket named “datacourses-007” using the GCS dashboard as shown below.
Now that we have a bucket created inside GCS, let’s move to the Python programming part.
Uploading Files To GCS
To upload files on GCS we need to import the gcloud library and create a GCS client object in the way shown below. Also, to verify whether we are in the right directory or not, we need to print a list of buckets.
import gcloud
# create gcs client
storage_client = storage.Client()
buckets = list(storage_client.list_buckets())
print(buckets)
So here, we can verify that we are indeed in the right directory.
Now, let’s upload the file “testing.txt”. We have to pass the name of the file we want to upload, the name of the bucket (where the file will reside) and the name of the destination file.
destination_file_name= 'testing.txt'
source_file_name = 'testing.txt'
bucket_name = 'datacourses-007'Now, we get bucket object using storage_client.
bucket = storage_client.bucket(bucket_name)Let’s go on and create a blob object using bucket object. Blob object will be used to upload the file to the correct destination.
blob = bucket.blob(destination_blob_name)Now we call blob.upload_file() to upload file.
from gcloud import storage
# create storage client
storage_client = storage.Client()
# give blob credentials
destination_blob_name= 'testing.txt'
source_file_name = 'testing.txt'
bucket_name = 'datacourses-007'
# get bucket object
try:
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(source_file_name)
print('file: ',source_file_name,' uploaded to bucket: ',bucket_name,' successfully')
except Exception as e:
print(e)Let’s run the code.
So our program prints that our file has been uploaded successfully. Let’s verify it on the GCS dashboard.
You can see in the image below that the file testing.txt is available on the dashboard.
Downloading File From GCS
Follow these steps to download files from Google cloud storage:
Create a storage client just like we did while trying to upload a file.
Then we declare source_blob_name (file on bucket that we want to download), destination_file_name (name that we assign to that file on local machine) and bucket_name.
We create a bucket and blob as we did at the time of uploading a file. Then, we call blob.download_to_filename() method to download the file as following:
from gcloud import storage
# create storage client
storage_client = storage.Client()
# give blob credentials
source_blob_name= 'testing.txt'
destination_file_name = 'downloaded_testing.txt'
bucket_name = 'datacourses-007'
# get bucket object
try:
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print('file: ',destination_file_name,' downloaded from bucket: ',bucket_name,' successfully')
except Exception as e:
print(e)
Summary
In this topic, we learned to use Google cloud provided Python SDK called gcloud. We learned to use gcloud in Python to upload and download files in Google Cloud Storage. We also learned about buckets and project structure while creating the first project on GCS.
References
https://cloud.google.com/docs/authentication/getting-started#cloud-console