How to Decrypt PGP Encrypted files in Databricks

As a Data Engineer you may come across a project where you need to Decrypt the PGP Encrypted files in order to get the data and apply transformation logic on it.

This typically happens when you are working on highly confidential projects like Employee information management, Payment Management etc.

This blog is going to help you do that in Databricks using Shell command. So, let’s get started.

What you need to decrypt the files:

• Private Key: It is a key that allows you to decrypt the files. It should be in .asc format. It is created alongside with public key and passphrase at the time of encryption.
• Passphrase: is a word or phrase that protects private key files. It prevents unauthorized users from encrypting them.

Both private key and passphrase should be provided by the user who has done the encryption.

Steps to decrypt and store the decrypted files in Datalake:

1. Upload PrivateKey.asc to Datalake.
2. Create a .txt file to store the passphrase and upload the file to Datalake.
3. Create folder in Datalake to store the decrypted files.
4. Open a Databricks notebook.
5. Use the below shell command:

%sh
# Apply looping for each file in the Source folder.
for entry in /dbfs/mnt/Ization/EncryptedFiles/*
do

# Create input variable to get .pgp file names.
input=$(echo "$(basename "$entry")" | cut -f 1 -d '.')".csv.pgp"

# Create output variable to get .csv file names, it will be used to name the Decrypted files.
output=$(echo "$(basename "$entry")" | cut -f 1 -d '.')".csv"

# Import the Private key.
gpg --no-tty --batch --import /dbfs/mnt/Ization/PrivateKey.asc

# Start the decryption process.
gpg --no-tty --batch --yes --ignore-mdc-error --pinentry-mode=loopback --passphrase-fd 1 --passphrase-file /dbfs/mnt/Ization/Passphrase.txt --output /dbfs/mnt/Ization/DecryptedFiles/$output --decrypt /dbfs/mnt/Ization/EncryptedFiles/$input

done

The above code is to decrypt .csv.pgp files, you can replace .csv with .xlsx in the code if you need to decrypt .xlsx.pgp files.

Thank you and Happy learning.

Ready to elevate your skills? Click here for my Python book and here for my Machine Learning book on Kindle.