SoLake

SoLake

Extractor class for the Azure Data Lake Container Storage.

__init__()

Extractor for Azure Data Lake Container Storage data.

This class handles downloading and uploading data to the Azure storage containers (also known as blobs)

download_from_blob_storage(container, file_path, location_to_save_file)

Function to download a file from the Azure storage account. This will save the file in the location_to_save_file + file_path location

Parameters:

Name	Type	Description	Default
container(str)		Required. Name of the container. Ex: 'so-connect-migration-data'	required
file_path	str	Required. Location of the file you want to download. Must include directory/file. Example: 'Test/test.rtf'	required
location_to_save_file	str)	Required. Location of where you want to save the file locally. Must end with a / Ex: '/Users/user/Desktop/'	required

get_excel_file_from_blob_storage(container, file_path)

Function to download a file from the Azure storage account This is for excel due to the blob_data.content_as_bytes() piece, then you can read it like a normal read excel file with pd.read_excel()

Parameters:

Name	Type	Description	Default
container(str)		Required. Name of the container. Ex: 'she-data',	required
file_path	str	Required. Location of the file you want to download. Must include directory/file. Example: 'Test/test.rtf'	required

list_files_in_blob(container, container_folder_path='', print_files=False)

Function to get a list file names located within the Azure container storage.

Parameters:

Name	Type	Description	Default
container(str)		Required. Name of the container. Ex: 'she-data'	required
container_folder_path	str)	Optional. Folder path within directory. Ex: 'Metrics Charts'	''
print_files(boolean)		False does not print the file names and True will print the file names	required

upload_folder_to_blob_storage(container, toplevel_localpath, toplevel_azurepath)

Function to upload a folder with all folder paths to the Azure storage account.

Parameters:

Name	Type	Description	Default
container(str)		Required. Name of the container. Ex: 'so-connect-migration-data'	required
toplevel_localpath	str	Required. Location of the top level of the local folder you want to upload. Example: '/Users/username/Desktop/Test'	required
toplevel_azurepath	str	Required. Location of the top level of the folder that you want to upload the data to in Azure. It will create a folder on Azure with that name if the folder does not already exist. Example: 'Test'	required

upload_to_blob_storage(container, file_path, file_name, container_folder_path='')

Function to upload a single file to the Azure storage account.

Parameters:

Name	Type	Description	Default
container(str)		Required. Name of the container. Ex: 'so-connect-migration-data'	required
file_path	str)	Required. Location path to file you want to upload. Ex: '/Users/user/Desktop/test.rtf'	required
file_name	str)	Required. Desired file name. Ex: 'test.rtf'	required
container_folder_path	str)	Optional. Additional folder path within directory. Ex: 'RawData', 'FormattedData' or 'Images'	''