Data Solutions

Search This Blog

Contact Me

Posted by Tushar Bhalla

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

I'd love to hear from you! |

Subscribe to my Feeds

Name

Email *

Message *

Get link
Facebook
Twitter
Pinterest
Email
Other Apps

Move file from one S3 folder to another using AWS Glue

Today we will learn on how to move file from one S3 location to another using AWS Glue Steps: Create a new Glue Python Shell Job Import boto3 library This library will be used to call S3 and transfer file from one location to another Write the below code to transfer the file Change the bucket name to your S3 bucket Change the source and target file path Run the job Check whether the file has been transferred to the target location or not. You are done! :)

Upload a file to Azure Blob Storage (ADLS Gen 2) using Python

Today we will learn on how to upload a csv file to Azure Blob Storage (ADLS) using Python Steps: Create a file to be uploaded to Azure Storage Create a python script Install Azure package from pip pip install azure-storage-file-datalake Import the azure module import os, uuid, sys from azure.storage.filedatalake import DataLakeServiceClient from azure.core._match_conditions import MatchConditions from azure.storage.filedatalake._models import ContentSettings Create a connection to your Azure storage in the python script service_client = DataLakeServiceClient(account_url="{}://{}.dfs.core.windows.net".format("https", 'Storage_account_name'), credential='Storage_Account_Key') Specify the container name file_system_client = service_client.get_file_system_client(file_system="your_container_name") Specify the directory in your Azure storage directory_client = file_system_client.get_directory_client("my-directory") Create a blank txt f

AWS EMR: Read CSV file from S3 bucket using Spark dataframe

Today we will learn on how to use spark within AWS EMR to access csv file from S3 bucket Steps: Create a S3 Bucket and place a csv file inside the bucket SSH into the EMR Master node Get the Master Node Public DNS from EMR Cluster settings In windows, open putty and SSH into the Master node by using your key pair (pem file) Type "pyspark" This will launch spark with python as default language Create a spark dataframe to access the csv from S3 bucket Command: df.read_csv("<S3 path to csv>",header=True,sep=',') Type "df_show()" to view the results of the dataframe in tabular format You are done

Powered by Blogger

(C) TUSHAR BHALLA