AWS Glue - Unpivot columns into rows using python shell job

Today we will learn on how to unpivot columns into rows using AWS Glue python shell job

Pre-requisites:

• You will need to download two python libraries (Pandas and sf3s)
• Pandas Library:
•  Link: https://files.pythonhosted.org/packages/c0/95/cb9820560a2713384ef49060b0087dfa2591c6db6f240215c2bce1f4211c/pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl
• Version: pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl
• S3fs Library:
• Link: https://pypi.org/project/s3fs/#files
• Version: s3fs-0.4.2-py3-none-any.whl  

Steps:

• Upload the above downloaded libraries in a S3 bucket
• Create a new AWS Glue job
• Type: python shell
• Version: 3
• In the Security configuration, script libraries, and job parameters (optional) > specify the python library path to the above libraries followed by comma ","
• E.g.: s3://library_1.whl, s3://library_2.whl
• import the pandas and s3fs libraries 
• Create a dataframe to hold the dataset
• Call the dataframe.melt() function to unpivot columns into rows

• Save and run the job
• You are done.