Today we will learn on how to unpivot columns into rows using AWS Glue python shell job
Pre-requisites:
- You will need to download two python libraries (Pandas and sf3s)
- Pandas Library:
- Link: https://files.pythonhosted.org/packages/c0/95/cb9820560a2713384ef49060b0087dfa2591c6db6f240215c2bce1f4211c/pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl
- Version: pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl
- S3fs Library:
- Link: https://pypi.org/project/s3fs/#files
- Version: s3fs-0.4.2-py3-none-any.whl
Steps:
- Upload the above downloaded libraries in a S3 bucket
- Create a new AWS Glue job
- Type: python shell
- Version: 3
- In the Security configuration, script libraries, and job parameters (optional) > specify the python library path to the above libraries followed by comma ","
- E.g.: s3://library_1.whl, s3://library_2.whl
- import the pandas and s3fs libraries
- Create a dataframe to hold the dataset
- Call the dataframe.melt() function to unpivot columns into rows
- Save and run the job
- You are done.
ReplyDeleteEscorts Service in Paharganj
Call Girls in Mahipalpur
Female Escorts Job in Delhi
Charges of Call Girls in Gurgaon
Russian Escorts in Paharganj, New Delhi
Escorts Service in Manesar
The article demonstrates how AWS Glue Python Shell jobs can use the Pandas melt() function to unpivot datasets by converting columns into rows. This transformation is commonly required during ETL processes to reshape wide datasets into a normalized format that is easier to analyze, integrate, and load into data warehouses. By combining AWS Glue with Pandas and S3, developers can build efficient serverless data processing pipelines for cloud-based analytics.
ReplyDeleteAWS Glue is a fully managed ETL service that simplifies data extraction, transformation, and loading without requiring infrastructure management. It integrates seamlessly with Amazon S3, Python libraries, and cloud data lakes, enabling organizations to automate large-scale data preparation and workflow orchestration. Students interested in building cloud-based data processing solutions can explore Big Data Projects, featuring practical implementations involving AWS Glue, Apache Spark, ETL pipelines, cloud analytics, and distributed data processing.
Pandas provides flexible DataFrame transformation functions such as melt(), pivot(), and groupby() that simplify reshaping and preprocessing structured datasets. These capabilities make Pandas an essential library for data engineering, analytics, and machine learning workflows where high-quality data preparation is critical. Those looking to strengthen their practical DataFrame manipulation skills can further explore Pandas Training, covering hands-on techniques for data cleaning, transformation, aggregation, and analysis.
ReplyDeleteReaders interested in exploring modern cloud-based ETL and large-scale data processing can also refer to 15 Big Data Projects for Final Year Students, which showcases practical project ideas covering AWS-based analytics, Hadoop, Apache Spark, ETL workflows, cloud data engineering, and enterprise big data applications.
ReplyDelete