Today we will learn on how to ingest weather api data into S3 using AWS Glue
Steps:
- Create a S3 bucket with the below folder structure:
- S3BucketName
- Libraries
- Response.whl
- Download python response library (in .whl format) and save in the Libraries folder within S3
- Download link: https://pypi.org/project/requests/#files
- Sign up in Openweathermap website and get the api key to fetch the weather data
- Create a new Glue ETL job
- Type: Python Shell
- Python version: <select latest python version>
- Python Library Path: <select the Response.whl library path>
- This Job runs: <A new script to be authored by you>
- Click Next
- Click "Save job and edit Script"
- Import response library
- import boto3 library for saving in S3 bucket
- Write the code to ingest data
- Run the glue job
- View the glue job results
- Job run status = Succeeded
- Verify if the data is saved in S3 bucket
- Download the saved json file from S3 and check if it is correct
- You are done. Cheers!
Please bear in mind this tutorial needs to be adjusted for the latest state of the AWS Glue Python script runner.
ReplyDelete1) Python libraries for requests and json do not need to be bundled as a dependency. So, the WHL library import can be dropped.
2) Putting the guarded main invocation doesn't work. Remove the guards to get it to run
This was an interesting walkthrough of building a weather data pipeline using AWS services. Bringing weather API data into Amazon S3 using Lambda is a practical example of how cloud-native architectures can automate data collection and storage with minimal operational overhead. I especially liked how the workflow demonstrates the integration between APIs, serverless computing, and cloud storage, which are common building blocks in modern data engineering projects. Those interested in similar cloud-based implementations can also explore Cloud Computing Projects for more ideas on scalable cloud architectures and automation.
ReplyDeleteAnother valuable aspect of this post is how it introduces a real-world data ingestion scenario rather than a purely theoretical example. Collecting, storing, and processing external data sources is an important skill for anyone working with analytics or data platforms. Students looking to gain experience in handling large datasets and data pipelines can also check out Big Data Projects, which cover many of the concepts used in modern data processing environments.
ReplyDeleteThis article presents a useful practical approach for ingesting weather data from an external REST API into Amazon S3 using AWS Glue. The step-by-step structure, including the S3 bucket and Libraries folder arrangement, makes the workflow easy to understand. It also highlights an important aspect of data engineering: connecting external API sources with cloud storage for centralized data processing and downstream analytics.
ReplyDeleteThe workflow demonstrates how API-based data ingestion can be incorporated into a cloud data pipeline, while the inclusion of external library dependencies shows the practical considerations involved in configuring an AWS Glue job. Understanding REST-based integration is valuable when building automated ingestion workflows, and resources such as RESTful API Course can provide useful background for working with API-driven data sources.
It is also helpful that the example uses weather data, as this provides a realistic scenario for retrieving frequently changing information and storing it in S3 for further processing. The combination of cloud ETL and API integration demonstrates how different technologies can work together in a data pipeline, while learning about FastAPI Course can further support understanding of modern API development and integration concepts.
ReplyDelete