Ingest data from external REST API into S3 using AWS Glue

Today we will learn on how to ingest weather api data into S3 using AWS Glue 

Steps:

• Create a S3 bucket with the below folder structure:
• S3BucketName
• Libraries
• Response.whl

• Download python response library (in .whl format) and save in the Libraries folder within S3
• Download link: https://pypi.org/project/requests/#files

• Sign up in Openweathermap website and get the api key to fetch the weather data
• Link: https://openweathermap.org/
• Create a new Glue ETL job
• Type: Python Shell
• Python version: <select latest python version>
• Python Library Path: <select the Response.whl library path>
• This Job runs: <A new script to be authored by you>
• Click Next
• Click "Save job and edit Script"

• Import response library
• import boto3 library for saving in S3 bucket
• Write the code to ingest data 

• Run the glue job

                                    

• View the glue job results 
• Job run status = Succeeded

• Verify if the data is saved in S3 bucket

• Download the saved json file from S3 and check if it is correct
• You are done. Cheers!

Informatica Cloud - Ingest Data from Rest Api into Azure SQL

Today we will learn on how to fetch data from Rest Api and ingest into Azure SQL using Informatica Cloud data integration

Steps to Ingest data from REST API to Azure SQL DB via Informatica:

1.    Create a test SQL Server Database in Azure

·        Setup a new server for the database

·        For testing, use “Public endpoint” while setting up Azure server for database (this has to be properly setup when using for production)

·        Add your current computer’s IP address in the Azure server firewall

·        Add Informatica server’s IP address in the Azure server firewall

o   If you don’t add informatica server’s IP, Informatica will throw error “client with IP *** is not allowed to access the server”

·        Click Create the server.

·        Once the server is created, click OK to create the database