AX 2012 - Create a custom number sequence

Today we will learn on how to create a custom number sequence and apply the same on a table

Create a table TEC_NumberSequence with following fields:
  •  Name
  • TEC_NumSeqID
Create an EDT named TEC_NumSeqID (string type)

Make a NumberSequence of following pattern “TEC00000”. And display on form when creating the new field number sequence should auto reflect. The form should be placed in Account Payable (vendor) Module.

         A)     Create the above mentioned table and EDT

Now you should find the class named NumberSeqModuleVendor (this is the class which opens number sequence module in AP module), then go to LoadModule method in that class and write the following at last:

// ID for NumSeq Table
    datatype.parmDatatypeId(extendedTypeNum(TEC_NumSeqID));
    datatype.parmReferenceHelp(literalStr("@TEC42"));
    datatype.parmWizardIsContinuous(true);
    datatype.parmWizardIsManual(NoYes::No);
    datatype.parmWizardIsChangeDownAllowed(NoYes::No);
    datatype.parmWizardIsChangeUpAllowed(NoYes::No);
    datatype.parmSortField(1);
    datatype.parmWizardHighest(99999999);

    datatype.addParameterType(NumberSeqParameterType::DataArea, true, false);
    this.create(datatype);

Then go to Table node and find VendParameters table, go to methods node then add a new method and write the following:
    server static NumberSequenceReference TEC_NumberSequenceID()
      {
         return NumberSeqReference::findReference(extendedTypeNum(TEC_NumSeqID));  //Extended datatype
      }

Write the job to refresh and load the newly created EDT for NumberSequence
static void LoadNumberSequence(Args _args)
{
    NumberSeqModuleVendor  numberSeqModuleVendor = new NumberSeqModuleVendor();
    ;
    numberSeqModuleVendor.load();
}

Then go to AP content pane --> Setup --> Parameters --> Number Sequences Tab

Now here you will see the Extended Data type TEC_NumSeqID and an empty Sequence number code, right click the empty lookup and click Go to the main table Form.
Add a new number sequence code in the opened form and save it [for example]:
  • Number sequence code: TEC0001
  • Name: Unique number sequence ID
  • Smallest: 1
  • Largest: 99999
  • Next: 1
  • Format: TEC#####

Note: the format TEC##### has 5 #’s, so largest will also be five nine’s (99999).
Adding References is important, area – which module you want to attach your number sequence, reference – your EDT’s Label name.



Now create and Design your form



Now we want that Number Sequence in form level (Test Table):

Write below code in class declaration  :

public class FormRun extends ObjectRun
{
    NumberSeqFormHandler numberSeqFormHandler;
}

Add a new method in the existing form:-

NumberSeqFormHandler numberSeqFormHandler()
{
    if (!numberSeqFormHandler)
    {
        numberSeqFormHandler = NumberSeqFormHandler::newForm(VendParameters::TEC_NumberSequenceID().NumberSequenceId,
                                                            element,
                                                            TEC_NumberSequence_ds,
   fieldNum(TEC_NumberSequence, TEC_NumSeqID)
                             );
    }
    return numberSeqFormHandler;
}

Override the  Create(),Delete(),Write() , Validate Write(),Link Active() on the Data source methods node. Form>data source>table>Methods

Create () Method:

void create(boolean append = false,
            boolean extern = false
{
    element.numberSeqFormHandler().formMethodDataSourceCreatePre();

    super(append);

    if (!extern)
    {
        element.numberSeqFormHandler().formMethodDataSourceCreate(true);
    }
}

Delete () Method:

public void delete()
{
    element.numberSeqFormHandler().formMethodDataSourceDelete();
    super();
}

Write () Method:

public void write()
{
    super();
    element.numberSeqFormHandler().formMethodDataSourceWrite();
}

Validate Write () Method:

public boolean validateWrite()
{
    boolean         ret;
    ret = super();
    ret = element.numberSeqFormHandler().formMethodDataSourceValidateWrite(ret) && ret;
    if (ret)
    {
        TestTable.validateWrite(); //(your numbersequence table name)
    }
    return ret;
}

Link Active () Method:

public void linkActive()
{
    ;
    element.numberSeqFormHandler().formMethodDataSourceLinkActive();
    super();
}

Finally Override Close () method on form:

void close()
{
    if (numberSeqFormHandler)
    {
        numberSeqFormHandler.formMethodClose();
    }
    super();
}

Now test your form.



SQL - check invalid logins in your SQL Server

Today we will learn on how to scan all the logins created in SQL Server that are no longer valid

Run the below SQL query by replacing your SQL Server name:

IF (OBJECT_ID('tempdb..#invalidlogins') IS NOT NULL)
BEGIN
DROP TABLE #invalidlogins
END

IF OBJECT_ID('tempdb..#invalidlogins2') IS NOT NULL
DROP TABLE #invalidlogins2

CREATE TABLE #invalidlogins(
perrACCTSID VARBINARY(85)
, NTLOGIN SYSNAME)

CREATE TABLE #invalidlogins2(
ServerName Varchar(100)
, NTLOGIN SYSNAME)

INSERT INTO #invalidlogins
EXEC [<enter your server name>].master.sys.sp_validatelogins

if not exists(select * from #invalidlogins2 where NTLOGIN in (select NTLOGIN from #invalidlogins) and servername = '<enter your server name>'  )
begin
  insert into #invalidlogins2
select '<enter your server name>', NTLOGIN from #invalidlogins
end

truncate table #invalidlogins

SELECT * FROM #invalidlogins2 order by 1

IF OBJECT_ID('tempdb..#invalidlogins') IS NOT NULL
DROP TABLE #invalidlogins

IF OBJECT_ID('tempdb..#invalidlogins2') IS NOT NULL
DROP TABLE #invalidlogins2




AzCopy - copy files from on-premise location to Azure Blog Storage (ADLS)

Today we will learn on how to copy files from on-premise to Azure Blob Storage (ADLS) using AzCopy


Pre-Requisite:
  • Download AzCopy utility from here  (microsoft site)
  • Install the AzCopy.exe in the machine that you'll be using to run the copy commands
  • Open the Azcopy command line utility and start writing the commands
Steps to deploy AzCopy Script to copy Images:
  1. Authenticate a user identity:
    1. Use the script in CMD to authenticate your identity.


    • Replace the <tenant-id> placeholder with the tenant ID of the organization to which the storage account belongs. To find the tenant ID, select the Azure Active Directory > Properties > Directory ID in the Azure portal.

    • Eg.: azcopy login --tenant-id = 5dasda9fa7da79da7-abec-4505-81ea6-8csda2


Once you enter the command "azcopy login --tenant-id = <tenant-id>, it will return a code and URL of a website.
  • Open the website
  • Provide the code
  • Click Next button
  • A Sign-in window will appear.
  • Sign-in using your azure account credentials
  • After you've successfully signed in, you can close the browser window and begin using AzCopy
To login using other means (service principle or User assigned identity), get the appropriate command from the below link:
      
2. To copy files from On-prem location to ADLS, use the below command:
  • azcopy sync " <Select-folder-path>" "https:// <storage-account-name>/ <container-name> <SAS Token>" --recursive=true 
Note: Please use double quotation for file path if you’re using Windows machine. 
  • AzCopy sync will start a non-stop copy process where AzCopy will keep on copying the files unless manually terminated.
  • If you just want to just trigger the process one time, use the copy parameter instead of sync.
Copy single file:
  • azcopy cp "/path/to/file.txt" "https://[account].blob.core.windows.net/[container]/[path/to/blob]?[SAS]"
Copy entire directory:
  • azcopy cp "/path/to/dir" "https://[account].blob.core.windows.net/[container]/[path/to/directory]?[SAS]" --recursive=true
To get the SAS token:
  • Go to your storage account from Azure Portal
  • Click Shared Access Signature 
  • Copy the SAS token



Data Factory - exception handling with files with no data rows but headers only

Today we will learn on how to apply exception handling when loading csv files into Azure SQL database that has no data rows but one header line
Scenario: Suppose you want to load data into Azure SQL incrementally that from csv files generated every N minutes into Azure data lake storage (ADLS). You want to apply exception handling to make sure when empty files comes in, data factory will not fail.

Solution: Please make sure you are generating empty files (from the source) when there is no data (i.e.: without header row). Data factory will succeed if the file is completely blank. This is out of the box data factory's functionality. In case of header row, it will fail. So tweak your source logic.


Steps after source file is generated:
  • Create a Get Metadata task in data factory
    • In the Field list, specify an argument "exists"
    • We will use this argument in the if condition to see if the file exists in the folder. 
      • This is done to prevent from error "path doesn't exist / invalid source"

  • Drag "If Condition" task and connect it to the Get Metadata task (as shown above in the first picture)
    • Specify an expression to check if the file exists (dataset path is valid)
    • Expression: @and(bool(activity('check if file exists').output.exists),bool(true))

    • In the True condition, insert your incremental load logic.
      • copy activity to copy data from source to target
  • You are done.
  • Run your Pipeline.
  • Here after successfully loading data into SQL, i am moving the file to an archive ADLS container.




Note: Above image shows that now both blank files (0 bytes) as well as files with data rows are being copied to the Archive folder after successful load in SQL.
  • In case of Blank file without header row, Data factory will succeed and proceed to the next task. No data is written in the database and data factory pipeline is successful.


GCP Cloud - Capture Data Lineage with Airflow

Today we will learn on how to capture data lineage using airflow in Google Cloud Platform (GCP)


Create a Cloud Composer environment in the Google Cloud Platform Console and run a simple Apache Airflow DAG (also called a workflow).

An Airflow DAG is a collection of organized tasks that you want to schedule and run. DAGs are defined in standard Python files.
  • Enable Cloud composer API in GCP
  • On the settings page to create a cloud composer environment, enter the following:
    • Enter a name
    • Select a location closest to yours 
    • Leave all other fields as default
    • Change the image version to 10.2 or above (this is important)                     
  • Upload a sample python file (quickstart.py - code given at the end) to cloud composer's cloud storage
  • Click Upload files
  • After you've uploaded the file, cloud composer adds the DAG to Airflow and schedules the DAG immediately. It might take a few minutes for the DAG to show up in the Airflow web interface.
  • Open the Airflow web server to see the logs, view metadata, etc.

  • To see lineage, click: Data Profiling >Ad Hoc Query
    • Data Lineage is captured in the xcom table

  • Execute the following SQL Statement to view Lineage:
  • You are Done.

You can also integrate Airflow with Apache Atlas.

Airflow can send its lineage metadata with Apache Atlas. For this, you need to enable the atlas backend and configure it properly, e.g: in you airflow.cfg  (configuration):


[lineage]
backend = airflow.lineage.backend.atlas.AtlasBackend
 
[atlas]
username = <my_username>
password = <my_password>
host = <host>
port = 21000
Note: Please make sure to have the atlas client package installed. 





To install Apache Atlas on GCP:
  • Spin up a GCP compute engine instance (Linux OS)
  • Open SSH
  • Install Java
    • sudo apt install openjdk-8-jdk
  • Install Maven
    • sudo apt-get install maven 


Python file code for Lineage Testing (quickstart.py):

import airflow
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.lineage.datasets import File
from airflow.models import DAG
from datetime import timedelta

FILE_CATEGORIES = ["CAT1", "CAT2", "CAT3"]

args = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(2)
}

dag = DAG(
    dag_id='example_lineage', default_args=args,
    schedule_interval='0 0 * * *',
    dagrun_timeout=timedelta(minutes=60))

f_final = File("/tmp/final")
run_this_last = DummyOperator(task_id='run_this_last', dag=dag,
    inlets={"auto": True},
    outlets={"datasets": [f_final,]})

f_in = File("/tmp/whole_directory/")
outlets = []
for file in FILE_CATEGORIES:
    f_out = File("/tmp/{}/{{{{ execution_date }}}}".format(file))
    outlets.append(f_out)
run_this = BashOperator(
    task_id='run_me_first', bash_command='echo 1', dag=dag,
    inlets={"datasets": [f_in,]},
    outlets={"datasets": outlets}
    )
run_this.set_downstream(run_this_last)


Azure Devops - Connect Data Factory to Azure DevOps Git repository

Today we will learn on how to set the code repository in Data Factory and link it with Azure Git
There are two ways to set up code repository in Data Factory
  1. Git
  2. Azure Git
Today, we will be using Azure Git as a code repository in Data Factory.

To Link Azure Git to Data Factory, you first have to setup Azure DevOps:

  1. Go to Azure Devops (https://aex.dev.azure.com/me?mkt=en-US)
  2. Create an Organization (if not created already)
  3. Add a project in that organization
  4. Assign team members (if any) and give them the appropriate permissions:
    1. For team members to create Git repos, they should have project admin rights
    2. If you want them to only contribute to existing project/Git Repo but not have them as an admin, assign them the contributor rights.
    3. Now, go inside the project and create a Test git repo

Once you have created an organization, project and a repo inside the project, you are ready to link the Azure code repository to Data Factory. 

Your Azure Devops structure should look like below:

Let's Link Code repository to Data Factory.

Go to your data factory and click "Author and Monitor" in the Overview Section. 
  • It will open a new link on the browser and take you inside the data factory instance.
  • In the data factory Overview section, click "Setup code repository"
  • Select the below options
    • Azure Git as Code repository as repository type
    • Your Azure Devops account (Organization)
    • Your Project (created within organization)
    • Your Git repo (created inside project)
    • Collaboration Branch: you can choose master.
      • Once you publish in data factory, it will create a new branch "Adf_Publish" where it will store it's ARM deployment templates.
  • Click Apply.
  • You are done.
Go inside the data factory by clicking "Author" and you will see your code repository linked.
Enjoy :)

Microstrategy Web Dossier - Data Internationalization

Today we will learn on how to change the language of Microstrategy Web Dossier visuals based on user's default language



For Labels and data, You have two options: 
  • You can create separate tables for each lanaguage (table level)
  • You can create separate column with postfix in the same table for each language (e.g.: Column1_EN for english, Column1_NL for dutch)
For this experiment, will be using Column pattern for changing the data language

First, you need to add the new language for the Dutch user (in Microstrategy Developer):
  • Set the default language to Dutch
  • Set the number and date format to Dutch (used to change the locale)
  • Set the I-Server language to Dutch  (used when exporting reports)
  • Enable “Data Internationalization” (to be enabled at project level in Microstrategy developer)
    • Go to Project Settings > Language > Data
    • Enable “SQL based”
    • Add dutch language
    • Let the English language be default
    • In the column pattern, add “_DTCH” in from of Dutch language
    • Leave column pattern for English as blank
    • Refresh/Update the project

Azure Data Factory: Upsert using Data Flows

Today we will learn on how to perform upsert in Azure data factory (ADF) using data flows

Scenario:
  • We will be ingesting a csv stored in Azure Storage (ADLS V2) into Azure SQL by using Upsert method
Steps:

1. Create two connections (linked Services) in the ADF:
       1. One for the csv stored in ADLS
       2. One for the target Azure SQL

                                                 


Informatica Cloud - Copy Files from source to target

Today we will learn on how to copy files using Informatica cloud from source to target server using File processor connector

Steps:

1. Create an Informatica File processor connection to define source and target destination
       1. Source file directory: enter source file path (e.g.: \\Source_Server_name\directory1)
       2. Target file directory: enter target file path (e.g.: \\target_Server_name\target_dir\current)
       3. Select file: All (for selecting all the files)

Informatica Cloud - Ingest Data from Rest Api into Azure SQL


Today we will learn on how to fetch data from Rest Api and ingest into Azure SQL using Informatica Cloud data integration

Steps to Ingest data from REST API to Azure SQL DB via Informatica:

1.    Create a test SQL Server Database in Azure
·        Setup a new server for the database
·        For testing, use “Public endpoint” while setting up Azure server for database (this has to be properly setup when using for production)
·        Add your current computer’s IP address in the Azure server firewall
·        Add Informatica server’s IP address in the Azure server firewall
o   If you don’t add informatica server’s IP, Informatica will throw error “client with IP *** is not allowed to access the server”
·        Click Create the server.
·        Once the server is created, click OK to create the database

Azure Data Factory - Upsert using Pipeline approach instead of data flows

Today we will learn on how to perform upsert in Azure data factory (ADF) using pipeline approach instead of using data flows

Task:
  • We will be loading data from a csv (stored in ADLS V2) into Azure SQL with upsert using Azure data factory. Once the data load is finished, we will move the file to Archive directory and add a timestamp to file that will denote when this file was being loaded into database
Benefits of using Pipeline:
  • As you know, triggering a data flow will add cluster start time (~5 mins) to your job execution time. Although, you can make use of the Time to live (TTL) setting in your Azure integration runtime (IR) to decrease the cluster time but, still a cluster might take around (2 mins) to start a spark context. so, if you are thinking of creating a real time data load process, the pipeline approach will work best as it does not need a cluster to run and can execute in seconds.
Pre-requisites
  • An Azure Data Factory resource
  • An Azure Storage account (Gen 2)
  • An Azure SQL Database

Informatica Cloud - Copy data from SQL to Azure ADLS

Today we will learn on how to load data from SQL database to Azure ADLS as file

  1. Purchase and enable Microsoft Azure data lake (ADLS) Gen 2 connector for Informatica 
  2.  Register your informatica app with Azure AD
    1. This app will act as a service principal 
    2. Note down the client secret and client ID
    3. Give appropriate permissions to Informatica app (in Azure)
      • Storage blob data contributor
      • Internet access to Informatica to talk to Azure ADLS
  3. Create an Informatica connection to Azure ADLS Gen 2
    1. Enter the following information:
      • Client ID
      • Client Secret
      • Tenant ID
      • Account Name (Azure Storage account name)
      • File System Name (Container name within a storage account)
      • Directory Path: /   (leave as default)

Informatica Cloud - Incremental Load (Data Ingestion)

Today we will learn how to create incremental data load (ingestion) in Informatica cloud data integration

  1. Create two SQL server connections in Informatica cloud administration
    1. Connection for source database
    2. Connection for target database
  2. Create a mapping from Informatica data integration
    1. Add source
      1. Select source connection (point to the newly created source connection in the Informatica administration)
      2. Select source type as query
      3. In the query window, enter the query to get the maximum timestamp from your target table
      4. The timestamp should come from the target database table here. This max timestamp will act as a filter to fetch latest records from the source