Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using WatermarkIn a data integration solution, incrementally (or delta) loading data after an initial full data load is a widely used scenario. The tutorials in this section show you different ways of loading data incrementally by using Azure Data Factory. In this case, you define a watermark in your source database. A watermark is a column that has the last updated time stamp or an incrementing key. The delta loading solution loads the changed data between an old watermark and a new watermark. 00:00 Introduction -Agenda of Demo 03:30 Create Azure SQL Database 05:30 Create Azure Data Lake 06:15 Create an Azure Data Factory 07:35 Create required tables/SP in Azure SQL 10:00 Create Linked Services 12:40 Create ADF Pipeline 20:30 Run ADF Pipeline 21:30 Verify Azure Data Lake for File 22:40 Update Azure SQL Records 24:20 Run ADF Pipeline 25:10 Verify Azure Data Lake for Delta File
Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using WatermarkIn a data integration solution, incrementally (or delta) loading data after an initial full data load is a widely used scenario. The tutorials in this section show you different ways of loading data incrementally by using Azure Data Factory. In this case, you define a watermark in your source database. A watermark is a column that has the last updated time stamp or an incrementing key. The delta loading solution loads the changed data between an old watermark and a new watermark. 00:00 Introduction -Agenda of Demo 03:30 Create Azure SQL Database 05:30 Create Azure Data Lake 06:15 Create an Azure Data Factory 07:35 Create required tables/SP in Azure SQL 10:00 Create Linked Services 12:40 Create ADF Pipeline 20:30 Run ADF Pipeline 21:30 Verify Azure Data Lake for File 22:40 Update Azure SQL Records 24:20 Run ADF Pipeline 25:10 Verify Azure Data Lake for Delta File
Comments
Post a Comment