Azure data factory

Azure data factory elements: 



ADF VS SSIS (Integration Services)


Runtimes:

1. Azure SSIS runtime - it' used when you want to lift and shift existing SSIS.

2. Self hosted runtime - it's used when you want to connect to on premises or private cloud sources.

3. Azure runtime - it's used when you want to connect azure resources which is available/access publicly.




Linked Services

1. Data Stores - Example: SQL server, cosmosdb, blob storage etc.
2. External Compute - Example: Azure ML, Data Bricks etc



Triggers:

1. Schedule  -- it will run from next schedule time only.
2. Tumbling Window -- it will run from missed days also (use properly tumbling window and schedule trigger, both looks like similar but not)
3. Event based -- it will run based on the event.


Data Flows: It's used for ETL process. 



Comments

Post a Comment

Popular posts from this blog

Azure Service Bus Azure CLI Commands

Image
ServiceBusNamespaceCommands: az servicebus namespace create --location SouthIndia --name learning-service-bus-in-depth --resource-group ramadoss-rg --sku Standard ServiceBusQueueCommands: az servicebus queue create --name demoqueue --namespace-name learning-service-bus-in-depth --resource-group ramadoss-rg ServiceBusTopicCommands: az servicebus topic create --resource-group ramadoss-rg --namespace-name learning-service-bus-in-depth --name demotopic    ServiceBusTopicSubscriptionCommands: az servicebus topic subscription create --resource-group ramadoss-rg --namespace-name learning-service-bus-in-depth --topic-name demotopic --name first-demo-subscription Commands History:  Get-History | Where-Object { $_.CommandLine -like "az *" }    2        1.082 az servicebus -h    3        0.998 az servicebus namespace -h    4        0.974 az servicebus namespace create -h    5  ...
Read more

Securing a Web API using Azure AD and Consuming it with Swagger - Step by Step Guide

Image
 Introduction: we will walk you through the process of securing a Web API using Azure AD (Azure Active Directory) and generating tokens through Swagger to call this protected API. Securing your APIs is crucial to protect sensitive data and ensure only authorized applications can access them. We'll break down the process into easy-to-follow steps. Prerequisites: Before we begin, make sure you have the following prerequisites in place: An Azure account with the necessary permissions. Visual Studio or a similar code editor. Basic knowledge of .NET Core and Swagger. Github Repository:  Secure the Web API using Azure AD Steps to Secure the Web API using Azure AD: Step 1: Create Two App Registrations We need to create two Azure AD App Registrations, one for the Web API and the other for the Swagger client application. Create a new App Registration named "EmployeeWebApi." Leave the Redirect URI empty. Create a new App Registration named "EmployeeClientApp." Click ...
Read more

Service Bus Queues and Topics

Image
Service Bus Namespace: Serves as a container for managing queues and topics. Queues: Used for point-to-point communication. When multiple consumers are involved, the queue utilizes the competing consumers pattern to distribute messages. Topics: Support a publish-subscribe model, allowing one topic to have multiple subscriptions. Messages sent to the topic are delivered to all the associated subscriptions. Features Duplicate Detection: By default, duplicate detection is handled using the message ID. If a message with the same message ID is sent again, it will not be added to the queue. With Partition Key: Duplicate detection is based on the combination of message ID and partition key. With Session ID: Duplicate detection is based on the combination of message ID and session ID. The session ID and partition key must be the same. Scenarios: If a message with message ID "1" is pushed to the queue and is not processed, pushing another message with the same ID "1" wi...
Read more