Questions tagged [azure-data-factory]

Microsoft Azure Data Factory is a cutting-edge cloud platform that efficiently coordinates and manages various Azure or external services, controlling their execution based on time slices. With its impressive capabilities, ADF enables seamless parallel data transformations. The development of data factories predominantly involves the meticulous creation of bespoke JSON scripts, providing the necessary guidance for the tool to execute specific activities.

Tips for manipulating JSON data in Azure Data Factory

Received the data in env_variable.json using a Lookup variable and looking to extract "NO" and "BR" programmatically for iteration within a ForEach activity. The content of the file is as follows: { "countries" : { "NO" : { "wells": ["0015/abcd"] }, "BR" ...

pythonjsonazure-data-factory

Discovering insights from JSON files in Data Lake Gen 2 using Azure Data Factory

Currently, I have an ADF pipeline that loads raw log data in the form of JSON files into a Data Lake Gen 2 container. Our objective now is to extract valuable information from these JSON files. My aim is to discover the most efficient method for extractin ...

jsonazure-data-factoryazure-data-lakeazure-data-lake-gen2

Transforming dynamic JSON data into SQL Server using Azure Data Factory

I've been attempting to flatten a dynamic JSON structure and insert the resulting data into a SQL Server table without success. I've explored several methods thus far: Copy Activity: The JSON data from a datab ...

jsonsql-serverazure-data-factoryflatten

Implementing date values in Rest API parameter with Azure Data factory

One of the parameters in my API Url is TIMESTAMP, with a specific value like: Actual Url: The timestamp value could expire, so I've created a parameter TS with the value utcNow() to prevent that. Now, my URL looks like: I'm using this updated ...

jsonazureazure-data-factoryrest

Transforming API data with complex nested JSON structure into a CSV file using Azure Data Factory (

After receiving a Json file from the Log Analytics API in ADF, I am attempting to convert it into a CSV file. The structure of my Json file is as follows: { "tables": [ { "name": "PrimaryResult", ...

jsonazurecsvazure-data-factoryflatten

Writing to Json files in Azure Data Factory

Can I get your thoughts on this? So, working in Azure Data Factory, I have a series of activities that generate a JSON segment at the end of each run {"name":"myName", "email":"<a href="/cdn-cgi/l/email-protection" cla ...

jsonazureazure-functionsazure-data-factory

Encountering problems with the build script in an npm package

We have been utilizing Azure DevOps pipeline for constructing the Azure Data Factory (ADF) ARM template. The process I followed is in alignment with the guidelines provided by Microsoft, which can be found here. To proceed with this build, it is necessary ...

node.jsnpmcontinuous-integrationazure-data-factorynpm-scripts

ADF CI Build Failure: Execution unsuccessful: node /home/shaadmin/myagent/_work/5/s/adf/build/downloads/main.js validate

Yesterday, our ADF CI Build was running smoothly based on the configuration outlined in the [Microsoft documentation][1]. However, today we encountered a sudden failure with the following error message: Command failed: node /home/shaadmin/myagent/_work/ ...

azurenpmazure-devopsazure-data-factoryazure-pipelines-yaml

Challenge with JSON Output Formatting in Azure Data Factory

I'm facing a problem with formatting the JSON output in Azure Data Factory. The structure I want for my JSON output is: { "AccessionNum": "12345", "test": [ { "orderunit": "JMJ", "Testname": "TVK" }, { "orderunit": "P ...

jsonazureazure-data-factoryazure-logic-appscustom-data-type

Parsing JSON file using U-SQL

Can anyone assist with parsing this Json file using USQL? I keep encountering errors. Json file@ {"dimBetType_SKey":1,"BetType_BKey":1,"BetTypeName":"Test1"} {"dimBetType_SKey":2,"BetType_BKey":2,"BetTypeName":"Test2"} {"dimBetType_SKey":3,"BetType_BKey" ...

jsonazureazure-data-factoryazure-data-lakeu-sql