As a beginner in AWS Glue and Pyspark, I'm facing some challenges with a transformation task. My issue involves working with two DynamicFrames; one contains values in a specific column that need to be added as a new column in the other DynamicFrame. The values in this column should correspond to values from another column with matching IDs in the first table. Here is an example of the initial data structure:
Table 1 Table2
+--+-----+-----+ +--+-----+-----+
|id|name |value| |id|col1 |col2 |
+--+-----+-----+ +--+-----+-----+
| 1|name1| 10 | | 1|str1 |val1 |
+--+-----+-----+ +--+-----+-----+
| 2|name2| 20 | | 2|str2 |val2 |
+--+-----+-----+ +--+-----+-----+
The desired output format should look like this:
Table2
+--+-----+-----+-----+-----+
|id|col1 |col2 |name1|name2|
+--+-----+-----+-----+-----+
| 1|str1 |val1 | 10 | | <--- add 10 only here because the id from the row in the first table must match the id from the second table
+--+-----+-----+-----+-----+
| 2|str2 |val2 | | 20 | <--- add 20 only here because the id from the row in the first table must match the id from the second table
+--+-----+-----+-----+-----+