I am in need of constructing the final dataframe name dynamically based on the configuration (by joining final_df and suffix). Whenever I execute the code provided below, it throws an error - "SyntaxError: can't assign to operator". Strangely, if I replace each["final_df"]+'_'+ each["suffix"] with a different name, the code works flawlessly.
Data :
df_source_1 = spark.createDataFrame(
[
(123,10),
(123,15),
(123,20)
],
("cust_id", "value")
)
Configuration:
config = """
[
{
"source_df":"df_source_1",
"suffix": "new",
"group":["cust_id"],
"final_df": "df_taregt_1"
}
]
"""
Code:
import json
for each in json.loads(config):
print("Before=",each['final_df'] ) # str object
print(each["final_df"]+'_'+ each["suffix"]) # df_taregt_1_new , print statement works
each["final_df"]+'_'+ each["suffix"] = eval(each["source_df"]).groupBy(each["group"]).agg(sum("value")) # Errors out. Here I need to assign the dataframe to df_taregt_1_new
Any assistance would be greatly appreciated.