I have created python functions to execute jobs.I have also added functions to run spark-sumit commands thr my python orchestration.
I am able to run pyspark jobs [login to view URL] I need is :
[login to view URL] spark configuration to execute spark jobs(You can integrate only for a single job)
2. I have also created audit tables, somehow not able to log proper error in case of failure.
3. We might need to rewrite spark-submit execution function
[login to view URL] want to implement simple aes_encrypt function using scala and use it on csv fields inpyspark
Bu iş için 1 freelancer ortalamada ₹1500 teklif veriyor
I am 4+ years of experienced data scientist and had very good automation experience in present. I have done few automation using python , AWS s3 , AWS Athena and SQL. I am looking forward more similar automation kind Daha Fazla