Airflow , Kafka Kinesis Project

$8-15 USD / hour

Kapalı

İlan edilme:

3 yıldan fazla önce

$8-15 USD / hour

A>Airflow 1. Take any small source CSV file and copy it into target S3 using Airflow 2. Take any CSV in S3 and copy that into target local server using airflow 3. On local server, airflow calls a simple python file , that reads say a json and pushes data into output say CSV schedule above one after anotther to run every ten minutes ( since the files are small, we can run them back to back) B> Kafka (and / or) Kinesis 1. Take any Kafka producer code that writes into topic , pyspark consumer reads from topic 2. Say the kafka producer has hundred records. consumer reads say thirty records and goes down. Now the kafka consumer should be restarted from thirty first record. This is part of off-set management. Checkpoints to be created. feel free to take up PartA> Airflow or PartB> Kafka &/Kinesis ,separately based on your preference.

Python

Software Architecture

Linux

Data Processing

ETL

Proje No: 28252219

Proje hakkında

4 teklif

Uzaktan proje

Son aktiviteden bu yana geçen zaman 3 yıl önce

Biraz para mı kazanmak istiyorsunuz?

E-posta adresi

Freelancer'da teklif vermenin faydaları

Bütçenizi ve zaman çerçevenizi belirleyin

Çalışmanız için ödeme alın

Teklifinizin ana hatlarını belirleyin

Kaydolmak ve işlere teklif vermek ücretsizdir

4 freelancer bu proje için ortalama $18 USD/ saat teklif veriyor

@maadirajudata

Hi I am interested in part b Kafka producer and consumer. Hope it can be achieved in local environment please confirm I have expertise in Kafka based applications thanks maadiraju

$22 USD 20 gün içinde

4,9

(31 değerlendirme)

5,5

@anzarulislam

Hello, I want to do your Airflow , Kafka Kinesis Project But I have few questions. Can I ask? Let's talk and start now. Best, Anzarul.

$15 USD 40 gün içinde

4,7

(19 değerlendirme)

5,1

@DevraniAnuj

Hi I have worked with Airflow extensively. I am interested in the A part. Please explain in detail so that we can get started quickly.

$17 USD 10 gün içinde

0,0

(0 değerlendirme)

0,0

@bharatthe1

Greetings, How are you! After going through your requirements, I am interested in Part A and I would like to share that I can fulfill your requirement. Below is brief description about my self. I am having 8+ years of working experience in Technologies like Python, Java and other big data technologies like Spark, Hive and Hadoop. Also developed many spark applications in AWS by implementing AWS Services like Lambda, Step Functions, SNS, Glue, Redshift. Also I have created Airflow DAGs by creating custom hooks and operators to connect to s3, Lambda, EMR, and SSM(System Manager) and deployed these schedulers in our prod implementation. With above experience and skillset, I strongly feel that you will be highly satisfied with the quality of work and on time delivery. You can reach out to me any time. Looking forward to hear from you! Thanks.

$18 USD 40 gün içinde