develop end to end data pipeline in hadoop

1. Develop/Propose various ETL pipelines for the combination of hadoop eco-system with/NOT spark using AWS cluster;

2. Implement the multi-purpose ELT pipeline (like kappa or lambda architecture) that ingress data from various sources (NOT limited to mySQL, Oracle, NoSQL, Flat files like PDF, Excel, CSV, .docx, Live/streaming) to hadoop/hdfs by using wide range of bigdata tools;

3. Ensure the data quality like format, fields, precision, No. of rows etc.., during data migration from data sources to HDFS;

4. Pipeline must able to handle huge data in GB's;

5. Troubleshoot by creating breakpoint in the ETL pipeline at various levels (at each hadoop eco-system tool) like resource level & code level towards memory management, performace tuning, optimization;

Beceriler: Amazon İnternet Servisleri, Veri İşleme, Veri Depolama, Hadoop, Spark

Daha fazlasını gör: data entry front end excel, develop linux multimedia front end, develop data extract ssis, data pipeline course, data engineering pipeline, what is data pipeline in hadoop, data analysis pipeline, data pipeline tutorial, data pipeline definition, data pipeline examples, how to create data pipeline in hadoop, use vba access develop data centre, develop data access layer vbnet, collect user data joomla front end, excel data input front end, data entry front end, develop data access layer asp net , develop data entry screen mobile phone data entry sending sms, spreadsheet data entry front end, collect data joomla front end

İşveren Hakkında:
( 0 değerlendirme ) Atlanta, United States

Proje NO: #19016439

Bu iş için 8 freelancer ortalamada $176 teklif veriyor


Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Daha Fazla

$250 USD in 4 gün içinde
(179 Değerlendirme)

Have 10 years of IT experience with more than 4.5 years of experience in hadoop technologies like hive,pig,spark,sqoop,map reduce and [login to view URL] have very good experiemcence in Java,scala,Python and shell scripting. Daha Fazla

1 gün içinde %bids___i_sum_sub_32%%project_currencyDetails_sign_sub_33% USD
(11 Değerlendirme)

Hello there, We are a team of expert Big Data developers with more than 10 years of rich inductry experience & have succesfully delivered multiple projects in the past like a)recipe recommendation b)movie recom Daha Fazla

$333 USD in 3 gün içinde
(2 Değerlendirme)

Hi, I have 8 years of experience and working on hadoop, spark, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud with ha Daha Fazla

1 gün içinde %bids___i_sum_sub_32%%project_currencyDetails_sign_sub_33% USD
(3 Değerlendirme)

We have great young and dedicated team, trustworthy and proper completion of project is our moto. Please let me know if you are intesred

1 gün içinde %bids___i_sum_sub_32%%project_currencyDetails_sign_sub_33% USD
(0 Değerlendirme)

Hello there, Gone through your details and after reading over your application this looks like a perfect fit for my skill sets!! // Please ping me here so we can discuss details and after analysis, I will provid Daha Fazla

$200 USD in 3 gün içinde
(0 Değerlendirme)

I have built several data pipelines using AWS and can definitely design and develop the solution. I will need more details however on this.

$166 USD in 2 gün içinde
(0 Değerlendirme)

I work with similar use cases on day to day basis. I have around 3 years of hands on experience with big data technologies. Along with work I can help you with understanding. Reach out to me if you are interested.

$155 USD in 3 gün içinde
(0 Değerlendirme)