Hadoop Jobs

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

The goal is to design a future-proof relational backbone for a multi-channel retail platform, built natively on PostgreSQL. I already run an operational system, and its live transactional records will have to be migrated into the new structure without downtime. Everything else—users, inventory, orders—will start fresh. Core design expectations The operational schema should be fully normalised to 3NF and optimised from day one for high concurrency. Think table partitioning (range and list), well-chosen GIN and B-Tree indexes, plus JSONB columns where semi-structured flexibility makes sense. Triggers, constraints and stored procedures must enforce business logic consistently, so the application tier can stay lightweight. Analytics layer Alongside the OLTP schema I need a se...

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

I have a Hadoop cluster holding several large data sets, and I need a seasoned PySpark developer who also writes rock-solid SQL. The immediate aim is to connect to the cluster (YARN/HDFS with Hive metastore), develop or refine PySpark jobs, optimise the accompanying SQL, and make sure everything runs smoothly end-to-end. You’ll receive access to a staging namespace plus a sample of the data. Once the logic checks out we’ll promote the code to the full environment. Deliverables • A clean, well-commented PySpark notebook or .py job that executes successfully on the cluster • The corresponding SQL script or view definitions ready for Hive or spark-sql • A concise README detailing execution steps, parameters, and expected outputs Acceptance criteria &bul...

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

Hiring: DevOps Developers DevOps Engineer Requirements: -Expertise in CentOS, Ubuntu, Debian -Experience with HBase, Hadoop, Storm, ArangoDB, Prometheus -Strong benchmarking and system scalability testing skills Responsibilities: -Configure and optimize high-availability servers and databases -Implement monitoring and performance evaluation tools -Review and enhance DevOps workflows Long-term opportunities available!

- **Core Architecture:** Spring Cloud + Kafka + Hadoop + Python Automation，This project requires a certain level of technical expertise.

I have an existing analytics initiative that now needs a dedicated Redshift-based warehouse. The core objective is to design and implement a robust schema in Amazon Redshift, then ingest data coming from three different sources—our operational SQL databases, a set of RESTful APIs, and periodic flat-file drops in CSV or JSON. Here is what I’m aiming for: • A well-structured Redshift warehouse (star or snowflake schema, whichever is most appropriate) built to scale and documented clearly. • Reliable, automated ingestion pipelines for each source type. For SQL we currently use PostgreSQL and MySQL; for APIs the payloads are mostly JSON; the flat files live in S3. • Transformations that standardise data types, handle slowly changing dimensions, and enforce dat...