Hadoop Jobs

I need an experienced AWS data engineer to design and build production-ready ETL pipelines in AWS Glue. The work centres on moving and transforming data from several source systems—relational databases (e.g., PostgreSQL, MySQL), NoSQL stores, real-time streams coming through Kafka/Kinesis, and a handful of internal/external REST APIs—into a clean, query-friendly layout in S3 and, ultimately, Redshift. Requirements: Strong AWS data engineering experiences with various aws services Experience building end-to-end data pipelines (schema discovery, ingestion, transformation, orchestration, monitoring) Experience working with relational databases like Oracle, MySQL, and SQL Server etc Experience with data ingestion from on-prem systems to cloud Experience with streaming platforms...

I need a seasoned DevOps engineer to stand up and run the entire backend that powers our transport-tracking platform. The system has to ingest GPS data from roughly 300 buses every 10 seconds, which works out to about 2.5 million write events each day, so resilience and low-latency processing are critical. Platform & core stack • Cloud: we’ll build everything on AWS (EC2/ECS/EKS, VPC, IAM, S3, Route 53 ‑ whatever fits best). • Messaging: Redis is my first choice for the real-time pub/sub layer, though I’m open to Kafka if you can justify the trade-offs. • Monitoring: Prometheus for metrics, with dashboards in Grafana; CloudWatch can complement for AWS-native alerts and logs. Key things I expect you to deliver • A fully scripted, infrastructur...

I’m building a virtual DeepseekV3 environment that emulates Jet Nano hardware for research and development on machine-learning models. The goal is to give my team a sandbox where we can move seamlessly from data preprocessing and feature extraction through model training, evaluation, deployment, and monitoring—without touching the physical board until we are ready. Here’s what I need: • A reproducible simulation that mirrors Jet Nano’s CUDA-enabled GPU, memory constraints, and I/O. • Containerised tool-chain (PyTorch, TensorRT, cuDNN, etc.) with scripts that cover the full life-cycle: preprocessing, training, hyper-parameter sweeps, evaluation metrics, and a mock-deployment stage that tracks resource usage and latency. • Clear documentation so a...

For the next round of hiring I want an accomplished Senior Data Engineer to sit in on our technical interviews for roughly two hours each day. The role is purely evaluative: you will craft probing questions, join live video calls, and quickly score each candidate’s depth of knowledge across Python, Scala and SQL. Our stack centres on Azure and Databricks, so practical insight into large-scale Spark/PySpark jobs, data-model design, ETL orchestration and cloud performance tuning is essential. Candidates frequently discuss streaming, optimisation strategies and modern AI/ML add-ons, so any hands-on exposure to libraries such as PyTorch, NumPy, SciPy or TensorFlow will help you challenge them at the right level, though it is not mandatory. Availability is limited to two focused hou...