Apache Spark Jobs

We are seeking an experienced Data Engineer to build and maintain scalable, high-performance data pipelines and infrastructure for our next-generation data platform. The platform ingests and processes real-time and historical data from diverse industrial sources such as airport systems, sensors, cameras, and APIs. You will work closely with AI/ML engineers, data scientists, and DevOps to enable reliable analytics, forecasting, and anomaly detection use cases. Shape Major Skills : Spark, Flink, Iceberg Key Responsibilities · Design and implement real-time (Kafka, Spark/Flink) and batch (Airflow, Spark) pipelines for high-throughput data ingestion, processing, and transformation. · Develop data models and manage data lakes and warehouses (D...

I need a clear, scalable architecture that can detect and instantly alert on unusual spending patterns in our payment flow. The focus is narrow and well-defined: draw insights from live and historical transaction history, spot deviations from normal customer behavior, and trigger actionable alerts before settlement is completed. Scope of work • Map the entire transaction-processing path and identify points where real-time analytics can be inserted without adding noticeable latency. • Define the data lake or streaming layer that will store raw and enriched transaction history. • Recommend the analytics engine—rules, machine-learning models, or a hybrid—best suited for spotting spending anomalies while remaining extensible to other fraud signals later (for e...