
Closed
Posted
Microsoft Fabric / PySpark Data Engineer Needed We need an experienced Microsoft Fabric and PySpark freelancer to work with our backend team on modernizing a high-volume SQL-based data processing platform. Currently, large raw datasets are loaded into SQL Server and processed through multiple stored procedures for data quality, validation, transformations, and business rules. We want to move the heavy data-processing components to Fabric Lakehouse + Delta + Spark/PySpark, while keeping appropriate workloads in SQL/Fabric Warehouse. You should be able to: Build Fabric Lakehouse/OneLake architecture Develop Spark/PySpark transformations Work with Delta tables Build/modify Fabric or ADF pipelines Convert suitable SQL stored-procedure workloads to Spark Implement incremental/batch processing Optimize processing of very large datasets Integrate curated data with Fabric Warehouse/SQL Work closely with our existing backend developers Must have: Microsoft Fabric, PySpark/Spark, Delta Lake, SQL, Fabric Warehouse, and ADF/Fabric Data Factory experience. This is a hands-on development role, not an architecture-only engagement.
Project ID: 40775173
138 proposals
Remote project
Active 2 days ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
138 freelancers are bidding on average $34 USD/hour for this job

Greetings, One thing I would pay close attention to in this migration is keeping the existing business rules intact while moving the processing from SQL Server to PySpark. Even when the new pipelines run successfully, small differences in validation, null handling or transformation logic can produce different results. Do you have a set of sample datasets and expected outputs that we can use to compare the existing stored procedures with the new PySpark transformations? Our team will work with your backend developers to migrate the processing logic, organize the data in Fabric Lakehouse and Delta tables, and configure the required pipelines. We’ll also check the results against the current SQL processing and test performance with representative data volumes before moving the workloads into production. Timeline and budget are tentative and can be finalized after few questions. Let's discuss further thru' chat. Regards, YK LEADconcept P.S: Relevant work examples can be shared thru' chat.
$30 USD in 40 days
8.6
8.6

The key is not to translate stored procedures line by line. I will first separate set-based SQL workloads from transformations that benefit from distributed Spark processing, then design idempotent Delta pipelines that can be rerun safely without duplicates or partial results. I will profile the current procedures, dependencies, data volumes, and SLAs; define Bronze/Silver/Gold Lakehouse layers; and implement PySpark transformations with explicit schemas, data-quality checks, incremental watermarks, MERGE logic, partitioning, and audit tables. Fabric Data Factory/ADF pipelines will handle orchestration, retries, parameters, and monitoring. Curated Delta data will be exposed to Fabric Warehouse/SQL where relational serving remains the better fit. Performance work will include reducing shuffles, controlling file sizes, addressing skew, optimizing joins, and validating output against the existing SQL process before cutover. I can work directly with your backend developers and deliver production code rather than architecture documents only. I have a few questions: Q1. What are the largest table sizes, daily increments, and required processing SLA? Q2. Is there a priority stored-procedure chain for the first migration milestone? To begin, I will need Fabric workspace access, pipeline definitions, table schemas, stored procedures, sample data, and current runtime metrics.
$40 USD in 40 days
7.8
7.8

Hello, We carefully studied the description of your project and we can confirm that we understand your needs and are also interested in your project. Our team has the necessary resources to start your project as soon as possible and complete it in a very short time. We are 25 years in this business and our technical specialists have strong experience in Data Processing, SQL, Oracle, MySQL, Database Administration, Data Integration, Data Architecture, PySpark and other technologies relevant to your project. Please, review our profile https://www.freelancer.com/u/tangramua where you can find detailed information about our company, our portfolio, and the client's recent reviews. Please contact us via Freelancer Chat to discuss your project in details. Best regards, Sales department Tangram Canada Inc.
$30 USD in 5 days
7.8
7.8

Move the scan-heavy stored procedures to PySpark first, not the entire SQL chain. Use SQL Server execution statistics to identify the biggest reads before rewriting anything; small transactional steps may be cheaper to keep in SQL. We'll build a first migration slice in 7 days: OneLake landing tables, a converted validation/transformation flow, and curated output connected to Fabric Warehouse. We'll trace dependencies with your backend developers, preserve business-rule results, and orchestrate incremental loads through Fabric Data Factory or existing ADF pipelines. Delta merge keys and Spark join plans will be chosen from the actual data distribution, rather than defaulting to full reloads. This is the first milestone, not a deadline for the whole platform. Are your stored procedures already ranked by runtime? Sid Techno LLC
$35 USD in 40 days
7.2
7.2

Hi, I’m an experienced backend developer with a strong background in SQL, database-driven applications, data processing, API integrations, and performance optimization. I’m interested in helping your team modernize its existing SQL-based data processing platform. I can review your current stored procedures, understand the validation and transformation logic, and help migrate suitable workloads to a Lakehouse-based processing approach. My proposed approach includes: - Reviewing existing SQL queries, stored procedures, and business rules. - Developing and optimizing data transformation logic. - Working with SQL and PySpark for batch and incremental processing. - Supporting Delta table workflows and pipeline integration. - Validating migrated results against the existing SQL implementation. - Collaborating with your backend team to ensure reliable integration and maintainable code. I would first review your current architecture, data volumes, and processing requirements to identify the most suitable implementation approach. I’m happy to discuss the technical requirements and provide relevant examples of my work. Best regards, Mangal Singh
$25 USD in 40 days
7.0
7.0

Hi there, I have carefully reviewed your brief and understand you need to modernize a high-volume SQL processing platform by moving computationally intensive transformations from stored procedures to Microsoft Fabric Lakehouse, Delta Lake, and PySpark while preserving reliable integration with your existing SQL environment. My approach would focus on three stages which includes workload assessment, hands-on development, and performance optimization. First, I’ll review your existing stored procedures, business rules, data volumes, and processing dependencies to identify workloads best suited for Spark. Next, I’ll implement PySpark transformations, Delta tables, incremental processing, and Fabric Data Factory pipelines, while integrating curated datasets with Fabric Warehouse where SQL remains the better fit. Finally, I’ll validate results against existing SQL outputs, optimize partitioning and execution performance, and test pipeline reliability on large datasets. Are your current stored procedures primarily batch-based, and do you already have a Fabric Lakehouse environment available for development and testing? I can also help prioritize the highest-cost processing stages first, allowing your team to measure performance improvements while minimizing disruption to existing backend services. Please initiate a chat so we can discuss the technical requirements and next steps. Warm Regards, Aneesa.
$25 USD in 40 days
6.8
6.8

Hello! I'm a seasoned data engineer with 9+ years of experience in Microsoft Fabric, PySpark, and large-scale SQL migration work. Your project is a solid fit: I can move the heavy processing from stored procedures into Fabric Lakehouse while keeping appropriate workloads in the Warehouse. Here's how I can help: - Build the Fabric Lakehouse and OneLake architecture with a clear medallion structure - Develop PySpark transformations on Delta tables for quality, validation, and business rules - Convert suitable stored-procedure logic to Spark, and keep SQL/Fabric Warehouse where it belongs - Build or modify Fabric and ADF pipelines for incremental and batch processing - Optimise Spark jobs for very large datasets, with partitioning and file-sizing tuned properly - Integrate curated data back into Fabric Warehouse/SQL for downstream use - Work hands-on alongside your backend developers, following your standards Two quick questions: roughly how large are the datasets and how many stored procedures are in scope? And is your Fabric capacity already provisioned, or should the plan include sizing and setup? Once I know, I can outline a phased migration plan and confirm timeline.
$38 USD in 40 days
6.6
6.6

I have extensive experience in modernizing high-volume data processing platforms using Microsoft Fabric, PySpark, Delta Lake, and SQL. By transitioning to Fabric Lakehouse + Delta + Spark/PySpark architecture, we aim to enhance performance, scalability, and maintainability. I will architect a robust Fabric Lakehouse/OneLake setup, develop efficient Spark/PySpark transformations, leverage Delta tables for optimized storage and processing, and integrate curated data with Fabric Warehouse/SQL seamlessly. Implementing incremental/batch processing and optimizing large dataset operations are crucial for sustained high performance. Converting SQL stored-procedure workloads to Spark aligns with my expertise, ensuring a smooth transition and maximizing the capabilities of distributed computing. Collaboration with your backend team is essential for seamless integration and knowledge transfer. Beyond meeting technical requirements, I focus on long-term partnership and proactive support, balancing existing SQL/Fabric Warehouse workloads with the new architecture for an efficient data processing environment. I am committed to reliability, responsiveness, and timely delivery. Please provide insights into scalability projections and future data volume expectations to design a solution that scales seamlessly as your business grows. I am excited about contributing to this project's success and fostering a collaborative partnership with your team.
$45 USD in 5 days
6.6
6.6

Hello!, This is James from Hollywood... I’d approach your Microsoft Fabric Lakehouse integration the way a real data project should be handled: first map the current Oracle/MySQL sources and confirm the exact ingestion pattern, then build the PySpark pipeline in Fabric with clean transformations, then validate schema, freshness, and reconciliation so the lakehouse is actually reliable for reporting and downstream use. A lot of these projects fail because people rush into code without locking down table structures, incremental logic, and error handling. I won’t do that. I focus on making the pipeline stable, easy to maintain, and clear enough that you can extend it later without rework. Relevant work I’ve done includes: - a Fabric-based sales analytics pipeline for a retail dashboard - a PySpark ETL flow for syncing ERP and CRM data - a SQL-driven lakehouse ingestion setup for an ops reporting team - a data validation workflow for multi-source financial reporting A few quick questions so I can scope this properly: 1) Are you pulling from Oracle, MySQL, or both, and is the goal full load or incremental sync? 2) Do you already have the Fabric workspace, Lakehouse, and permissions ready? 3) What should the final output be: curated tables only, or also notebooks, parameterized jobs, and documentation? If you want, I can help you turn this around cleanly and quickly without the usual ETL mess.
$50 USD in 7 days
6.5
6.5

Modernize your high-volume SQL-based data platform by migrating heavy processing to Microsoft Fabric Lakehouse with PySpark and Delta Lake. I will design a scalable OneLake architecture, build incremental and batch processing pipelines using PySpark, and convert suitable stored procedures into optimized Spark jobs. I’ll implement Delta table workflows with proper versioning and schema enforcement, and integrate curated data into Fabric Warehouse or SQL Server as needed. All pipelines will be developed in Fabric Data Factory or ADF, with clear logging and monitoring. I’ll work directly with your backend team to ensure seamless handoffs and maintain data consistency across systems. My approach prioritizes performance, idempotency, and maintainability, especially for large datasets, using partitioning, caching, and adaptive execution. I have delivered complex data workflows and automation systems, with a 4.99/5 rating from 150 client reviews on Freelancer, 144 projects completed, 99.5% on time and 99.8% on budget. I can start immediately and deliver the first milestone within a week.
$50 USD in 10 days
6.5
6.5

Stored procedures using cursors and temp tables need a set-based rewrite before Spark speeds them up. I'd move the slowest procedures first, as PySpark notebooks writing Delta tables in the Lakehouse. Final business rules can stay in Fabric Warehouse, which queries Lakehouse tables without copying them. I spent 4 years on Python pipelines and warehousing at MacKenzie & MacKenzie Consulting. Incremental loads would use a watermark and Delta MERGE, so reruns never duplicate rows. Each converted procedure is checked against the SQL Server output by row counts and checksums. First comes one heavy procedure on PySpark with that check, 5 days from award. Your team sees it working before anything else moves. Which procedure is slowest, and on how many rows?
$35 USD in 40 days
6.6
6.6

Hello, MICROSOFT FABRIC + PYSPARK {{{ I CAN HELP MODERNIZE YOUR HIGH-VOLUME SQL PROCESSING WITH SCALABLE LAKEHOUSE AND DELTA-BASED DATA PIPELINES! }}} I have 12+ years of experience in IT development, including data engineering, SQL optimisation, backend systems, and scalable data processing. I CAN HELP MIGRATE SUITABLE STORED-PROCEDURE LOGIC INTO PYSPARK TRANSFORMATIONS WHILE PRESERVING YOUR EXISTING BUSINESS RULES AND DATA QUALITY CHECKS. My approach includes: - Building Fabric Lakehouse and OneLake data structures. - Developing PySpark transformations and Delta Lake tables. - Converting suitable SQL procedures into efficient Spark jobs. - Implementing incremental and batch processing pipelines. - Optimising large-volume processing and integrating curated data with Fabric Warehouse/SQL. - Collaborating with your backend team on testing, validation, and performance improvements. I will focus on hands-on implementation, reliable data quality, scalable processing, and maintainable pipelines that fit your existing platform. Lets Connect and discuss project in detail with an opportunity to start work together... Thanks, Christina
$25 USD in 40 days
6.3
6.3

Hi, I'm Kausar. I see you're looking to modernize your SQL-based data processing platform using Microsoft Fabric and PySpark. My experience with data integration and transformation aligns well with your needs for building a Fabric Lakehouse architecture and developing Spark transformations. To start, I would analyze your current SQL stored procedures to identify which components can be efficiently converted to Spark. Next, I would set up the Fabric Lakehouse and Delta tables while ensuring smooth integration with your existing SQL Warehouse. Finally, I would implement batch processing strategies to optimize the handling of large datasets. Please start the chat and share your vision so I can suggest the right next step and align the work with what you want. Best regards, Kausar
$35 USD in 3 days
6.0
6.0

I will execute your Microsoft Fabric Lakehouse integration using PySpark notebooks and Delta Lake tables to deliver automated medallion data pipelines. I configure schema evolution handling, automated V-Order optimizations, and Data Factory orchestration to prevent Delta transaction write-conflicts, small-file performance degradation, and pipeline failures. What are your primary upstream source systems, and will ingestion be scheduled batch or real-time streaming? Best Regards, Waqas
$35 USD in 40 days
6.3
6.3

Hi, Your goal is to move heavy SQL Server processing into Microsoft Fabric Lakehouse, Delta tables, and Spark/PySpark while keeping the right workloads in Fabric Warehouse. I can help build the data flow, convert suitable stored procedures into SQL and PySpark transformations, and set up reliable batch or incremental processing for large datasets. I will also work with your backend team to validate results, improve performance, and connect curated data to the warehouse through ADF/Fabric Data Factory pipelines. MySQL, Data Processing, SQL, and Microsoft experience will help me understand the existing logic before changing it. Which stored procedures and datasets should be migrated first, and what are the current volume, refresh, and processing-time targets? Please start the chat and let’s discuss your current stored procedures, data volumes, and migration priorities.
$25 USD in 20 days
6.1
6.1

Hello, I am senior data engineer with number 7 plus years of experience in Microsoft Fabric and lakehouse and also have worked with more than 132+ here.I will work based on requirement . Let’s connect
$25 USD in 40 days
5.8
5.8

Hello I checked your project, and the main point is moving heavy SQL Server stored-procedure processing into a Fabric Lakehouse while keeping the right workloads in Fabric Warehouse. I have hands-on experience with data processing, SQL, PySpark, Delta Lake patterns, and pipeline-based integrations, including large datasets that require incremental processing and careful validation. I would build the OneLake and Lakehouse flow around bronze, silver, and curated Delta tables, then convert suitable business rules into PySpark transformations. Fabric Data Factory pipelines can manage batch loads, checkpoints, and dependencies, while optimized Spark reads and partitioning would help control processing costs and performance. The curated data can then be exposed through Fabric Warehouse or SQL for your backend team, with data quality checks and clear operational logging included from the start. Could we discuss which stored procedures are the best first candidates and how your current SQL Server tables handle incremental changes? That will help me shape the migration without disturbing existing backend workflows. Best regards
$35 USD in 40 days
5.4
5.4

Hi there, I have reviewed your brief and understand that you need hands-on data engineering support to modernize a high-volume SQL Server processing platform using Microsoft Fabric, Lakehouse, Delta tables, and PySpark while retaining SQL-based workloads where appropriate. I will assess the existing stored procedures and processing logic, identify suitable candidates for Spark-based transformation, and implement the migration in manageable stages. I will develop PySpark transformations, organize raw and curated data in the Lakehouse, implement incremental and batch processing, and configure Fabric Data Factory or ADF pipelines as required. I will also optimize partitioning, joins, and data processing patterns for large datasets while integrating validated outputs with Fabric Warehouse and existing backend services. Throughout development, I will validate results against the current SQL implementation, test data quality and business rules, and document performance improvements and any remaining SQL dependencies. Could you share the approximate data volume, current processing duration, and whether your Fabric workspace and Lakehouse are already configured? I will use these details to define a practical implementation plan, realistic timeline, and cost estimate. Regards, Imran A.
$25 USD in 40 days
5.5
5.5

I can help modernize your high-volume SQL processing platform with a practical Microsoft Fabric and PySpark implementation. My approach would combine a well-structured OneLake/Lakehouse architecture with Delta tables, incremental processing, and Spark-based transformations for data quality, validation, and business rules. I can assess existing stored procedures, identify workloads best suited for PySpark, and preserve SQL Server/Fabric Warehouse where it remains the right fit. I’m also comfortable developing and modifying ADF/Fabric Data Factory pipelines, optimizing batch processing across large datasets, and integrating curated Lakehouse outputs with Fabric Warehouse and SQL consumers. The work would be hands-on and coordinated closely with your backend developers, with attention to maintainability, observability, and reliable production execution. I would begin by mapping the current processing flow, data volumes, dependencies, and performance bottlenecks, then establish a focused migration plan and implement the highest-value components first. I look forward to contributing to a robust, scalable Fabric data platform. Best regards
$25 USD in 24 days
4.6
4.6

**Free to start on your Fabric Lakehouse migration right away and ready to work hands on with your backend team** I am Anand I build Fabric Lakehouse and OneLake architectures with PySpark transformations on Delta tables and convert SQL stored procedure workloads to Spark where it fits best I also build and modify Fabric and ADF pipelines for incremental and batch processing tune Spark for large datasets and load curated data into Fabric Warehouse and SQL Recent work: Moved a SQL Server workload to Fabric Lakehouse with Delta incremental loads and built PySpark cleansing pipelines for millions of daily records Can we do a quick call today to pick your first stored procedures for conversion?
$50 USD in 7 days
4.8
4.8

Little Elm, United States
Member since Oct 19, 2026
£250-750 GBP
$30-250 USD
$1500-3000 USD
₹750-1250 INR / hour
$15-18 USD / hour
₹12500-37500 INR
₹12500-37500 INR
₹750-1250 INR / hour
$25-50 USD / hour
$15-25 USD / hour
₹750-1250 INR / hour
$15-25 USD / hour
$10-30 USD
$10000-20000 USD
₹750-1250 INR / hour
₹12500-37500 INR
$25-50 USD / hour
₹1500-12500 INR
£10-15 GBP / hour
$250-750 AUD