Data Engineer Needed to Organise & Clean 4.8M Records Across 230 Excel Files

@gagansahay695

Hi, I'd be glad to help organise and clean 4.8M records across 230 Excel files to create a structured and standardised dataset for further processing and analysis. I will utilise Python, Pandas, and data processing techniques to audit the existing files, identify inconsistencies, and provide an initial data-quality report. I will then consolidate the data into a master dataset, standardise fields such as company name, email, and telephone, and perform deduplication using appropriate matching logic. I will also create clear classifications for data quality and deliver the final structured output in a format suitable for database import, Python processing, and future enrichment. I can cover: • Clean and standardised master dataset • Duplicate/removed records dataset • Data quality classification report • Final structured output in a recommended format such as CSV, Parquet, or SQLite • Record counts and summary of changes at every stage I work with clear daily progress updates and honest scoping. Please share your expected timeline and any specific requirements for the final output format, and I can get started right away. Best regards, Gagan

@mobasherreza10

Hi there, Throughout my career, I have worked on projects involving large datasets, metrics consolidation, aligning different data sources, and creating database-ready structures, skillset truly essential in your project. Handling millions of records is not just a preference, but an innate attribute of mine that ensures meticulous work throughout every step of the process. Moreover, my expertise in Python and related libraries like Pandas has enabled me to develop efficient software solutions for data cleaning, normalisation and deduplication over the years. Precisely the skills you're looking for to tackle the incongruencies in your valuable dataset. Beyond just merging files, I'll extensively audit all the 230 Excel files to track inconsistencies, map equivalent columns into standard fields, and provide you with an initial audit that spotlights each and every pertinent data-quality issue I identify. Best regards, Mobasher Reza

@robertb297

One thing worth flagging before anything else: Excel caps out around 1.048 million rows per sheet, so 4.8M records physically cannot land in a single xlsx regardless of how it is structured. I would land the master dataset in Parquet or SQLite/Postgres and export Excel/CSV slices from that on request, whichever you prefer. Process: audit all 230 files first, columns, dtypes, row counts, naming variants, before touching anything, so the standardization map is built from what is actually there. Then process in chunks with Pandas/Polars rather than one in-memory frame, since 4.8M rows across inconsistent schemas will blow up memory if loaded naively. For dedup, matching all 4.8M rows against each other is not computationally realistic. I would block first, by normalized email domain or phone, then fuzzy-match company name and address only within each block. Catches real duplicates exact matching misses, without the runtime blowing up. Every stage logs a count: raw in, cleaned, deduplicated, flagged invalid, so nothing disappears silently. You get the clean set, duplicates set, invalid set and a schema mapping doc separately. Are the 230 files roughly the same structure or genuinely different formats? That changes how long the audit phase takes.

@pvsysgroupinc

★★★ TOP 1% IN FREE LANCER WORLD ★★★ ★★★ 20+ Year Experience in IBD being CMD★★★ ★★★ 200+ Country Satisfied Clientele ★★★ ★Linkedin★ ★Data Entry★ ★Business Plans★★★ ★★★ Operational Strategic planner Customer Support 24*7★★★ ★★★Excel/Word Operation★★★ ★★★Chat Support★★★ ★★★Calling Support★★★ ★★★Business Plans / Marketing Strategy ★★★ * Digital Marketing★★★ ★★★Social Media Marketing ★★★ ★★★Internet Marketing ★★★ ★★★Any type of Data Projects★★★ ★★★★★★ Regards, ★★★CMD★★★ ★★★PVSYS GROUP (INDIA)★★★ ★★★IF YOU THINK THEN I CAN★★★

@bakhtawarishfaq9

Hello, I can consolidate, clean, standardize, validate, and deduplicate your 4.8M records across 230 Excel files using Python, Pandas/Polars, and database-ready workflows. I will maintain complete record counts, source tracking, backups, duplicate/problematic datasets, processing logs, and reusable scripts to prevent data loss. I can also provide the required data-quality report, schema/column mapping, and documentation. I’m ready to review your sample files and propose the most efficient approach, timeline, and milestones. Regards, Bakhtawar

@marvind18

4.8M DATA You need a repeatable data pipeline that preserves every source record and makes each merge, correction, and duplicate decision traceable—not a one-off Excel consolidation. I have 9+ years working with Python services, PostgreSQL, data pipelines, indexing, and production data workflows. At Marin Software, I built real-time Python/AWS pipelines; at Poundit, I worked with operational data and automation where integrity and reproducibility matter. I would use Python with Polars for fast, memory-efficient processing, plus PostgreSQL or Parquet for the master output rather than one oversized Excel file. First, I would profile schemas and create a column-mapping report. The pipeline would retain source filename and row IDs, write immutable raw copies, normalize fields into a defined schema, and produce row counts plus change logs at every stage. Deduplication would use tiered rules: exact identifiers first, then scored candidates based on email, normalized phone, domain, company, address, and city. Ambiguous matches remain flagged, never silently deleted. Outputs will include clean master data, duplicate groups, invalid/problem records, quality classifications, reusable code, schema documentation, and rerun instructions. Can you provide a representative sample covering the different source formats and countries?

@AZTURK1

4.8M DATA Consolidating 4.8 million records across 230 unaligned Excel sheets requires an automated, memory-efficient ingestion pipeline with zero in-memory truncation or silent dropouts. ✅ Ingestion and schema mapping using Python (Polars and DuckDB) to process 4.8M rows without RAM bottlenecks, logging file source per record ✅ Standardization across all fields: E.164 phone formats, normalized URLs/domains, RFC email checks, and cleaned company/address strings ✅ Multi-tier deduplication engine matching compound keys (email, phone, domain, company + city), segregating clean, duplicate, and invalid data ✅ Audit trail at every transform step (Raw -> Cleaned -> Deduped -> Flagged) outputting to Parquet and indexed SQLite/CSV with full run scripts ✅ 10+ years production experience | 400+ completed projects | 320+ five-star reviews ✅ Microsoft® Certified: MCSA | MCSE | MCT & cPanel Certified (CWSA-2) Are the 230 files strictly single-sheet .xlsx/.xls, or do some workbooks contain multiple tabs that need parsing? I can deliver this in 1 day(s) for 5100 INR and can jump on it right now.

@muzafar042

4.8M DATA Hello, I’m an experienced data engineer specializing in large-scale Excel/CSV processing, Python, Polars/Pandas, cleaning, normalization, and intelligent deduplication. I can audit all 230 files, create a standardized schema, consolidate the records, preserve source tracking, and produce clean, duplicate, invalid, and problematic datasets without accidental data loss. I’ll use a staged pipeline with backups, record counts, validation logs, and reusable Python code; for 4.8M records I’d recommend CSV/Parquet or a database rather than Excel as the master format. I can provide a clear data-quality report, schema mapping, processing documentation, and future reprocessing workflow. I’m comfortable with milestone-based delivery and can provide pricing and timeline after reviewing a sample. Regards, Muzafar

@alfatrade2

230 Excel files with different column layouts means the mapping step decides everything, not the cleaning. Deduplication on 4.8M rows is also risky without a agreed key: name plus phone match is not the same as an exact-row match. Which fields define a duplicate for you, and what should the final output be, Postgres or clean CSV? I've built Python pipelines for data processing and FastAPI services.

@venkatsreekanth

4.8M DATA Hi, I can consolidate and clean all 230 Excel files using Python + Pandas/Polars, handling schema mapping, standardisation, validation, and multi-field deduplication. I’ll preserve source records, maintain before/after counts, and provide: - Clean master dataset - Duplicate & problematic records - Data-quality report - Reusable Python scripts - Processing logs & documentation I’ll first audit a sample and recommend the most suitable CSV/Parquet/database structure. I can work milestone-by-milestone with data integrity as the priority.

@Ranataimur

How consistent are the key identifiers across the files, especially email, phone, website and company name? I’d audit a sample first, then build a reusable Python/Polars pipeline that tracks every record from raw to cleaned to deduplicated, with counts and logs at each stage. You’ll also get separate duplicates/problem records, schema documentation and reusable processing code. I can start with the sample files and give you a realistic timeline after the audit. Regards Rana.R

@clearlense2026

4.8M DATA Hello, I can consolidate, clean, standardize and deduplicate your 230 Excel files using a **reusable Python ETL pipeline** with strong data-integrity controls. My approach: • Audit all files and create an initial **data-quality and schema report**. • Use **Python, Pandas/Polars** with efficient batch processing for 4.8M+ records. • Map different column names/structures into a standardized master schema while preserving source tracking. • Normalize company names, names, emails, phones, URLs, addresses, postal codes, countries and other fields. • Deduplicate using reliable combinations such as **email, phone, domain, company + address/city**. • Keep clean, duplicate and problematic records separately—**no silent data deletion**. • Track counts and changes through **Original → Cleaned → Deduplicated → Validated** stages. For this volume, I recommend **CSV/Parquet or SQLite/PostgreSQL** rather than Excel as the final storage format. Deliverables: cleaned master dataset, duplicate dataset, problematic records, quality report, schema/column mapping, reusable Python scripts, processing logs and future-run instructions. I have experience with **Python automation, Excel/CSV processing, data cleaning, transformation and structured datasets**. I focus on accuracy, validation and reproducibility. I can begin with your sample files, complete the audit first, and then process the full dataset through clear milestones.

@gabriel211084

4.8M DATA Hello, 18 years in IT: ETL admin for a telecom data warehouse (Millicom/Tigo) and DBA (Oracle, SQL Server, PostgreSQL). Data integrity first is how I work. Approach (Python + Polars + DuckDB/SQL): 1. Load all 230 files untouched into staging, each row tagged with file, sheet and row number; counts checked vs source. 2. Audit: column profiling, mapping of equivalent columns to one schema, data-quality report for your review before transforming. 3. Standardise names, emails, phones (international format), domains, postcodes, cities, countries, nulls. 4. Dedup on email, phone, domain, company+address, company+city, with fuzzy name matching. Each group gets a cluster ID; extras go to the duplicates dataset, never deleted. 5. Classify: complete, with/without email or website, invalid email/phone, potential duplicate, unusable. No data loss: every stage saves its output and counts (Original > Cleaned > Deduplicated > Validated), and a check confirms master + duplicates + invalid = original total. Deliverables: 3 datasets (CSV + Parquet), SQL-ready schema, quality report, mapping docs, reusable scripts, logs, README to rerun on new files. Timeline: 7 days. Milestone 1 is the audit and mapping so you validate before the full run. Happy to check a sample first. Gabriel

@Lennyfixesall

4.8M records across 230 Excel files is the kind of cleanup where one quiet loss can ruin trust in the whole set. I’d treat the counts and source tracking as the anchor from the start, so you can see what changed at each stage. You’ll get one clean master dataset, a duplicate/removed-records file, database-ready output, and a plain change summary. Do you already have a preferred master schema, or should I infer it from the files? INR 7,000, 1 day.

@RevS412

4.8M DATA. I’ll start with a sample audit to map the different source schemas, identify equivalent fields, define the canonical dataset, and agree on safe normalization and deduplication rules before processing the full collection. I’ll build a reusable Python pipeline using Polars/Pandas, with Parquet or SQLite where appropriate for dependable processing at this scale. Each stage will retain source traceability and generate record counts and validation summaries. Deliverables: - master cleaned dataset; - duplicate and invalid/problematic-record outputs; - data-quality report and column-mapping documentation; - reusable processing scripts, logs, and instructions for future datasets. Duplicate matching will use normalized email, phone, website/domain, company name, and location combinations, while uncertain matches remain flagged for review. I propose a funded sample-audit milestone first, followed by the full processing and validation milestone after you approve the schema and rules. I can complete the workflow within 10 days after receiving the files.

@diyaluo

230 Excel files with inconsistent column structures and roughly 4.8 million records is exactly the kind of consolidation job I handle with pandas and openpyxl. I'd start by profiling each file to map its columns to a master schema, then standardise naming, trim whitespace, deduplicate on your chosen key, and flag missing fields in a separate review sheet. You'll get a clean combined dataset as both Excel and CSV, plus a reconciliation report showing record counts per file before and after cleaning, so you can verify nothing was lost before database import. I can deliver a sample of 5 cleaned files within a day so you can confirm the schema before I run the full set. Happy to leave the final import step documented and repeatable.

@chamanrajragu

Hi! 230 files with different structures is a schema-mapping problem before it's a cleaning problem. How I'd work: - Profile all 230 files first: columns, types, row counts, encodings and date formats, producing an inventory so we can see exactly how many distinct structures exist (usually far fewer than 230) - Agree a target schema with you, then build a mapping from each source structure to it, kept in a config file so it's auditable and re-runnable rather than a one-off manual clean - Standardise: dates to one format, numbers as real numbers, phone and email formats, consistent casing, trimmed whitespace, and country/state values normalised - Deduplicate on the keys you choose, with a report of what was merged so nothing disappears silently - Validate: invalid emails and phones flagged rather than deleted, outliers and impossible values listed for your decision - Output: one clean dataset (Parquet/CSV plus a ready-to-import SQL schema), an exception file of everything rejected, and the pipeline code so you can re-run it when new files arrive Processing in Python with pandas/Polars, in chunks, so 4.8M rows is comfortable. Delivery: in 2 days, the file inventory plus a cleaned sample across the main structures, so you can confirm the target schema before the full run. INR 1,500 for this first phase. Roughly how many distinct column layouts, and what's the main use after import?