
Closed
Posted
I keep a collection of text-only records in several CSV files and need a solid validation pass before the next release cycle. The task is to build and run checks that confirm each field adheres to agreed formats, catches missing or duplicate entries, flags unexpected characters or line breaks, and produces a concise report of every issue found. You are free to use Python (pandas, Great Expectations, or similar), Bash + awk, SQL import routines—whatever gets reliable, repeatable results. Everything must run headless on Ubuntu 22.04 and be callable from the command line so it can slip into our existing CI pipeline. Deliverables: • A runnable script or notebook with clear setup instructions • A summary report (CSV or Markdown) listing all validation errors and their locations • Brief documentation showing how to rerun the checks and adjust rules in the future I am only focusing on validation for now; any cleaning or schema changes may follow as a separate phase once the error landscape is clear.
Project ID: 40772594
60 proposals
Remote project
Active 11 hours ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
60 freelancers are bidding on average ₹941 INR/hour for this job

Rules live in a YAML file, not the code, so you can tune formats later; the script exits non-zero on any failure so CI stops cleanly. Embedded line breaks inside quoted fields are the usual trap. We parse with a quoting-aware reader and report row, column and raw file offset, so each hit is easy to find. Day 1-2: draft rule set, run on one file, send the report for a sanity check. Day 7: full run, docs, CI invocation notes. 1. Are the agreed formats written down somewhere, or do we derive them from the data and confirm with you? 2. Duplicates: whole row, or match on specific key columns? Happy to start today if you want to share a sample file.
₹860 INR in 7 days
7.9
7.9

Hello, CSV validation becomes challenging when malformed records, hidden line breaks, encoding issues, or inconsistent field formats silently pass through automated checks and affect downstream systems. With my background in Python development, engineering automation, and data analysis, I can build a reliable, configurable validation utility tailored to your Ubuntu 22.04 environment. My approach: • Develop a Python command-line validator using Pandas and standard CSV parsing tools. • Store field formats, required values, allowed characters, and uniqueness rules in an editable YAML/JSON configuration. • Detect missing fields, duplicate records, invalid formats, unexpected characters, and problematic line breaks. • Handle quoted multiline fields correctly to avoid false positives. • Generate CSV/Markdown reports identifying the exact file, record, field, and validation error. • Return meaningful exit codes for seamless CI integration. • Include automated tests, setup instructions, and documentation for modifying validation rules. The tool will be fully headless, reproducible, and designed to leave your original CSV files unchanged. I recommend validating one representative file first, then applying the confirmed rules across the complete dataset. Could you share a sample CSV and clarify whether duplicate detection should use entire rows or specific key columns?
₹900 INR in 10 days
6.8
6.8

I understand you're looking for a robust validation process for your CSV files, ensuring each field adheres to specific formats and identifying any issues like duplicates or unexpected characters. My approach will involve using Python, particularly Pandas, to develop a script that performs the necessary checks and generates a detailed report on the findings. I'll ensure it's runnable on Ubuntu 22.04 and provide clear setup instructions for integration into your CI pipeline. You'll receive a concise report detailing all validation errors, along with documentation for future adjustments. Do you have specific validation rules in mind, or should I propose a standard set? Also, how many CSV files are we looking to validate? Looking forward to your response. Best, Hussein.
₹750 INR in 3 days
6.5
6.5

Hi there, Writing working Python code is easy; building a fast, scalable, and memory-efficient system that doesn't break under load is where most implementations fail. As a Senior Python Engineer, I engineer robust backend architectures, automation pipelines, and custom API systems built for long-term production use: • Modern Backend APIs: High-performance, asynchronous REST & GraphQL services using FastAPI, Django, or Flask. • Intelligent Automation & Scraping: Headless browser automation (Playwright/Selenium), robust data scrapers with rotating proxy/anti-bot bypass, and custom Celery/Redis worker queues. • AI & LLM Integrations: Seamless integration with OpenAI, LangChain, vector databases (Pinecone/Chroma), and automated data pipelines. • Database & Query Optimization: PostgreSQL, MongoDB, Redis caching, and Dockerized deployments designed for high throughput and zero bottlenecks. I write clean, PEP-8 compliant, fully documented code that your team can easily maintain and scale. Do you have a technical spec, repo, or API documentation ready? Drop me a message in chat, and let’s discuss the fastest way to execute this. Best regards, Deva | Senior Python & Backend Engineer
₹1,000 INR in 40 days
6.7
6.7

I can help you build a reliable, headless validation pipeline for your CSV records on Ubuntu 22.04 that fits directly into your CI pipeline. Using Python and pandas, I will write a script to check field formats, catch missing or duplicate entries, and flag unexpected characters or line breaks. The deliverables will include the runnable script with setup instructions, a clear summary report in Markdown or CSV listing all errors and their exact locations, and brief documentation on how to rerun the checks and adjust your rules later. My approach focuses strictly on validation without altering your source data, giving you a clear error landscape before your next release cycle. When you are ready, let us discuss how you would like to structure the specific field formats and validation rules.
₹750 INR in 7 days
6.1
6.1

I am very interested in applying for your job since it seems to fit very will with my experience and skills. Regards SamirBanna
₹2,000 INR in 40 days
5.7
5.7

Your CSV validation will fail silently in CI if the script doesn't handle encoding edge cases and malformed line breaks that pandas.read_csv skips by default. This creates false negatives where corrupt records pass validation. Quick questions - what's the expected row count per file so I can recommend chunked processing vs in-memory validation? And are you enforcing referential integrity across multiple CSVs or validating each file independently? Here's the architectural approach: - PYTHON + PANDAS: Build a modular validator that reads CSV with explicit encoding detection, applies field-level regex rules, checks for duplicates using hash-based deduplication, and outputs structured error logs with line numbers. - GREAT EXPECTATIONS: Configure expectation suites for data type validation, null checks, and range constraints, then generate JSON reports that your CI can parse for pass/fail thresholds. - BASH INTEGRATION: Wrap the Python validator in a shell script that accepts file paths as arguments, exits with non-zero codes on validation failure, and pipes summary stats to stdout for CI logging. I've built similar data quality pipelines for 2 fintech clients processing 10M+ transaction records daily. Let's schedule a 15-minute call to review your field rules and CI requirements before I draft the validation spec.
₹900 INR in 30 days
6.1
6.1

The main challenge is making the checks strict enough to catch malformed records without mistaking valid CSV content—such as quoted fields—for errors. The validator should report issues precisely and leave the source files untouched, so the error landscape is clear before any cleaning or schema changes are considered. I’d build a headless command-line check that reads the CSV files and applies configurable field rules for formats, required values, duplicates, unexpected characters, and line breaks. Each finding would include the file, row, field, and a concise reason, with a CSV or Markdown summary suitable for reviewing and sharing. I’d keep the checks separate from any correction logic and make the rules straightforward to adjust as the agreed formats evolve. For reliable CI use on Ubuntu 22.04, the command would have predictable output and an exit status that distinguishes clean data from validation failures. I’d include clear setup and rerun instructions, document how to update the rules, and check edge cases such as empty fields and quoted CSV values so the report points to useful locations rather than producing avoidable noise. The deliverable would be a runnable script and its documentation, with no cleaning or schema changes folded into this phase. Are the agreed field formats and duplicate-key rules already documented, and if so, in what form?
₹750 INR in 4 days
4.9
4.9

Hi, I see that you're dealing with a collection of text-only records in CSV files and need a solid validation pass, I'd approach this by using Python with pandas to confirm each field adheres to agreed formats. I'll set up checks to catch missing or duplicate entries, flag unexpected characters or line breaks, and produce a concise report of every issue found. My focus will be on delivering reliable, repeatable results that can be easily integrated into your existing CI pipeline. In my experience, the common mistake here is not accounting for edge cases like unexpected line breaks, I'd start by setting up a robust validation framework using Great Expectations to handle these cases. You can expect a well-structured report that lists all validation errors and their locations, making it easy to identify and fix issues, and I'll provide clear documentation on how to rerun the checks and adjust rules in the future.
₹1,000 INR in 40 days
4.1
4.1

As an experienced and versatile Python developer, I would be the ideal choice for your CSV data validation needs. Over the years, I've developed a deep understanding of data structures and I have an excellent command over core CSV processing libraries in Python like pandas. Additionally, I'm familiar with BHashes and awk scripting languages, SQL import routines that you have mentioned as alternatives. My proficiency in CLI-based tooling on Ubuntu 22.04 ensures seamless CI pipeline integration. In terms of deliverables you desire - a runnable script with clear setup instructions as well as error reporting in a concise file format—allow me to assure you, not only can I provide that, but moreover, I will document the whole process thoroughly so that future customization or maintenance can be effortlessly done.
₹750 INR in 40 days
3.0
3.0

CSV validation passes usually miss the quiet problems: invisible characters (BOMs, non-breaking spaces, trailing whitespace) that make "valid" values fail downstream, dates in mixed formats, and duplicates that differ only by case or spacing. I'd write a reusable Python validation script with your agreed rules per field (type, format, allowed values, required, unique), run it on every file, and output a clear report listing each failing row, the field and the reason, so it can be rerun every release cycle. Can you share the field rules (or a sample file) so I can encode them?
₹750 INR in 10 days
2.3
2.3

I see you need a thorough validation pass for your CSV files to ensure data integrity before your next release cycle. With over 4 years of experience in Python and data processing, I can develop a robust validation script using libraries like pandas to check for format adherence, duplicate entries, and unexpected characters, all while ensuring compatibility with your CI pipeline on Ubuntu 22.04. To start, I would first outline the specific validation rules you need, then create a script that runs headless and generates a comprehensive report of all issues found. This approach will ensure reliability and ease of integration into your existing workflow. I have a strong track record of delivering concise documentation, so you'll have clear instructions for rerunning the validations or adjusting the rules in the future. Looking forward to discussing your specific requirements further!
₹750 INR in 7 days
1.6
1.6

Hello, I will build one Python command-line validator that runs headless on Ubuntu 22.04 and fits straight into your CI. All rules will sit in one YAML file per CSV, so changing a format, required field or allowed character later means editing a line, not code. It will check field formats, missing and duplicate entries, encoding and BOM issues, stray characters and line breaks inside fields, and exit non-zero on any error so CI fails cleanly. The report will list every issue in CSV and Markdown with file, row, column, rule and the value found. You will get the script, setup steps and a short guide to rerun the checks and add rules, giving you a clear map of the errors before any cleaning phase. Amul
₹755 INR in 40 days
1.5
1.5

Hello, I understand you need a reliable CSV validation tool to detect format violations, missing or duplicate entries, unexpected characters, and line breaks on Ubuntu 22.04. The goal is to deliver repeatable, CI-ready validation with a clear error report. Here’s what I can provide: - A Python/Pandas CLI script with configurable validation rules and field-level checks. - A CSV or Markdown report identifying each issue, file, row, and column. - Setup instructions, rerun commands, and documentation for updating rules. I bring over 4+ years of experience in Python, SQL, and backend development, focusing on reliable data processing and maintainable applications. Just to clarify a few things: 1. Can you share sample CSV files and the expected formats or validation rules? 2. Should duplicate checks apply to entire rows, specific columns, or both? Please come to the chat box to discuss more about your project. Best regards Indresh Kushwaha
₹1,000 INR in 40 days
1.6
1.6

Hi, I can build a repeatable Python command-line validator for your CSV files that runs headless on Ubuntu 22.04. It would check agreed field formats, missing values, duplicates and unexpected characters, then report the file, row, column and failed rule. I’ve built a CSV/Excel processing tool. I’d keep this phase strictly focused on validation without changing the source files, and include setup instructions, documented rules and a clear exit status for your CI pipeline. My rate is ₹850/hour with an initial 10-hour weekly limit. I’d target three days for the first agreed batch after reviewing its size and rules. Could you share sample files, total file sizes, and any checks needed between files so I can confirm the estimate?
₹850 INR in 10 days
0.3
0.3

Hello, I am Jeff Smith, a seasoned Software Architect with 12 years of experience in data validation, particularly in CSV files. I have a strong background in Python, SQL, and automation tools that can effectively handle the validation requirements outlined in your project description. I have thoroughly reviewed your project requirements and am confident in providing a professional solution that ensures each field in your CSV files meets the specified formats, identifies and reports any anomalies accurately. I am keen on discussing the project further in chat to understand your specific needs and tailor the solution accordingly. Looking forward to connecting with you to delve deeper into the project details. Regards, Jeff Smith
₹1,000 INR in 40 days
0.0
0.0

Hello, I can build a headless, repeatable CSV validation tool for Ubuntu 22.04 that fits directly into your CI pipeline. I would implement it in Python with pandas and a configuration-driven rules layer, so formats and checks can be changed without rewriting the core logic. The validator will report the exact file, row, column, invalid value, and failed rule for missing fields, duplicates, type or regex mismatches, unexpected characters, embedded line breaks, and structural inconsistencies. It will produce both machine-readable CSV and concise Markdown output, return a meaningful CI exit code, and include setup instructions, example configuration, tests, and rerun documentation. Large files can be processed in chunks to control memory use. I can start immediately and deliver an initial working validation pass within 24 hours after receiving sample files and rules. Do you already have a schema or data dictionary for each CSV, and what are the typical file sizes, delimiter, and encoding? Best regards, Alvaro
₹750 INR in 20 days
0.0
0.0

I can deliver 'CSV Data Validation Maintenance' end to end — clean architecture, tests, and deployment. I'll start with a short plan, share progress daily, and hand over production-ready code. Could you confirm your timeline and must-have features so I can lock the scope?
₹842.79 INR in 7 days
0.0
0.0

Hi, I can build this as a small command-line tool in Python (pandas) that runs headless on Ubuntu 22.04 and drops straight into a CI step. What you would get: - A rule-driven validator: per-column format checks (regex/type/length), missing values, duplicate rows and duplicate keys, unexpected characters, and stray line breaks inside fields - Rules kept in a simple YAML/JSON file, so you can adjust them later without touching the code - A summary report (CSV and Markdown) listing every issue with file, row, column, rule and offending value - A non-zero exit code when errors are found, so CI can fail the build - A short README covering setup, how to rerun the checks, and how to add or change rules I work with Python, SQL (PostgreSQL/MySQL) and Linux daily, and I keep scripts like this repeatable and easy to read. I will not touch or clean your data, only report, as you described. If you prefer, I can also do this as a fixed-price job, which is easier for a clear, small deliverable like this one. To start, could you share a sample CSV (or the list of columns and the agreed formats) and roughly how many files and rows there are? Pingkui Zheng
₹908 INR in 3 days
0.0
0.0

Hi, this is a good fit for me. I'd build a Python (pandas) validation tool that runs headless on Ubuntu 22.04 from the command line: 1) A rules file (YAML) defining each field's format: type, regex, allowed values, required or optional 2) Checks for missing values, duplicates, unexpected characters and embedded line breaks 3) Output as a summary report (CSV and Markdown) listing every issue with file, row, column and reason 4) Exit codes, so it slots straight into your CI pipeline 5) A README with setup, how to run, and how to add or change rules You can adjust rules later without touching code. How many CSV files and roughly how many rows, and do you have a list of the agreed field formats? I estimate 6–8 hours in total, under ₹8,000.
₹1,000 INR in 8 days
0.0
0.0

Tamil Nadu, India
Member since Sep 27, 2023
$30-250 USD
₹12500-37500 INR
₹600-1500 INR
$10-30 USD
$750-1500 CAD
₹1500-12500 INR
₹37500-75000 INR
₹12500-37500 INR
$30-250 USD
₹1500-12500 INR
$2-8 USD / hour
₹1500-12500 INR
₹750-1250 INR / hour
$2-8 USD / hour
€8-30 EUR
₹600-1500 INR
$750-1500 USD
$30-250 USD
₹12500-37500 INR
£20-250 GBP