Objective:
Upgrade an existing ETL process to enhance data validation, error handling, and user feedback for structured Excel files uploaded to SharePoint 2019, for subsequent transfer to a database.
The solution could be also implemented via Azure Functions and Sharepoint online if deemed possible.
Existing Setup:
A basic ETL process developed with the Luigi library, lacking advanced validation and user feedback mechanisms. Solution uses parsing code (Pandas) for Excel files data parsing that can be utilized for the upgrade also.
There are 4 different excel files, so 4 different parsers will be provided. The system should manage four unique Excel file types, each in its own SharePoint folder, requiring separate parsers for their distinct structures. An initial selection feature will enable users to choose the file type they're uploading, ensuring the correct parser is used.
Required Upgrades:
Data validation at the time of upload.
User notifications regarding the outcome of the upload/validation process, including detailed error messages if applicable.
Support for simultaneous multiple file uploads with validation for each file
Functional Requirements
Document Preparation and Submission:
Users will continue to submit documents to SharePoint 2019, following the current document hierarchy and organization.
Document Review Process:
Upon detecting new documents, initiate a detailed review process including:
Validation of data presence in predefined fields.
Syntax validation of data (type, length, and optionally, min/max values if defined).
Error Handling and User Feedback:
If discrepancies or errors are detected during review, the import of the affected document will be halted.
An notification detailing the errors, including the specific worksheet and cell location, is shown to the submitting user for corrections.
Database Import:
Upon successful validation, data will be transferred to the database within a transaction, ensuring all-or-nothing import to maintain data consistency.
Specific tables will be designated for each document type, determined during the analysis and planning phase.
SharePoint Integration:
Update the status of the document in SharePoint based on the import outcome, allowing for an overview of import success.
Notify the user (and optionally the IT solution administrator in case of technical errors like SharePoint or database access issues) about the import outcome.
Hello Sir,
I can develop a desktop app will which can upload data from Excel Files to Sharepoint (online or on premise). The data validation rules shall be applied accordingly and failed transactions shall be logged. Let's discuss and proceed
Thank you,
Usman
Hello, I hope you are doing great.
I have good experience with doing similar jobs. You can check my portfolio here:https://www.freelancer.com/u/AITSoft
I can do this job because have great experience with Apache, .NET, ETL and Python. Please start the chat and discuss this. Thanks! Shamshad
As an experienced data engineer with a knack for analytical problem-solving and a passion for crafting creative solutions, I believe I'm perfectly suited for enhancing your ETL process to enable better data validation. With a solid background in managing and manipulating diverse datasets using technologies such as Python, SQL, Spark, Hive, etc., I'm confident in my ability to upgrade your existing Luigi-based system while incorporating advanced validation and user feedback mechanisms required in the project. Furthermore, I'm fluent in Apache, Python and have years of experience working with ETL tasks - refining my skill set to compliment your need.
In addition to technical proficiency, I bring distinct value to the table with my enhanced creativity. By leveraging this quality, I can enhance not only your ETL process but also the SharePoint integration aspect of your project. Whether it's a dynamic notification system or an easily understandable visual representation of import outcomes in SharePoint - I can deliver an aesthetically pleasing UI/UX design within the scope of this project.
But don't just take my word for it- let me demonstrate my talents with your data! Let's have a deeper conversation about how we can elevate data reliability and transform your ETL process into a more robust, efficient and visually appealing operation!
Hello,
We carefully studied the description of your project and we can confirm that we understand your needs and are also interested in your project. Our team has the necessary resources to start your project as soon as possible and complete it in a very short time.
We are 25 years in this business and our technical specialists have strong experience in Python, .NET, Apache, Azure, ETL and other technologies relevant to your project.
Please, review our profile https://www.freelancer.com/u/tangramua where you can find detailed information about our company, our portfolio, and the client's recent reviews.
Please contact us via Freelancer Chat to discuss your project in details.
Best regards,
Dima K.
Tangram Canada Inc.
As a seasoned software developer, project manager, and virtual assistant spanning over 12 years of substantial industry experience, I have what it takes to deliver outstanding results for your "Enhancing ETL Process for Data Validation" project. I have successfully completed more than 300 projects, with a reputation for quality that my clients appreciate. With a strong competence in Python and extensive experience in .NET development, I am well-versed in deploying efficient parsing code, making me suitable for your Excel data parsing upgrade.
Additionally, my knowledge of SharePoint 2019 platform and strong troubleshooting abilities will facilitate a smooth integration process post validation. Importantly, I perform with the understanding that integrity of data is critical which is why I always deliver 'all-or-nothing' imports for data consistency while offering notifications at every stage to users involved about import consequences. Choosing my services will not only guarantee robust enhancements for your ETL process but also stress-free end-to-end document review; A SharePoint update system that outlines import success status alongside detailed user notifications when necessary.
As a premier full-stack development team, we've honed our skills in ETL processes, with a particular emphasis on data validation and error handling - precisely the areas where your project could benefit greatly. I have extensive experience implementing and upgrading ETL processes to ensure seamless data transfer from various sources to databases.
My experience in utilizing Pandas for Excel data parsing aligns perfectly with your existing setup, which means I can improve your Luigi-based solution without any disruption. Whether through SharePoint 2019 or Azure Functions and Sharepoint online, I assure you a highly customized solution that caters to your unique requirements.
Moreover, my understanding of multiple file uploads and the need for separate parsers based on distinct structures equips me to manage your four different excel files effectively. I can not only provide effective validation mechanisms but also create detailed error messages showcasing spreadsheet and cell locations while simultaneously maintaining user-friendly feedback systems that notify users in case of any errors or discrepancies before the import process is commenced.
In summary, my team ਨe and I have the relevant skills, experience, and proven track record to enhance your ETL process for data validation purposes while providing improved user feedback mechanisms
I am excited about the opportunity to enhance your ETL process for data validation. With my expertise in Python, .NET, Apache, and ETL, I can deliver exceptional results by upgrading the existing process to include advanced data validation, error handling, and user feedback mechanisms. Could you provide more details on the specific requirements for simultaneous multiple file uploads and the integration with SharePoint 2019?
Hello, I can start immediately. I have 13+ years of experience and I ensure you I can complete the project within budget and timeline. I'm not just a developer; I'm your dedicated partner. I'm committed to working with you. Count on me for any upgrades or specific requirements you may have along the way.
I have few questions about the requirement. Can we discuss this over chat? Looking forward to hearing from you soon.
Best regards, Sneha
