Tamamlanmış

Build a python module that can stream data and extract it to then store it in a MySQL

I want to have a python project which can consume a stream of data from a KAFKA topic.

For example, when you send this command:

bin/[login to view URL] --bootstrap-server localhost:9092 --topic myTestTopic --from-beginning

It will send an output like this: (there will be 1000's of these lines.... just taking top 2 for this example)

&time=1561768216000&gameCategory=PINPOINT&category=ONE&uniqueId=2518Z-0892A-0030O-16H70&transactionType=CRD&familyId=000-222-115-11119&realTs=1561768319000&sortId=1&msg=SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes.&remoteIpAddress=127.0.0.1&userAgent=HTTP&

&uniqueId=872541806296826880&time=1571988786000&gameCategory=NOTIFY&category=TWO&transactionType=CRD&familyId=401-222-115-89387&sortId=1&realTs=1571988989000&msg=This-is+a+reminder.&remoteIpAddress=127.0.0.1&userAgent=HTTPS&

I want to consume the following from the output:

-realTs

-familyId

-msg

-uniqueId

and you can see that each element is seperated by an ampersand ('&'). They are not always in same index/place so I'm not sure if I need a regex? These will be processed and batch inserted. I want this batch to be of size 100 let's say which can be configurable from a properties file.

Eventually when I do the query on a local running MySQL, i'd see this:

SELECT * FROM testTable;

+---------------+-------------------+-----------------------------------------------------------+-------------------------+

| realTs | familyId | msg | uniqueId |

+---------------+-------------------+-----------------------------------------------------------+-------------------------+

| 1561768319000 | 000-222-115-11119 | SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes. | 2518Z-0892A-0030O-16H70 |

| 1571988989000 | 401-222-115-89387 | This-is+a+reminder. | 872541806296826880 |

+---------------+-------------------+-----------------------------------------------------------+-------------------------+

It will also extract value of "time" element of the last line in a batch, and store this in another MySQL table called STATE.

This table will only have 1 column:

SELECT * FROM state;

+---------------+

| time |

+---------------+

| 1571988786000 |

+---------------+

Once this is done, I will have another python module in same project that will connect to the MySQL database and extract data from MySQL table. It will take an argument and it will start extracting all rows from a given (realTs) and print them to console.

So, in total, 2 modules. I want to keep code clean and keep passwords and other configs in properties file.

1) One which streams from KAFKA topic and extracts and processes it to store in MySQL

2) One which then connects to this local MySQL and starting from a given realTs then print whole row to console.

Beceriler: Python, Linux, MySQL

Daha fazlasını gör: python etl pipeline example, store tweets in mysql database, save api data to database python, twitter database, twitter streaming api python, data pipeline python, building data pipelines with python pdf, python data pipeline library, yell data extract, upload data csv file mysql, web data extract php, script extract data html perl mysql, web data extract python, extract data html tables mysql database, python parse email store mysql, script extract data inbound email store mysql database, extract data messages store database, extract data website store database, php api store mysql data, data entry online html need store mysql database

İşveren Hakkında:
( 1 değerlendirme ) Hammersmith, United Kingdom

Proje NO: #22887347

Seçilen:

polarjin2017

Hello? How are you? I am excited to work with you on this project. I have done a lot of jobs with python like Django admin, Flask, python scrap, pysql, python tkinter GUI etc Here is on of my scrap with python wor Daha Fazla

%selectedBids___i_period_sub_7% gün içinde 150%project_currencyDetails_sign_sub_9% %project_currencyDetails_code_sub_10%
(75 Değerlendirme)
6.5

Bu iş için 9 freelancer ortalamada £154 teklif veriyor

mtdevil369

Hi, I have understand the project requirement & interested to work for this project. I will start right now and I will do it within 12 hours. Please send a message so we can discuss more & start the project. Thanks.

1 gün içinde %bids___i_sum_sub_32%%project_currencyDetails_sign_sub_33% GBP
(14 Değerlendirme)
4.2
Alexsfodik

Hi. Dear. Your job posting has caught my attention. I have many experiences in python, tkinter, django. I can complete your project successfully. Thanks.

£150 GBP in 3 gün içinde
(2 Değerlendirme)
3.4
Misha100top

Hi, Mr Joey. Hope you are going well! I checked your project carefully. I have rich experience with python [login to view URL] is one of my top skill. If you give me all data for the project, I will start working immediatel Daha Fazla

£125 GBP in 3 gün içinde
(2 Değerlendirme)
2.5
DavidSeabolt

Hello, I'm David, I can provide you perfect work as you looking for . we can discuss more in chat . I will be waiting for your response . I have very great experience about your project , if you want discuss more plea Daha Fazla

£135 GBP in 7 gün içinde
(0 Değerlendirme)
0.0
tabishadnan

Hi there It seems like an interesting project.I would like to discuss all the requirement with you,so I can make the scope of this [login to view URL] also note that final estimation will be revealed from your end. Hence, Daha Fazla

£222 GBP in 5 gün içinde
(0 Değerlendirme)
0.0
Kayteakane

I have a lot of experience using python, I have a degree in Computer and electronic engineering from the university of Cape Town and I am currently doing a data science course. I have worked with various frame works on Daha Fazla

£222 GBP in 5 gün içinde
(0 Değerlendirme)
0.0
£127 GBP in 3 gün içinde
(0 Değerlendirme)
0.0
chomsengdev

Nice project. As you can see from my profile,I have full experience on blockchain, smart contract and bitcoin . So I think it is very suitable for me. If you give me a chance, I will do my best for you. I want to dis Daha Fazla

£135 GBP in 7 gün içinde
(0 Değerlendirme)
0.0