Improvement of Existing Project (Ruby, Golang, JS, PostgreSQL, ClickHouse)

We have some improvement plan for our old website/metrics collector (written in Ruby for the metrics collector, Go backend, standard JS-jQuery for frontend, PostgreSQL for databases)

The existing tables:

- users

- groups

- stations

- station_logs_x --> metrics per station (splitted in the past from station_logs table)

- station_hours_x --> metrics per station (splitted in the past from station_minutes table)

- station_minutes_x --> metrics per station (splitted in the past from station_hours table)

attachments are existing code examples:

- [login to view URL] -> model

- [login to view URL] -> old collector

- [login to view URL] -> controller

- *.html -> view

The plan was, adding a new type of station but with same/similar metrics, rewrite tcpserver using golang, and migrate station_* table to ClickHouse.

priorities for improvements are:

1. Create a UI for newly MQTT inputs

database columns still the same, the MQTT server already implemented in [login to view URL] (it still inserts to stations, station_logs_x, station_hours_x, station_minutes_x.

see [login to view URL]


1.1 add a flag on stations that mark that this station have different UI than others, if flag present, show this UI, else use old graph UI, the metadata can use something like this (stored as one kv in data column in stations table): [





1.2 implement new UI

2. Fix the bug in previous [login to view URL] changes [new imei received doesn't automatically create logs_x/hours_x/minutes_x tables], by rewriting into Go


2.1 rewrite [login to view URL] to [login to view URL], add the creating a new table when station never exists before

2.2 write a unit test (connect, disconnect, receive, response) to make sure it works properly without memory leaks (not holding disconnected socket)

3. Migrate some part of database to ClickHouse (see previous code [login to view URL])

change all queries related to logs/minutes/hours from PostgreSQL to ClickHouse

for the hours and minutes, no need to create trigger anymore, ClickHouse materialized view is automatically updated from source table.

the purpose of this action is that ClickHouse can save space (88 million records = 600MB in clickhouse, vs 19GB in normal RDBMS), have really fast insert performance (600k recs/s vs normal rdbms that only 1500 recs/s) so it would be faster to backup, also it should be faster for retrieval


3.1 write a code to initialize ClickHouse tables when service is up and when new station inserted, this also solves problem 2.1

3.2 add a code to double write to ClickHouse (not just to PostgreSQL), with 2 feature flag/toggle: 1 for disabling writing to the PostgreSQL, 1 for enabling writing to ClickHouse. Replace the old code that writes/queries to stations, station_hours, station_minus with ClickHouse version.

3.3 write a code to migrate data from PostgreSQL to ClickHouse (called after 3,1, make it resumable, if you want to upsert use ReplacingMergeTree, but if not, use other table engine)

3.4 write a script to backup ClickHouse and Restore, test the toggle feature flag by disabling PostgreSQL or enabling ClickHouse

If you are interested and want to do this job, please reply with estimated number of hours you think you will need to solve each subtask you can and the bid for each subtasks/milestone

1.1 db flag for ui: xxx hours, xxx $

1.2 implement new graph: xxx hours, xxx $

2.1 rewrite [login to view URL]: xxx hours, xxx $

2.2 unit test tcpserver: xxx hours, xxx $

3.1 table initializer: xxx hours, xxx $

3.2 toggle and double write: xxx hours, xxx $

3.3 migrator: xxx hours, xxx $

3.4 backup/restore script: xxx hours, xxx $

total: xxx hours, xxx $

note, you must bid/send proposal with formats above (replace the xxx with your estimation)

there's a previous project manager/developer that could guide you to learn the previous source code.

Beceriler: Golang, PostgreSQL, JavaScript, Ruby, Linux

Daha fazlasını gör: integrating wordpress existing project, dunamic web site existing project vbnet, flex php existing project, rails upgrade existing project version 123, aspnet adding silverlight video existing project, upgrade rails existing project, microsoft access existing project, crystal reports add existing project vs05, existing project best game puzzle, converting existing project maven build, change rails version existing project, integrate substruct existing project ror, updating rails existing project, add vue js to an existing project, swagger node js existing project, add next js to existing project

İşveren Hakkında:
( 20 değerlendirme ) Surabaya, Indonesia

Proje NO: #29894701

Bu iş için 6 freelancer ortalamada $583 teklif veriyor


Hi there, ★★★ Ruby on Rails (ROR) Expert ★★★ 10+ Years of Experience ★★★ I've read requirements and ready to work on your project. Here's estimate that you require: 1.1 db flag for ui: 11 hours, 110 $ 1.2 implement n Daha Fazla

$750 USD in 7 gün içinde
(66 Değerlendirme)

Hello, Warm Greetings!!     I read your job description completely and I found my skills relevant to your project. I am a professional web developer with 5 years of programming experience for a multitude of different t Daha Fazla

$500 USD in 7 gün içinde
(19 Değerlendirme)

Hi there! I have read your project of improvement of existing project. I am a top freelancer and I will do it in a professional way! WAITING FOR YOU IN CHAT BOX! Wajid

$500 USD in 7 gün içinde
(21 Değerlendirme)

Hi. I am senior web designer with 6+ years experience. I am very interested in this project(Improvement of Existing Project (Ruby, Golang, JS, PostgreSQL, ClickHouse)) and want to discuss with you about the project det Daha Fazla

$550 USD in 5 gün içinde
(3 Değerlendirme)

Hi there I just noticed the project you posted (Improvement of Existing Project (Ruby, Golang, JS, PostgreSQL, ClickHouse)) and very interested in job details. I can give you the perfect result with my skills and dedi Daha Fazla

$500 USD in 7 gün içinde
(7 Değerlendirme)

Hello, Thank you for your care. I've just checked your job description carefully. I am a Full-Stack developer of PHP with 7+ years experiences with web developing. Javascript and PHP are my major and all Javascript, PH Daha Fazla

$700 USD in 7 gün içinde
(2 Değerlendirme)