Script - threaded/parallel script to analyze json blob - histogram c#, C++, Python, R

Hello I'm interesting your project very well I'm a Good C/C++, Python, Data Processing expert. I understand your req exactly. I m quite well experienced in these jobs. Let's go ahead with me I want to service for you continously. Thanks

Hi. Interesting project. Some time ago I had similar task and managed to implement it by spliting the file into multiple file. Run a multithreading program on then(which will be much faster depending on the machine you have). The problem I see here is not the big file size, it's the html format. Basically each blob needs to be parsed using some html parsing tool and extract only text. This mean that the size of data will be at least twice reduced(because of html tags). Can you give me a simple file example? So that I can do some simple tests on them? Thx, home we will collaborate.

i can write this in python multithreaded, prefer to run on linux.

Hello I have some experience on this problem. As you said the main problem here is the size of data. Otherwise reading a json file and counting words is easy. It's very good that we can be sure about memory. But you should know that size of memory that we need is not exactly equal to the size of data file. One key point is the implementation json read algorithm in that language or library. Sometimes we need 3GB~4GB memory for a 1GB data file. But if we have enough memory we can be sure that the time of processing is linear. A 3GB file consumes 3x seconds than 1GB file and it's very important for big files. Also be sure that the storage of target machine is a SSD not HDD. SSD give you at least 2 times faster file reading. I think there is a better solution dependent to the structure of your json data. The regular solution is: 1. read the file 2. parse json data 3. process data But might be we can remove parse json step. In this case we reduce two times processing of data from start to end to one time. But it dependent to your data structure and your needed results. Regarding threading: Not all threading applications use all of cpu cores. For example if you have 4 cpu cores and the application run 4 threads to process data OS might put these 4 threads to just 1 or 2 cores. The solution is using multi-process instead of multi-thread. It's same in coding and the way of running. But in multi-process we can be sure that with 4 process we will use all 4 cores. Regards Iman

Hi I have immense experience data processing using python and c++.I have over 12 years of experience in this field and I can deliver your project on time and within budget. Drop in a mail we can talk further. My gmail and skype handles are bg32014 Best regards, Bala

Hi! I am experienced Python coder. I can develop a script to read and interpret your files quickly and efficiently. Please contact me for further discussion. Thanks

A proposal has not yet been provided

Hi, billy01 i worked with you before in the lastweek i created a c# programe to seach html files for img tags without alt attribute about the new job i can do another c# programe to parse the json files , create the histigrame then save it to csv will send you the code as usual payment after job completion waiting your answer Regards

Δεν έχει γίνει καμιά πρόταση ακόμα

Hello, I am glad to build the json parser app with golang for you, golang file can build a command line file which can be running on both linux and windows. Golang can use each core of the CPUs to speed up the json parser. Regards, Joe

I am proficient analyst using R, C++(I like using it together with R) and Python. I would like to help you in writing the code(s) that would help you use the analyse the json files in R, with objects such as ff to help in memory usage and easy writing in word. I would, if given the job, use c++ codes to hasten the codes and reduce immensely the computation time while taking care of the RAM usage with use of ff objects.

A proposal has not yet been provided

Sounds like a great project. Multiprocess/multithread for the load, and the analysis is really easy. How do you want the output, another json file, text file, or database? I can do all of them perfectly.

I made a low bid because I need to gain reputation here and I don't want to spend money on doing so. Having that in attention I think I would be a very good fit for the task you asked. I think I am more than qualified to do the task and the low bid is the advantage I can offer you. I hope you are interested, Best regards, André Meneses