Tamamlanmış

Python data processing with numpy or pandas

Introduction:

We are developing an algorithm to react to certain value changes, right now we are using pure Python

for calculation results but would like to start using data analysis libraries such as Numpy and Pandas

for faster compilations. To make multiple tests changing variables we will need to optimize our code/model

with vectorization or with better data preparation in general.

Each data entry in the database is approximately 30 seconds apart from each other.

This means they should look like this (showing relevant values only).

[

{

date: '06/18/2019 20:00:00',

data1: Number,

data2: Number,

...

},

{

date: '06/18/2019 20:00:30',

data1: Number,

data2: Number,

...

},

{

date: '06/18/2019 20:01:01',

data1: Number,

data2: Number,

...

},

...

]

Current state:

At the start of our algorithm we get all the data necessary from the DB in a single array

and iterate through it comparing them with an emulated date (variable, ex. 15 days from now) and adding 30 seconds each loop,

this emulates the calculations like if it was live on that date.

Through each loop we make a series of calculation, making a backtest of 2 week takes in average

8 minutes but would like to reduce that number as much as possible.

Main part:

For the algorithm to run correctly we need to get an array of objects, each object contains the following from each entry:

{

date: (Date object of data gathered),

data1: Number,

data2: Number

}

With this data we join them in periods (variable, example: 10 periods of data) divided in a

defined timeFrame (variable, example: periods of 15 minutes each). In each period we will insert

all data where [login to view URL] is between that timeFrame.

For each period we need to calculate average of data1, data2, and data1 + data2, also getting the highest

value (peak) of each data value in every period resulting on each period generating an object like this:

period1: {

avgData1: avg(arrayOfData1),

avgData2: avg(arrayOfData2),

avgData1+Data2: avg(arrayOfData1+Data2),

peakData1: max(arrayOfData1),

peakData2: max(arrayOfData2)

}

period2:...

...

period10:...

Once we have all the averages and peak values of each period then we proceed to calculate collective

averages of all the periods results. For example sum(period[avgData1] for period in periods) / [login to view URL],

sum(period[avgData2] for period in periods) / [login to view URL], ...

Final result will return an object like this:

{

data1Result: Number,

data2Result: Number,

data1+Data2Result: Number,

data1PeakResult: Number,

data2PeakResult: Number

}

Problem:

Translate this algorithm with Numpy or Pandas and reduce the compilation time for big data analysis.

We've tried putting all data of each period in independent Numpy arrays and calculating averages

then but the results took longer, maybe we are not using Numpy as intended.

Beceriler: Data Analysis, Veri Madenciliği, Veri İşleme, NumPy, Python

Daha fazlasını gör: numpy data analysis, numpy pandas tutorial, python for data analysis, numpy and pandas interview questions, pandas vs numpy, pandas python, pandas dataframe, numpy and pandas for data science, data processing skills, data processing spreadsheet, data processing forum, data processing health care, python workflow data processing, python data processing, python csv data processing, python data processing amazon aws, data processing machine learning python, big data processing python, python numpy pandas, big data processing with apache spark part 1 introduction

İşveren Hakkında:
( 2 değerlendirme ) Monterrey, Mexico

Proje NO: #20019378

Seçilen:

TalAkerele

[login to view URL] I am very happy to bid your project. i'd like to work with you. I read your requirements carefully,i see what you mean. i've been experienced with Data Analysis, Data Mining, Data Processing, NumPy, Python. Daha Fazla

1 gün içinde %selectedBids___i_sum_sub_4%%project_currencyDetails_sign_sub_5% USD
(7 Değerlendirme)
4.4

Bu iş için 16 freelancer ortalamada $165 teklif veriyor

widadsaghir1993

Hi there. Just read your job description carefully and I'm very interested in it. As you can see my profile, I have gained vast experiences in python pandas/numpy. I can reduce time for big data with pandas/numpy. Let Daha Fazla

$300 USD in 7 gün içinde
(96 Değerlendirme)
7.1
sky930320

Dear Sir. Glad to meet you. I'm very interested in your job post. I've full experience in responsive design. Please Hire me. I'll do my best, will make you pleasure with my work. Thank you. Relevant Skills and Experi Daha Fazla

$140 USD in 7 gün içinde
(17 Değerlendirme)
6.3
C3guru

Hi, very nice to meet you ! I've great experience in Python. When you read my profile page, you can find that I'm a python expert. I've 10+ years of experience in Python especially with Django and Flask for web develop Daha Fazla

$500 USD in 2 gün içinde
(17 Değerlendirme)
5.8
Mickelson

Hi Nice to meet you. I have enough experience in python script. Below the libraries are I used in past project. selenium, pandas, matplotlib, lxml, beautifulsoup, scipy, and other useful libraries. I have written Daha Fazla

$100 USD in 3 gün içinde
(42 Değerlendirme)
5.7
susanna2018

Hi, Client!! @$@$@$PLEASE CONTACT ME. I CAN DO IT WONDERFULLY.$@$@$@ ##% Your idea is attracted me to your world.%## I am Python expert. i have various skills. @@PLEASE GIVE ME CHANCE TO PROVE MY SKILLS@@ :-): Daha Fazla

$100 USD in 3 gün içinde
(28 Değerlendirme)
5.3
honeyocs803

Hello. How are you? I have read your post requirement. I am a highly skilled web development expert . I am a professional who has more than +7 years development experience since university. My main skills: ★ Daha Fazla

$155 USD in 3 gün içinde
(5 Değerlendirme)
4.8
RiHCh

Hello, Dear I have read your project description. I have lots of experience of Python, Django, Flask, PostgreSql, Wagtail, HTML5, Javascript. I'm a Senior Python Developer with 5+ years of hands-on expertise with both Daha Fazla

$200 USD in 7 gün içinde
(10 Değerlendirme)
4.5
amalphateam

Hello, sir. Thanks for your job posting! I read your job description carefully and I have confidence to do it. I am a C, C++, C#, QT, Python, JAVA Developer with +6 years experience and your project is very interesting Daha Fazla

$140 USD in 7 gün içinde
(6 Değerlendirme)
4.1
profvipabutaleb

I'm computer engineering TA with 10+ years of experience. Experienced with data structures and algorithms , computation theory , discrete math , database design using python Experienced with python programming numpy, Daha Fazla

$150 USD in 7 gün içinde
(16 Değerlendirme)
3.9
jk901

Dear As I am a senior software developer, have rich experience with various application development using C#, VC, VB.Net, NodeJS, Matlab, java, and python If you are interested with my proposal, please let me know i Daha Fazla

$120 USD in 3 gün içinde
(3 Değerlendirme)
3.6
gurumail10

Hi, i am a Data Scientist working in machine learning from past 3 years. i have done many projects like time series forecasting, anomaly detection and many more. i have readen your problem statement i can do it easilya Daha Fazla

$70 USD in 3 gün içinde
(3 Değerlendirme)
2.1
mukesh30march

hi i read all instruction please share more detail i did 5 similar task i will provide 5 star rating work

$155 USD in 3 gün içinde
(8 Değerlendirme)
3.5
fivestarOne

Hi, sir "SUCCESS = Faster Speed + Loyal Quality + High Credit" This is my slogan on working with client or employer I am an expert who have more 7 year's experience and have a powerful team. and we've ever devel Daha Fazla

$150 USD in 3 gün içinde
(0 Değerlendirme)
0.0
Lightcanon

I am signal processing Teaching Assistant and also a Computer& Electronics engineer . I know Python&Numpy very well and I used Numpy in many processig projects. I will give you the task finished efficiently and quickly Daha Fazla

$150 USD in 2 gün içinde
(0 Değerlendirme)
0.0
Kuttusarath7

i am a beiginner i can try my best but it take more time to type and learning its difficult i do some work in a note pad

$140 USD in 7 gün içinde
(0 Değerlendirme)
0.0