Performance Issue with Big Dataset

I am working on your site right now... Just to let you know. This turned out to be not simple as I expected.

Hi Sergey,

Happy new year! No problem, thanks for the update, I really appreciate it. Are you working now on AWS or haven't you got there yet? Let me know if you need anything from my side.

Thanks, Matias

matcorrao I think the price of good news worth it. All you did was restructuring the content? Or are you saying that what you did requires a better content estructure in order to work?

Not only, but mostly. I also applied some modifications for DB and SQL queries.

matcorrao On the other hand, do you think that this will work the same way on a VPS? I believed you were testing on AWS, right?

No. Only localy. The things I have discovers had not required test of server capacity. If you apply everything well, then even on low grade server it will be quick. The only increas in power is needed with increas of visits.

I'll create the list for you this week. It stil require some analysis.

Sergey,

see my skype comment. Don't forget to fix the filter count for text fields ...

Hi Sergey!

Not only, but mostly. I also applied some modifications for DB and SQL queries.

Ok, perfect!

No. Only localy. The things I have discovers had not required test of server capacity. If you apply everything well, then even on low grade server it will be quick. The only increas in power is needed with increas of visits.

Excellent!

I'll create the list for you this week. It stil require some analysis.

No problem, thanks a lot!

Best, Matias

Matias,

if you use notifications, make sure they are cleared early. If a user accumulates a lot of notifications, this can slow down the webpage massively. Even worse, it affects the memory consumtion of the individual user the moment he uses the intro view.

Example: I have an admin account, which monitors changes on sections. The problem with notifications is the way it is loaded. It loads ALL notifications from ALL sections and not only from the section where you display the records. In my small section with 2000 records the query loads 25K notifications. I had to increase the memory limit in php.ini since it used almost 200 MB just to read the result of the query. Fortunately i run the application locally. If it would be over the internet, it would take "ages" to send the data over. Also processing the data in PHP takes quite some time.

Sackgesicht if you use notifications, make sure they are cleared early. If a user accumulates a lot of notifications, this can slow down the webpage massively.

I have applied some fixes there. We will have to test.

Sergey I have applied some fixes there. We will have to test.

Phase 1 of notification improvements successful. Next level would be the section_id "fix".

Still some "publish=1" queries around ... multi_level_select filter count ... child field double queries etc ...

Marked this topic as Cobalt 9 so I do not forget to ad it to Cobalt 9.

Hi all,

I wanted to do some update by saying that I've been working on 80% of the suggestions that Sergey sent me.

Basically I spent a lot of time by merging the two business types (after doing this I had a few issues to solve based on my clients requirements, eg, using menu links for multilevelselect values and keeping itemId to set menu as active) and checking more generic suggestions (unfortunately, some of them didn't apply to my business case, but it was good to re-check, eg, disable publish/unpublish features-this is a must-, disable subscriptions -again, a must-).

Thanks to Sergey, I've tracked down current performance issue to find that 2 custom fields are causing what's left of the problem (features field and custom geo), which I plan to improve in the following week.

In the meantime, the currently implemented suggestions seemed to work reducing the page load from 40-60s to 10-20s approx. I've checked that if I unpublish both features and custom geo, the time is brought down to 5-8s approx, so now I know that I have to work on these 2 fields.

I'll keep you posted!

Thanks! Matias

When you finish that all, I'll have a look over and see what else can be done. I believe it is possible to bring it down to 1-2s.

That was very pleaseant read.

I think when you deal with neighbourhood you will win another 3-7 seconds.

I think AWS should be quick. Do not forget that is you purchase 3 years heavy reserved servers you save a lot. I purchase only $30 a month for my c3.large server with is regularry $75.

I suggest for your setup not less then c3.large or better c3.xlarge. Of course even better is new generation of compute optimized instances c4 family.

Hi Sergey,

Thanks a lot for that, I'll keep that in mind when suggesting the client to buy AWS.

Best, Matias