Sackgesicht VIP
Total posts: 1,636
04 Aug 2013 00:32

Since J3.1.4 the debug mode gives now better information about the queries. It is easy now to identify the slow queries and find solutions to improve.

Most of the users might use Cobalt with a 1 section <-> 1 Type approach, while only a few might use several types per section.

Therefore the queries should be optimized for the single TYPE mode.

When checking on the slow queries on an intro view, i noticed 2 queries responsible for a slow down for a sigle type approach:

1)


SELECT DISTINCT `type_id` FROM xxxxx_js_res_record WHERE section_id = 3

2)


SELECT id as value, title as text FROM xxxxx_js_res_record WHERE published = 1 AND hidden = 0 AND section_id = 3 AND type_id IN(3) AND ctime < '2013-08-04 03:25:22' AND (extime = '0000-00-00 00:00:00' OR extime > '2013-08-04 03:25:22')

If query 1's only purpose is to determine the TYPES used in a section, than it can be ignored with only 1 TYPE available and the condition " AND type_id IN (3) " in query 2 can be removed to avoid an index_merge under MySQL.

If type_id in js_res_record is only to determine Multi Type condition then a multi column index (section_id, type_id) might be a better approach for Multi Tytpe setup. (see here )

In my setup i have a section intro view with 57K records having a relation field to another section of 38K records.

Query time:

SELECT DISTINCT .... section_id = 2 --> 215ms

SELECT DISTINCT .... section_id = 3 --> 138ms

SELECT id as value ... AND type_id .. --> 310ms

Just this 3 queries account for 663ms where it could be reduced to only around 150ms in removing the SELECT DISTINCT and adjusting the SELECT id query.

It is not only the performance gain, it also put less burden on the MySQL server.

Last Modified: 02 Mar 2014


Sergey
Total posts: 13,748
04 Aug 2013 23:03

If query 1's only purpose is to determine the TYPES used in a section, than it can be ignored with only 1 TYPE available

This is how it was before. But it turned out we have problems like this. User connect one type, added few records, then connected other type. What I mean there may be difference between types psection parameter and real types in section.

If I relay on type parameter then there may be a lot of errors on the page.

If query 1's only purpose is to determine the TYPES used in a section, than it can be ignored with only 1 TYPE available

" AND type_id IN (3) " in query 2 can be removed to avoid an index_merge under MySQL.

I do not know how to take it away. Because I do not know if this is filter or what. Due to reason above because some people love to mess everythign up :)

I'll think out the wayt o optimize it.


Sackgesicht VIP
Total posts: 1,636
05 Aug 2013 07:29

I had no internet the last day, so i could play around with the settings and the queries.

I optimized my section with 58K records ...

See the result :

Before the page took** 2,340ms , now it is 412ms**

The SQL query time went down from 1,565ms to 69 ms.

Especially in the cobaltrelate.php, there is a useless query which consumes a lot of time and memory resources.

That shows that it is quite possible to fine tune it without loss of functionality


Sackgesicht VIP
Total posts: 1,636
05 Aug 2013 07:40

If I relay on type parameter then there may be a lot of errors on the page.

No, it should be from the section and not from the type. I will describe the idea later in detail.


stblink VIP
Total posts: 501
07 Aug 2013 08:45

Sack what tool do u used to check the queries?


Sackgesicht VIP
Total posts: 1,636
07 Aug 2013 09:38

the build in debug mode from joomla 3.1.4/3.1.5 ...


Sackgesicht VIP
Total posts: 1,636
07 Aug 2013 23:24

Cobalt 8.279 provides already some improvement.

The query time with my example went down from 1560ms to 950ms.

After removing the alpha index it became 710ms.

To avoid high memory consumption, the alpha index query could already include the number of records.

like

SELECT SUBSTRING(UCASE(r.title),1,1) AS letter, count(*)

  FROM hsb0a_js_res_record AS r

  WHERE r.section_id = 2 

  AND r.published = 1 

  AND r.hidden = 0

GROUP BY letter

After disabling a boolean filter it became 560ms.

A Section parameter for "published" and "hidden" could save another 200ms. I hope this can be implemented.

If you use a child relate field in your intro view, a query, returning ALL records from the relate section is executed. Not needed for this purpose. Please check cobaltrelate.php


Sackgesicht VIP
Total posts: 1,636
12 Aug 2013 22:37

A Section parameter for "published" and "hidden" could save another 200ms. I hope this can be implemented.

A way to do it would be a new section parameter like "Record process mode" similar to "Category count mode" with options like "fast process" and "smart process".

Fast process would ignore the publish and hidden state.

Like ignoring line 561 and 571 of models/records.php


andiamine VIP
Total posts: 295
26 Sep 2013 12:40

Woow great :D

Powered by Cobalt