Incomplete Category Index page with more than 2000 category

Could you investigate what is the problem? ,

I would love to! Speed is our greatest concern. We have tested with 80 000 records but not yet with lot of categories.

Please, give me temp superadmin access to your backend.

You can post in in Private blok through button in menu.

Hi Sergey, thanks for a fast respond,

Here are the links:

Скрытый текст

I have two sets joomla (exact copy), one with 3000 records and other with 2000 records, and each of it gave different result. The links above are for both sets.

If you need more deep access just let me know.

Hi Sergey,

Forgot to mention, just change a last path of the links "joomla25a" with "joomla25b" to switch between 3000 and 2000 records set.

Ok. This is cool. Cool because I will be able to optimise this category issue.

I need access to something like PMA (phpmyadmin) if you have please. I need to test queries right ther on your DB and be able to add/delete indexes.

Hi Sergey,

Sorry for the delay, My server was set behind a firewall, so I need to reconfigure the firewall to allow ssl connection for phpmyadmin to be accessed from outside.

Now you can try to access it

Скрытый текст

Thanks.

Nenda

:D Looks like I got your server down with my experiments :)

But, that was good. What i did is.

1. Now if you turn subcategories level to "Do not show subcategories" there is simple query.

2. If you want to show subcategories, query optimised from 7.8s to 0.019s.

And we release new version today. Please keep one eye on our blog where we post updates announcements.

Hi Sergey, yes it was down but it's OK I have turn it back on,

Thanks for testing on the problem, Looking forward for the new version...

Soon as it fixed, I will prepare fields and contents and see how its speed performance..

Thanks Sergey,

Nenda

Please after you try release we made today, report here. I want to be sure it works.

Hi Sergey, sorry to ask, what should I do to apply the new version?, do I need to uninstall the previous version first?, (that means all category record will be deleted)

Thanks.

(Thank's for the advice Maik)

Sergey, now I put my >5000 categories into the system, and create 2 set exact copy,

1. With Cobalt v.7.8747

The result is:

Opening main category page = 5 sec

Opening parent level = 35.5 sec

Opening child level = 28.5 sec

Level up = 34.5 sec

2. With Cobalt v.7.8799

The result is:

Opening main category page = 2.25 sec

Opening parent level = 36.5 sec

Opening child level = 28.2 sec

Level up = 35.5 sec

I don't know how to count a time like the way you did (" query optimised from 7.8s to 0.019s."), so I just manually count it using a timer (stopwatch), so the number above was not accurate but it is close enough.

Opening time for the main category page was decrease > 50%, that because the main category page was set up without showing the subcategory (as you've explained).

But when we go down to the sub category, the time consume for both version almost the same.

Another thing is that, in Cobalt v.7.8799, we cannot show subcategories anymore in main category page. I use the same copy from the first set and update the Cobalt, so I think the parameter will not be change.

(where can I change "Show subcategories" parameter?, I've tried on Section record mode parameter, category index parameter, but no luck).

Скрытый текст

Thanks

Nenda

I see.

Your pages are really takes long to load. But i do тещ count load time. This is may be because internet speed of your server. You have to look in to profile in debug output.

Site B

This is longest page. It takes 3 sec to form the page. All other time (1 minute for me) is spend for transfer content to my browser.

Your first page loads 1.5 s. If I click category agents it takes longer. But still only 2.2s and not 35. But it takes more than a minut to load page. I have turned ON Gzip compression but it did not helped a lot. I have checked and size of your uptput is the same w/ or w/o Gzip

And here is the category of section. It is almost 1M data and more than a minute time

although it is only 2.4 seconds

Conclusion

There is definitely something wrong with your server. I suspect it is connected to regular internet uplink with not very big upload speed. Because your server upload for my is download speed. And If I understood correctly it runs on XAMP.

I'll make backup of your Joomla and test it on average hosting server. Please keep my access opened. I'll do some more tests.

Ok, here I am :)

I have backed up your DB and installed it locally. Here is the result of benchmark

100 requests 10 concurrency

Concurrency Level:      10

Time taken for tests:   4.415 seconds

Complete requests:      100

Failed requests:        43

   (Connect: 0, Receive: 0, Length: 43, Exceptions: 0)

This mens that 100 request to section homepage were made in 4.415 seconds total. But you can see a lot of fail requests.

I tried 100 requests 1 concurrency

Concurrency Level:      1

Time taken for tests:   8.506 seconds

Complete requests:      100 

Failed requests:        49

   (Connect: 0, Receive: 0, Length: 49, Exceptions: 0)

It took a bit longer. But still goot time for 100 or even 50 full requests.

No I took URL of second level category

the same 100 requests 10 concurrency

Concurrency Level:      10

Time taken for tests:   3.720 seconds

Complete requests:      100

Failed requests:        55

   (Connect: 0, Receive: 0, Length: 55, Exceptions: 0)

And then again 100 requests 1 concurrency

Concurrency Level:      1

Time taken for tests:   10.250 seconds

Complete requests:      100

Failed requests:        47

   (Connect: 0, Receive: 0, Length: 47, Exceptions: 0)

The profile shows only 1.07 second for example Service category.

Although I noticed other thing. You had category filters enable in filter template. And it loaded All categories on every page. I have turned it off and now load speed in much much quicker.

Wow, that is a very useful test method, Sergey,

Based on your benchmark, do you think that is still acceptable performance? (I mean not too slow)

Because the content was not loaded yet, and I have more than 100,000 records.

I agree with you about the internet connection factor, you also have mention about Xampp, is that also have contribution to the speed performance?, could you suggest me which one a better one (Xampp, Wamp, UniServer) or should I use individual installation of Apache/MySQL/PHP (for local installation purposes)?

And about the filter template you have mention, which parameter you were referring to? (see attached pict)

What this filter use for?, if I disable it what feature that I will lose?

One more, how do you make the main category in site B to show subcategory again? (I still cannot find the parameter)

Again, highly appreciate for your support.

Nenda

One more, how do you make the main category in site B to show subcategory again? (I still cannot find the parameter)

There is a small bug in current Cobalt. You have to sow one level more. If you want to show 1 evel you have to select in parameter 2 levels. I have fixed it for next release.

One more, how do you make the main category in site B to show subcategory again? (I still cannot find the parameter)

And about the filter template you have mention, which parameter you were referring to? (see attached pict)

When you click small gear button by Filters template list (your second highlight)

You lose what you turn off. Just filtering ability by category. It means that people will not be able to see records in few categories at the same time.

But I have just remembered. There is parameter in the same template setting the category filter style.

By enabling AJAX method you can keep category filters.

Filter you can see when click on advanced search link under search text input.

I agree with you about the internet connection factor, you also have mention about Xampp, is that also have contribution to the speed performance?, could you suggest me which one a better one (Xampp, Wamp, UniServer) or should I use individual installation of Apache/MySQL/PHP (for local installation purposes)?

I would suggest manual setup. Everything separately. Although if you have Zend Server Pro you can install it for apache and PHP. Zend server and apache is different than XAMP and WAMP and others. Because they have to emulate server, and zend server just install it. I cannot explain it good on english.

The best of course to purchase cloud computing for example at softlayer.com. You will get very easy extendable and very stable platform. You can also use Amazon services or Windows Azure. Any of them are good but Amazon litte bit quicker.

More over you have 100 000 records. You have to prepare nice server anyway.

Make it *nix based. Optimise PHP with APC and so on... If you want your site to run quickly you have to prepare good server.

And also, if you get problems with your 100 000 records we will do everything we can to make it run instantly. may be you will have to sacrifice some features, but I want to ensure that we will achieve best result possible. We already have installation with 80 000. I hope he will mention it somewhere in the comments. So right now I believe that 100 000 records and 5 000 categories are ok for Cobalt. But it have to be good server. Let's say minimum

1G RAM, 1x 2.0 Ghz CPU, 100 Mbs Ethernet, *nix based (like CentOS)

I would also have a hope that I can contact your 80,000 records client, maybe he can share the method on how he load his records into the system.

Nenda,

as of now i have an installation with 88,786 records and 229,162 uploaded files with 6 TYPES/SECTIONS and 30 categories.

We experienced several slowdowns while populating the system with data, which all were solved by Sergey along the way. We always found a way to improve Cobalt, which you can see in Sergey's Blog-posts.

It is good to have "real" solutions to see the performance of Cobalt, rather than simulated installations. I believe there are still some small areas to improve, but as of now, at least for our solution, be have a solid base were we can work with.

The records were partially manual encoded, but the majority was imported by several variations of a script with the help of Sergey. This import process depends on a lot of factors like what type of fields will be imported and also on their parameters (especially "searchable"). To do this, you need to have intimate knowledge of the way Cobalt stores the content of individual fields and also the way records consolidate the data in the record table. It gets even more complex with fields like "Uploads" or "Gallery".

I had to modify the existing database index structure a little to speed up the import process and avoid an early timeout of our batch-import.

Import as of now is an individual process.

If I know more about your source data (format, etc) and your target fields, I might be able to give some suggestions how to work on it.

My advice would be to hire MintJoomla to do a script for it (also in appreciation of all the free outstanding support, they are providing). I know they are very busy at the moment for several things, therefore it would be good if we can prepare some "ground-work" first to make it easier for them to help you.

Another option would be to wait for Cobalt 8, which will have an import facility.

As for your question on the single field purchase, it is a "life-time" purchase. See more details about it here .

In our case, the import of 500 source records created around 12,000 internal SQL queries since we used the relation fields (parent and child) and 1 source record created or affected several Cobalt records.

Hallo Sackgesicht,

Thank you for your kind to share your experience and method in importing records. Your data records are definetly more complex than I have, but it give me some Idea for my future development.

It is good for us to have Sergey and his team to support us with their amazing work. Cobalt7 structure is definetly much more simple than Resources (from the way I see it) but it is much more faster in querying, before Cobalt7 born, I have tried my records with ME Resources and have an issue with speed performance. I believe with Cobalt now I can have a better performance.

Currently, I use a manual import to load my records, but the method is quite the same with what you have described, just I didn't use a script because the lack of my programming knowledge :D.

The way I did is to create some sample records of categories, contents, fields, section, set all the parameters the way I would like to be, investigate which table that has a relation with the changes, then export it to Excel. From that file, I'll be able to know how the data were being stored then I use it as a template for my real records. Final step is to import back the data into the system. That's how I did with my 5000 categories. The difficult things is to find a relations between fields and tables, we have to be careful otherwise it could make the system crash.

I hope the I/E feature in Cobalt8 could make this work easier.

As you suggested, now, I'm preparing the ground base work for my content records.

Again, thank you for your kind help,

BTW, Does your site already live?, if so, may I know is the address? so I can learn from it.

Hi Sackgesicht,

It is OK, I thought your site is for internet consumer.

Anyway, Thank you for your great contribution..