data provider tuning for larger data sets (one million items)

[RainerAtSpirit]

Getting closer with my first demo project that uses Wakanda server as a REST producer and a combination of KendoUI widgets and a Twitter bootstrap template on the client side (no WAF, REST only).

While the initial setup is working the REST server doesn't fly ...yet. It doesn't matter if I'm using a KendoUI grid or the native grid widget, sorting/filtering takes about 20-30 seconds to finish.

I've seen smoothly running demos in the video presentation so my assumption is that there are a couple of performance tweaks that can be applied. Just not sure yet, which ones.

It would be great if somebody could point me into the right direction.

Thanks,

Rainer

[RainerAtSpirit]

Update: Here's the link to the alpha version of the KendoUI, Twitter bootstrap, Wakanda REST demo:

http://wakanda.spirit.de:8081/

A native wakanda grid (as a baseline) is available at http://wakanda.spirit.de:8081/native.html.
While initial load time of the baseline is faster (unsorted), subsequent sorting is leading to the same slow loading times.

DISCLAIMER: Work in progress!

It would be great if somebody who was responsible for the one million items demo that was highlighted in various videos could get in contact with me to make that puppy really fly ;-).

Thanks,

Rainer

[Welsh]

Interesting to see this live. Have you indexed the datastore class attributes? I would think the native Wakanda grid would sort a bit faster on b-tree indexed fields...

Also, I know the Wakanda grid has a paging system that only loads the rows being displayed. To get the KendoUI grid running faster, you may need to find a way to only make REST calls to get the rows you want to display (probably not easy).

[RainerAtSpirit]

Interesting to see this live. Have you indexed the datastore class attributes? I would think the native Wakanda grid would sort a bit faster on b-tree indexed fields...
snip

Boy that makes a difference. I've indexed [b-tree] the relevant fields and now this feels a LOT better :).

Thanks for pointing this out.

Rainer

[BenHayat]

I just ran your KendoUI version and it's pretty fast. Great job to hook it up to KendoUI. This is excellent to combine these two technologies for certain cases.

Great job and thank you for sharing!

[laurent.ribardiere]

Hello Rainer,

It is a very interesting demo!!
I noticed that, in your REST request, you don't use the "$method=entityset" property. It would really speed up your data access. This is what is used by the native WAF framework and widgets.

Basically, when you have a REST request like that:

IP address/hostname:port/rest/DataClass?$filter="..."&$orderby="..."&$top=x&$skip=y

you can add: &$method=entityset

It will build a collection on the server based on your filter and orderby and then return in your JSON answer at the first level a property called __ENTITYSET which will contain a uri that you can use to further manipulate that collection.

the uri looks like this:

IP address/hostname:port/rest/DataClass/$entityset/DD6E035F6CEF224C80694C090DA2A536

the last part being a UUID generated by the server.
Then when you wan to navigate through your collection, you can just send the REST request based on the uri.

For example:

IP address/hostname:port/rest/DataClass/$entityset/DD6E035F6CEF224C80694C090DA2A536?$skip=1000&$top=30

to get the entities from 1000 to 1029

Using that entityset allow the server to avoid sorting and search for each request you make with the grid of KendoUI.

You can have a timeout in seconds for the entityset, the default is 2 hours. (for exemple: &$timeout=120 for a 2 min timeout)

When you are done with that entityset (in your case, when the user sorts on another column or enter chars in the query part) then you can force release it by:

IP address/hostname:port/rest/DataClass/$entityset/DD6E035F6CEF224C80694C090DA2A536?$method=release

[RainerAtSpirit]

Hello Laurent,

Thanks for providing additional information about the entityset method. I've opted to avoid it in this first version as I wasn't sure about its impact.
What would be some best practice recommendation when using the entityset method for larger result sets? e.g. I've added filtering based on company.id to the example. Comparing how the a native control handles this setup I see that for each company filter a new entityset is created. Having about 5000 companies most of the queries would be one time and not necessarily reusable in a given time frame.

I'll add another example using this method later the week using that method, so that we can compare both variations.

Thanks,

Rainer

[laurent.ribardiere]

You are right, when you say that, if your collection is to be used only once, then you don't need to build an entityset on the server. But most of the time, especially with an interactive widget like a grid, where people can scroll, you need to access entities from that collection later on. (For example when you need another page). And in that case, it is much faster to use the entityset than to rebuild the collection, especially if it is built out of a sequential query or sequential sort.

[RainerAtSpirit]

But most of the time, especially with an interactive widget like a grid, where people can scroll, you need to access entities from that collection later on. (For example when you need another page). And in that case, it is much faster to use the entityset than to rebuild the collection, especially if it is built out of a sequential query or sequential sort.

Thanks for the clarification. I'm sold ;-) and come back to you later once I find time to implement it.

Rainer

[Thibaud Arguillere]

Hi all,

Rainer, to speed up the whole thing you could also tune the size of WakandaDB cache. By default it is set to 200Mo, which is probably to low for one million entities plus the indexes. A big cache makes it faster to access the data, even when it is read sequentially. The firts time the data is loaded from disk, it is "slow" (I use quote, because if you're using a SSD, it's not that slow actually), but then accessing it is very fast. Of course, it depends on the application and on the full amount of data, but WakandaDB perfectly scales: just add more memory to handle more cache.

To increase the size of the cache, you can:

• Edit the settings of the solution. Just double click the {yourSolutionName}.settings. WARNING: don't open the project settings, you can't set the size of the cache here.
  Once the setting form is open, change the size. For example, set it to 1,000 or 2,000 (= 1 or 2 G, since the unit is Mo).
  You need to quit/relaunch the Server and the studio for the setting to be taken into account (let's say that the need to quit the sudio is a bug ;-), you should be able to just close/reopen the solution)
• Call setCacheSize() ssjs routine. It's not yet in the doc. But you can setup a bootstrap file that calls this routine:
  • Create a "bootstraps" folder, at the same level as the "data" folder
  • Put a javascript inside it (I usually name it initApp, but the name is up to you, the server executes all the .js files inside this "bootstraps" folder)
  • In the code of this file, just call the routine:
    

    ds.setCacheSize(2*1024*1024*1024);//2Gb

And, server-side, you also have the ds.getCacheSize() hat lets you know what's the value.

Also, those routines have the same name as the one used in the WAF, they don't give the same information (in the WAF, it's the number of entities for a datasource, not an amount of memory).

Some months ago, a blog was written about this.

Thibaud