data provider tuning for larger data sets (one million items)

Reply to Thread >
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 20
  1. #1
    Junior Member RainerAtSpirit's Avatar
    Join Date
    Mar 2012
    Location
    close to Cologne, Germany
    Posts
    15
    Thread score
    -1

    3

    +1

    You must log in to vote for this post.

    data provider tuning for larger data sets (one million items)

    Getting closer with my first demo project that uses Wakanda server as a REST producer and a combination of KendoUI widgets and a Twitter bootstrap template on the client side (no WAF, REST only).

    While the initial setup is working the REST server doesn't fly ...yet. It doesn't matter if I'm using a KendoUI grid or the native grid widget, sorting/filtering takes about 20-30 seconds to finish.

    I've seen smoothly running demos in the video presentation so my assumption is that there are a couple of performance tweaks that can be applied. Just not sure yet, which ones.

    It would be great if somebody could point me into the right direction.

    Thanks,

    Rainer

  2. #2
    Junior Member RainerAtSpirit's Avatar
    Join Date
    Mar 2012
    Location
    close to Cologne, Germany
    Posts
    15
    Post score
    -1

    4

    +1

    You must log in to vote for this post.

    Update: Here's the link to the alpha version of the KendoUI, Twitter bootstrap, Wakanda REST demo:

    http://wakanda.spirit.de:8081/

    A native wakanda grid (as a baseline) is available at http://wakanda.spirit.de:8081/native.html.
    While initial load time of the baseline is faster (unsorted), subsequent sorting is leading to the same slow loading times.

    DISCLAIMER: Work in progress!

    It would be great if somebody who was responsible for the one million items demo that was highlighted in various videos could get in contact with me to make that puppy really fly ;-).

    Thanks,

    Rainer

  3. #3
    Senior Member
    Join Date
    Jun 2011
    Location
    Kansas City
    Posts
    407
    Post score
    -1

    0

    +1

    You must log in to vote for this post.

    Interesting to see this live. Have you indexed the datastore class attributes? I would think the native Wakanda grid would sort a bit faster on b-tree indexed fields...

    Also, I know the Wakanda grid has a paging system that only loads the rows being displayed. To get the KendoUI grid running faster, you may need to find a way to only make REST calls to get the rows you want to display (probably not easy).

  4. #4
    Junior Member RainerAtSpirit's Avatar
    Join Date
    Mar 2012
    Location
    close to Cologne, Germany
    Posts
    15
    Post score
    -1

    1

    +1

    You must log in to vote for this post.

    Quote Originally Posted by Welsh View Post
    Interesting to see this live. Have you indexed the datastore class attributes? I would think the native Wakanda grid would sort a bit faster on b-tree indexed fields...
    snip
    Boy that makes a difference. I've indexed [b-tree] the relevant fields and now this feels a LOT better :).

    Thanks for pointing this out.

    Rainer

  5. #5
    Senior Member
    Join Date
    Jun 2011
    Location
    USA
    Posts
    1,456
    Post score
    -1

    0

    +1

    You must log in to vote for this post.

    I just ran your KendoUI version and it's pretty fast. Great job to hook it up to KendoUI. This is excellent to combine these two technologies for certain cases.

    Great job and thank you for sharing!
    Best Regards;
    ..Ben

  6. #6
    Moderator
    Join Date
    May 2011
    Posts
    131
    Post score
    -1

    3

    +1

    You must log in to vote for this post.

    Hello Rainer,

    It is a very interesting demo!!
    I noticed that, in your REST request, you don't use the "$method=entityset" property. It would really speed up your data access. This is what is used by the native WAF framework and widgets.

    Basically, when you have a REST request like that:
    Code:
    IP address/hostname:port/rest/DataClass?$filter="..."&$orderby="..."&$top=x&$skip=y
    you can add: &$method=entityset

    It will build a collection on the server based on your filter and orderby and then return in your JSON answer at the first level a property called __ENTITYSET which will contain a uri that you can use to further manipulate that collection.

    the uri looks like this:
    Code:
    IP address/hostname:port/rest/DataClass/$entityset/DD6E035F6CEF224C80694C090DA2A536
    the last part being a UUID generated by the server.
    Then when you wan to navigate through your collection, you can just send the REST request based on the uri.

    For example:
    Code:
    IP address/hostname:port/rest/DataClass/$entityset/DD6E035F6CEF224C80694C090DA2A536?$skip=1000&$top=30
    to get the entities from 1000 to 1029

    Using that entityset allow the server to avoid sorting and search for each request you make with the grid of KendoUI.

    You can have a timeout in seconds for the entityset, the default is 2 hours. (for exemple: &$timeout=120 for a 2 min timeout)

    When you are done with that entityset (in your case, when the user sorts on another column or enter chars in the query part) then you can force release it by:
    Code:
    IP address/hostname:port/rest/DataClass/$entityset/DD6E035F6CEF224C80694C090DA2A536?$method=release
    Last edited by Alexandre Morgaut; 03-27-2012 at 04:25 PM.
    Laurent Ribardiere

  7. #7
    Junior Member RainerAtSpirit's Avatar
    Join Date
    Mar 2012
    Location
    close to Cologne, Germany
    Posts
    15
    Post score
    -1

    0

    +1

    You must log in to vote for this post.

    Hello Laurent,

    Thanks for providing additional information about the entityset method. I've opted to avoid it in this first version as I wasn't sure about its impact.
    What would be some best practice recommendation when using the entityset method for larger result sets? e.g. I've added filtering based on company.id to the example. Comparing how the a native control handles this setup I see that for each company filter a new entityset is created. Having about 5000 companies most of the queries would be one time and not necessarily reusable in a given time frame.

    I'll add another example using this method later the week using that method, so that we can compare both variations.

    Thanks,

    Rainer

  8. #8
    Moderator
    Join Date
    May 2011
    Posts
    131
    Post score
    -1

    0

    +1

    You must log in to vote for this post.

    You are right, when you say that, if your collection is to be used only once, then you don't need to build an entityset on the server. But most of the time, especially with an interactive widget like a grid, where people can scroll, you need to access entities from that collection later on. (For example when you need another page). And in that case, it is much faster to use the entityset than to rebuild the collection, especially if it is built out of a sequential query or sequential sort.
    Laurent Ribardiere

  9. #9
    Junior Member RainerAtSpirit's Avatar
    Join Date
    Mar 2012
    Location
    close to Cologne, Germany
    Posts
    15
    Post score
    -1

    0

    +1

    You must log in to vote for this post.

    Quote Originally Posted by laurent.ribardiere View Post
    But most of the time, especially with an interactive widget like a grid, where people can scroll, you need to access entities from that collection later on. (For example when you need another page). And in that case, it is much faster to use the entityset than to rebuild the collection, especially if it is built out of a sequential query or sequential sort.
    Thanks for the clarification. I'm sold ;-) and come back to you later once I find time to implement it.

    Rainer

  10. #10
    Thibaud Arguillere
    Guest
    Post score
    -1

    0

    +1

    You must log in to vote for this post.

    Hi all,

    Rainer, to speed up the whole thing you could also tune the size of WakandaDB cache. By default it is set to 200Mo, which is probably to low for one million entities plus the indexes. A big cache makes it faster to access the data, even when it is read sequentially. The firts time the data is loaded from disk, it is "slow" (I use quote, because if you're using a SSD, it's not that slow actually), but then accessing it is very fast. Of course, it depends on the application and on the full amount of data, but WakandaDB perfectly scales: just add more memory to handle more cache.

    To increase the size of the cache, you can:
    • Edit the settings of the solution. Just double click the {yourSolutionName}.settings. WARNING: don't open the project settings, you can't set the size of the cache here.
      Once the setting form is open, change the size. For example, set it to 1,000 or 2,000 (= 1 or 2 G, since the unit is Mo).
      You need to quit/relaunch the Server and the studio for the setting to be taken into account (let's say that the need to quit the sudio is a bug ;-), you should be able to just close/reopen the solution)
    • Call setCacheSize() ssjs routine. It's not yet in the doc. But you can setup a bootstrap file that calls this routine:
      • Create a "bootstraps" folder, at the same level as the "data" folder
      • Put a javascript inside it (I usually name it initApp, but the name is up to you, the server executes all the .js files inside this "bootstraps" folder)
      • In the code of this file, just call the routine:
        Code:
        ds.setCacheSize(2*1024*1024*1024);//2Gb
    And, server-side, you also have the ds.getCacheSize() hat lets you know what's the value.

    Also, those routines have the same name as the one used in the WAF, they don't give the same information (in the WAF, it's the number of entities for a datasource, not an amount of memory).

    Some months ago, a blog was written about this.

    Thibaud

Reply to Thread >

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
Copyright 2013 4D - All Rights reserved