[Yanel-dev] YarepGroupManager.getGroup() very slow with 5000+ users

[Michael Wechner]

Hi Balz

I have refactored now the security library and updated Yanel to use the 
most recent version.

One can now configure the group implementation by seeting inside the 
realm configuration for example:

<ac-identities 
class="org.wyona.security.impl.IdentityManagerFactoryImpl" 
xmlns:yarep="http://www.wyona.org/yarep/1.0.0">
<yarep:repository-config>config/ac-identities-repository.xml</yarep:repository-config>
<yarep:group-implementation 
class="org.wyona.security.impl.yarep.YarepGroupImplV2"/>
</ac-identities>

instead

<ac-identities>config/ac-identities-repository.xml</ac-identities>

which of course still works

(see for example src/realms/from-scratch-realm-template/realm.xml)

The group implementation

org.wyona.security.impl.yarep.YarepGroupImplV2

is just not finished yet, but you should be able to get an idea from it 
and help you get started with your own
implementation if necessary.

Please let me know if you have any questions.

Thanks

Michael

Am 27.04.12 09:36, schrieb basZero:
> Hi Michael,
>
> sounds like a plan.
> First I will try to find a very simple workaround (e.g. by not storing 
> the member ids anymore in the group XML), as I think that most 
> functionality is not needing that, am I wrong? I think it is enough 
> just to have the group definitions in group XML and have the group 
> assignments in the user XML only.
>
> But to be clear: the above description is not the final solution :)
>
> Yes we can work together on this.
>
> Please also have a look at the pending pull requests which I posted 5 
> months ago, would be cool to have them finally in so that I can remove 
> all custom Yanel classes again...
>
> Cheers
> Balz
>
> On Fri, Apr 27, 2012 at 9:25 AM, Michael Wechner 
> <michael.wechner at wyona.com <mailto:michael.wechner at wyona.com>> wrote:
>
>     Hi Balz
>
>     Am 26.04.12 17:32, schrieb basZero:
>>     Hi,
>>
>>     I have noticed very poor performance (over 5 seconds) when using
>>     group = YarepGroupManager.getGroup(groupid)
>>
>>     and above all
>>     group.addMember(userObject) (this call is very slow!)
>>
>>     when having thousands of users.
>>
>>     1) YarepGroupManager.getGroup(groupid)
>>
>>     When looking at the code of getGroup(), it loops over ALL users and
>>     verifies whether the user really exists and only adds it when it
>>     exists.
>>
>>     From a performane point of view, I don't want the Yanel to verify
>>     the existance of 10'000 XMLs.
>>     Is there a way to speed this up? I would prefer the usage of JAXB
>>     (just converting XML to Java or the other way round). Then, if
>>     you really need verification of some items, you can loop over all
>>     items and do your logic.
>>
>>     2) group.addMember(userObject)
>>     on my local laptop, this method took 5 seconds alone with 1000 users.
>>
>>     the code mainly does:
>>     - first add the group to the user XML (which should be quick, did
>>     not measure it yet)
>>     - secondly the user id gets added to the group XML: here also
>>     createConfiguration() gets called which loops over all items, I
>>     think this is just too slow when not using a marshalling engine
>>     like JAXB.
>>
>>     What is your experience?
>
>     I have the same experience, whereas it depends on how you use
>     these classes. Also these were not built
>     for performance, but for functionality. Whereas this shall be no
>     excuse, but just an explanation.
>
>     I completely agree that we need to improve this with the intention
>     that these implementations should scale
>     for millions of users and groups.
>>     What can I do?
>
>     Do your own implementation of the group manager and configure it
>     inside your realm.xml configuration, e.g.
>
>     <ac-identities
>     class="com.zwischengas.wyona.security.impl.IdentityManagerFactoryImpl"
>     xmlns:zg="http://www.zwischengas.com/wyona/security1.0.0"
>     <http://www.zwischengas.com/wyona/security1.0.0>>
>     ...
>
>     whereas in order to save time you might want to derive all the
>     classes from Yarep which work and just overwrite
>     the implementations which don't scale well.
>
>     Even better, you create a "pull branch" such that you and I can
>     work together and I will be happy to help you on this,
>     because I also need such an implementation shortly.
>
>     We could call it org/wyona/security/impl/yarep2
>
>     or whatever
>
>     WDYT?
>
>     Thanks
>
>     Michael
>>
>>     Cheers
>>     Balz
>>
>>
>
>
>     --
>     Yanel-development mailing list Yanel-development at wyona.com
>     <mailto:Yanel-development at wyona.com>
>     http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wyona.org/pipermail/yanel-development/attachments/20120430/61b3e4fc/attachment.html>