[Yanel-dev] YarepGroupManager.getGroup() very slow with 5000+ users

[basZero]

Hi Michael,
I will have a look at it very soon and I will give a short update on this.
Cheers
Balz

On Mon, Apr 30, 2012 at 12:57 AM, Michael Wechner <michael.wechner at wyona.com
> wrote:

>  Hi Balz
>
> I have refactored now the security library and updated Yanel to use the
> most recent version.
>
> One can now configure the group implementation by seeting inside the realm
> configuration for example:
>
>   <ac-identities
> class="org.wyona.security.impl.IdentityManagerFactoryImpl" xmlns:yarep=
> "http://www.wyona.org/yarep/1.0.0" <http://www.wyona.org/yarep/1.0.0>>
>
> <yarep:repository-config>config/ac-identities-repository.xml</yarep:repository-config>
>     <yarep:group-implementation
> class="org.wyona.security.impl.yarep.YarepGroupImplV2"/>
>   </ac-identities>
>
> instead
>
> <ac-identities>config/ac-identities-repository.xml</ac-identities>
>
> which of course still works
>
> (see for example src/realms/from-scratch-realm-template/realm.xml)
>
> The group implementation
>
> org.wyona.security.impl.yarep.YarepGroupImplV2
>
> is just not finished yet, but you should be able to get an idea from it
> and help you get started with your own
> implementation if necessary.
>
> Please let me know if you have any questions.
>
> Thanks
>
> Michael
>
> Am 27.04.12 09:36, schrieb basZero:
>
> Hi Michael,
>
>  sounds like a plan.
> First I will try to find a very simple workaround (e.g. by not storing the
> member ids anymore in the group XML), as I think that most functionality is
> not needing that, am I wrong? I think it is enough just to have the group
> definitions in group XML and have the group assignments in the user XML
> only.
>
>  But to be clear: the above description is not the final solution :)
>
>  Yes we can work together on this.
>
>  Please also have a look at the pending pull requests which I posted 5
> months ago, would be cool to have them finally in so that I can remove all
> custom Yanel classes again...
>
>  Cheers
> Balz
>
>  On Fri, Apr 27, 2012 at 9:25 AM, Michael Wechner <
> michael.wechner at wyona.com> wrote:
>
>>  Hi Balz
>>
>> Am 26.04.12 17:32, schrieb basZero:
>>
>> Hi,
>>
>>  I have noticed very poor performance (over 5 seconds) when using
>> group = YarepGroupManager.getGroup(groupid)
>>
>>  and above all
>> group.addMember(userObject) (this call is very slow!)
>>
>>  when having thousands of users.
>>
>>  1) YarepGroupManager.getGroup(groupid)
>>
>>  When looking at the code of getGroup(), it loops over ALL users and
>> verifies whether the user really exists and only adds it when it exists.
>>
>>  From a performane point of view, I don't want the Yanel to verify the
>> existance of 10'000 XMLs.
>> Is there a way to speed this up? I would prefer the usage of JAXB (just
>> converting XML to Java or the other way round). Then, if you really need
>> verification of some items, you can loop over all items and do your logic.
>>
>>  2) group.addMember(userObject)
>> on my local laptop, this method took 5 seconds alone with 1000 users.
>>
>>  the code mainly does:
>> - first add the group to the user XML (which should be quick, did not
>> measure it yet)
>> - secondly the user id gets added to the group XML: here also
>> createConfiguration() gets called which loops over all items, I think this
>> is just too slow when not using a marshalling engine like JAXB.
>>
>>  What is your experience?
>>
>>
>>  I have the same experience, whereas it depends on how you use these
>> classes. Also these were not built
>> for performance, but for functionality. Whereas this shall be no excuse,
>> but just an explanation.
>>
>> I completely agree that we need to improve this with the intention that
>> these implementations should scale
>> for millions of users and groups.
>>
>>  What can I do?
>>
>>
>> Do your own implementation of the group manager and configure it inside
>> your realm.xml configuration, e.g.
>>
>> <ac-identities
>> class="com.zwischengas.wyona.security.impl.IdentityManagerFactoryImpl"
>> xmlns:zg="http://www.zwischengas.com/wyona/security1.0.0"<http://www.zwischengas.com/wyona/security1.0.0>
>> >
>> ...
>>
>> whereas in order to save time you might want to derive all the classes
>> from Yarep which work and just overwrite
>> the implementations which don't scale well.
>>
>> Even better, you create a "pull branch" such that you and I can work
>> together and I will be happy to help you on this,
>> because I also need such an implementation shortly.
>>
>> We could call it org/wyona/security/impl/yarep2
>>
>> or whatever
>>
>> WDYT?
>>
>> Thanks
>>
>> Michael
>>
>>
>>  Cheers
>>  Balz
>>
>>
>>
>>
>> --
>> Yanel-development mailing list Yanel-development at wyona.com
>> http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development
>>
>
>
>
>
>
> --
> Yanel-development mailing list Yanel-development at wyona.com
> http://lists.wyona.org/cgi-bin/mailman/listinfo/yanel-development
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wyona.org/pipermail/yanel-development/attachments/20120430/1b8137c0/attachment-0001.html>