For those who like a little more detail…

I just moved our grails app from using the shiro plugin to using the spring-security plugin(s). I like shiro’s filter-based config, but all the pre-built extension modules that Burt Beckwith has put together for spring-security (LDAP, CAS, etc.) makes it much easier for us to support the range of environments in which we have to deploy.

The one feature which took me a little while to figure out was how to have our app auto-create a user domain object when it is using an external authentication source. For example, say an instance of our app is configured to authentication against an LDAP server. The app has a MyUser class that holds local settings for users like preferences, documents, etc. When a user signs in for the first time and makes it past the authentication step, we need to automatically create a MyUser instance and associate it with the LDAP username. With the shiro-based authentication, we did this in the controller method which handled the authentication itself. Spring security works a little differently and there isn’t a central, post-authentication landing point.

If your app is always deployed with the same type of authentication (e.g. always with LDAP), you could put the persistence code into a custom UserDetailsService. There are several posts on the web that discuss creating a custom UserDetails object and a corresponding service for it, so this was the first approach I looked at. Chapter 11 of the spring-security-core plugin’s user guide has info on it as well. The primary shortcoming is that you can’t chain together UserDetailsServices. You have to implement one for each form of authentication with which you want to work.

If your app must work with a variety of authentication methods, it is easier to register a listener with Spring Security. Chapter 7 of the plugin guide discusses the two ways to do this. I found that handling the AuthenticationSuccessEvent was all I needed. Since we already had a Grail’s service that handles various user-related tasks, the listener object was dirt simple:

import org.springframework.beans.factory.InitializingBean
import org.springframework.context.ApplicationContext
import org.springframework.context.ApplicationContextAware
import org.springframework.context.ApplicationListener
import org.springframework.security.authentication.event.AuthenticationSuccessEvent

class MyAuthenticationEventListener implements ApplicationListener<AuthenticationSuccessEvent>, InitializingBean, ApplicationContextAware {
    ApplicationContext applicationContext
    def userService

    void afterPropertiesSet() {
        userService = applicationContext.getBean('userService')
    }

    void onApplicationEvent(AuthenticationSuccessEvent e) {
        //the principal field of the source object is a UserDetails object of some form.
        //The spring-security API contract guarantees that at least the username field will be populated.
        userService.createUser(e.source.principal)
    }
}

Since the whole class really just has one line of “functional” code, I could have used the closure-based approach described in section 7.3 of the user guide. I just prefer to keep true code out of the Config.groovy file.

Then, within the UserService.createUser method, the key lines look something like this:

    def user = MyUser.findByUsername(userDetails.username)
    if (!user)
    {
        MyUser.withTransaction {status ->
            user = new MyUser(username:userDetails.username /*, set any other props you want to store locally*/)
            user.save(flush: true, failOnError:true)
        }
    }
    return user

One note of interest – without the withTransaction statement, you may get an exception stating that no hibernate session exists and one cannot be opened. The withTransaction closure wraps this up nicely for you.

    GParsPool.withPool {
        urlList.eachParallel {
            ...get/post with Jersey client...
        }
    }

Some of my co-workers expressed concern that this would involve creating and destroying the pool data structures (specifically, Java threads) for every url or server collection we submitted. They thought this would take too much overhead. So I decided to put together a few tests to see what GPars could get us using its simplest form of concurrency. First, an interesting tidbit from the reference guide:

While the GParsPool class relies on the jsr-166y Fork/Join framework and so offers greater functionality and better performance, the GParsExecutorsPool uses good old Java executors and so is easier to set up in a managed or restricted environment. It needs to be stated, however, that GParsPool performs typically much better than GParsExecutorsPool does. (Section 3 intro).

and, from Groovy in Action v2:

GParsPool does not create threads. Instead, it takes them from a fork/join thread pool of the jsr166y library, which is a candidate for inclusion in future Java versions. GPars uses this library extensively, especially its support for parallel arrays that are the basis for all parallel collection processing in GPars.” (Section 17.2)

If these statements were correct, then hopefully we didn’t have to worry about maintaining an existing pool and setting up a countdown latch of some form.

Tests

I threw together some quick-and-dirty tests:

• A series of mathematical operations (i.e. pure cpu)
• Open and read in the text of 120 files, each about 1-4Kb
• Get the contents of a small web page (9kb) hosted on a machine on the local network

Obviously, these were not meant to be a definitive test of all of GPars capabilities. I just wanted to see if we could use the simplest form of GPars notation or if we had to do something more complicated.

I ran each test in a loop various numbers of times (100 up to 10000), just to see if there was significant difference over time. The results I list below are for the test that ran the loop 5000 times. The core bit of code I timed was something like this:

    int ms = 0
    5000.times {
        StopWatch timer = new StopWatch().go()
        GParsExecutorsPool.withPool {
            List result = data.collectParallel {
                //(1..100).sum {i -> i^it}
                //or
                //it.text.size()
            }
        }
        ms += timer.stop
    }
    def message = DebugUtils.logTimePerItem("GParsPool", numLoops, ms)
    println message

where “it” was a File or URL (or a number for test #1).

I ran the test using regular sequential code (i.e. commenting out the withPool block and changing collectParallel to the normal collect), and then using GParsPool.withPool and GParsExecutorsPool.withPool. I also tried using the GPars ParallelEnhancer class and the makeConcurrent, both of which let me just use the normal collect call rather than having to write collectParallel. For some reason, these conventions slowed down the collection processing by a noticeable amount. I did not dive into why that happens, but I suspect it has to do with the additional overhead of the custom MetaClass handling.

Results

These are the results on my Dell Precision M6400 running 32-bit Fedora 14. The JVM had 1.5G of RAM.

Test: Mathematical Operation

Normal: 5000 in 14594 (343/s)
GParsPool: 5000 in 12480 (401/s)
GParsExecutorsPool: 5000 in 15685 (319/s)

I reran this test with the timer inside the withPool call to see what the overhead of creating the pool was, i.e.:

        GParsExecutorsPool.withPool {
            MemStopWatch timer = new MemStopWatch().go()
            List result = data.collectParallel {
                (1..100).sum {i -> i^it}
            }
            ms += timer.stop
        }

with these results:
GParsPool: 5000 in 11253 (444/s)
GParsExecutorsPool: 5000 in 13308 (376/s)

So setting up the pool each time definitely has some overhead cost, but even doing that, the GParsPool is still faster than normal, single-threaded sequential execution, even on my little ol’ dual core machine.

Test: Open and Read Files

Normal: 5000 in 31726 (157/s)
GParsPool: 5000 in 24050 (207/s)
GParsExecutorsPool: 5000 in 25351 (197/s)

Very similar scale of results as with the straight mathematical operation.

Test: Get small web page over local network

All tests resulted in the same numbers – network latency was the deciding factor. Sorry for not having the exact metrics on this one.

Conclusion

So what does all this mean? I think it means that just using the simple GParsPool.withPool structure to iterate over a collection is perfectly fine for our needs. We could optimize a bit with different structures and a pre-existing pool, but it honestly won’t make a bit of difference in real performance given that network latency is the deciding factor for us. Your mileage may vary, especially if you are running an open server that has higher load requirements.

• There is no official standard. A qualified name is officially defined as two strings – the namespace and the local name. Oh, great.
• One of the first papers on this by James Clark says {namespace}local is proper. This is what javax.xml.namespace.QName.toString produces, and the QName.valueOf method will parse that format. This form is also what the groovy QName class uses, but, interestingly, the equals for that class will accept a string that uses a colon delimiter.
• http://docstore.mik.ua/orelly/xml/xmlnut/ch04_02.htm talks of both {namespace}local and namespace#local
• http://www.rpbourret.com/xml/NamespacesFAQ.htm#names_15 has great detail on namespaces overall. It talks of {namespace}local and another form, namespace^local, which is what SAX filter uses, according to the page. I found no other examples or mention of this “caret” format.
• javax.xml.soap.Name uses namespace:local. Apache axis does the same thing, which is not surprising considering I believe one came from the other.
• ECMAScript for XML (and, thus, Adobe ActionScript) uses 2 colons – namespace::local. This is partly because it uses the two colons as an operator of sorts, and needed to separate it from other uses of a colon in the ECMAScript syntax.
• Dublin Core (DC) explicitly defines the URIs of the terms in its schema. It uses “the path divider ‘/’ as the delimiter between namespace and local name. Of note, if you try to put one of those URIs into a web browser as a URL, it will redirect to a page which uses ‘#’ to note the fragment in an RDF schema. For example, http://purl.org/dc/terms/ will resolve to http://dublincore.org/2010/10/11/dcterms.rdf#name. I didn’t find any other schema/taxonomy that explicitly defines the URI for each element.
• Regardless of the above behaviour, the Dublin Core XSD defines the namespace to include the ending ‘/’.
• The namespaces of the RDF and OWL specifications include an ending ‘#’.
• All namespaces included in the output from pingthesemanticweb, which lists the most popular semantic schemas, end in ‘/’ or ‘#’. Even the few that use urn format end in ‘#’ (e.g. urn:x-inspire:specification:gmlas:HydroPhysicalWaters:3.0#).
• The Department of Defense Discovery Metadata Specification (DDMS) namespace, based heavily on Dublin Core, includes the ending ‘/’ just as DC does.
• I could not find any namespaces that end in ‘}’, ‘^’, or ‘:’ (the first two of which are illegal, I think)


So, you might be thinking that we could just concatenate the namespace and local name together to form the string. To parse it, we could then split the string at the last occurrence of the delimiter character, keeping the delimiter as part of the namespace if it is a ‘/’ or a ‘#’. But wait! There’s more…

• Many non-semantic-web schemas, like the XML Schema itself, xlink, and the OGC standards like gml, do not include the ending delimiter in their namespaces.
• National Information Exchange Model (NIEM) namespaces, arguably somewhat-semantic, also do not include a trailing delimiter.
• Neither does the Intelligence Community Metadata Standard for
  Information Security Marking (IC-ISM) namespace (which is in urn format).
• Nor does the DOD core metadata OWL schema, at least as far as I can tell. Sorry, I couldn’t find an exact reference to that one.

Resolution Rules

So if you want to represent a particular qualified name as a string and do it in a way that others are most likely to recognize as the “accepted” way to represent that particular QName and you want it to be reversible, at least within your own app, the best rules I could come up with are:

Creating the String

Call the path divider ‘/’ and fragment ‘#’ symbols sticky delimiters because they may be a part of (i.e. stick to) a namespace. Call the other possibilities (‘:’, ‘::’, ‘}’, ‘^’) formal delimiters because you know they only serve the purpose of being a delimiter.

1. If the namespace ends in a delimiter of any form, simple append the local name directly to it.
2. Else, use ‘:’, ‘^’ or, to be totally safe, surround the namespace string with ‘{}’ and then append the local name. I chose ‘:’ because I at least saw some uses of that form on various pages while I never saw any uses of the caret ‘^’ or the surrounding ‘{}’. If you have total control of your input and output, use the surrounding braces format since it is totally unambiguous.

Parsing the String

1. If there is a ‘{}’ pair, can assume form is {namespace}local
2. Else, find the last possible delimiter in the string. If it is a “formal” delimiter, then drop the delimiter and make the namespace the chars before it and local name the chars after it.
3. Else, if the last delimiter is “sticky”, you have to guess whether to keep it in the namespace. I put some basic logic in my code to recognize well known namespaces (like those above) that do not end in a delimiter, but then otherwise assume that a sticky delimiter should be included in the namespace.

It’s not a perfect solution, but that’s what you get when there is no standard.

You add a feature that requires some version of groovy greater than 1.7.4 (the version included with the latest gmaven-plugin, 1.3).  In my case, I used the @Delegate annotation with some inheritance in a test configuration and hit a bug that was fixed in groovy 1.7.6.  No matter what version I used in my dependencies section, my tests were executed under groovy 1.7.4.  I finally came up with the configuration below which let me run with a different groovy.  Note that it made no difference what I included in the dependencies section.  The gmaven-plugin configuration appears to be completely independent of that.

<plugin>
    <groupId>org.codehaus.gmaven</groupId>
    <artifactId>gmaven-plugin</artifactId>
    <version>1.3</version>
    <configuration>
        <providerSelection>1.7</providerSelection>
        <!-- This is only used if you want to run a groovy script from the command line using maven -->
        <source>${groovy.script}</source>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>compile</goal>
                <goal>testCompile</goal>
            </goals>
        </execution>
    </executions>
    <!-- This block is required in order to make the gmaven plugins use a groovy other than 1.7.4.
     This is independent of the groovy entry in the dependencies section.  This does not affect the class path.

     What is interesting is that there must be both the gmaven.runtime entry with the explicit excludes
     and the additional dependency on whatever version we do want to use.  If you exclude the former,
     it will throw an exception. -->
    <dependencies>
        <dependency>
            <groupId>org.codehaus.gmaven.runtime</groupId>
            <artifactId>gmaven-runtime-1.7</artifactId>
            <version>1.3</version>
            <exclusions>
                 <exclusion>
                     <groupId>org.codehaus.groovy</groupId>
                     <artifactId>groovy-all</artifactId>
                 </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy-all</artifactId>
            <version>1.7.6</version>
        </dependency>
    </dependencies>
</plugin>

Another example is yet another good post by my friend and coworker Jeff Erikson detailing his research into problems with Grails config options getting wiped during integration tests.  I had been meaning to write up a note about how the GrailsUnitTestCase class hides the true config settings, so you don’t want to accidentally use it in integration tests.  Jeff took my little tip a lot farther with his post.  Thanks, Jeff!

Rather than posting the details off in the wilderness of my own blog, I responded to a post on the m2eclipse (the maven-eclipse plugin) mail list from last Fall where someone was asking how to add groovy support to maven projects. The author of the m2eclipse plugin stated that the groovy plugin needed to support some hooks into m2eclipse. While that may be true to get fully automated setup and synchronization, you can get the two plugins working together. See this thread for the info:
http://www.nabble.com/Using-M2Eclipse-together-with-Groovy-td19261256.html

For those that cannot or do not want to move from Eclipse to IDEA or NetBeans, here are a few tips for getting by with the plugin for now:

1) The largest issue is a memory leak. I posted to the groovy user list about it over a year ago. (My explanation of the problem in that post was slightly incorrect. The problem will occur in any project with a lot of classes on the class path, not just one that includes another project.) I think the issue has gotten better in some circumstances, but it still makes it impossible for me to easily integrate groovy classes into my major projects.
Work Around: I have a small “GroovySandbox” project that has the class directory and key required libraries from my main project on its classpath. (Do not include the main project as a dependent project because you will have the exact same memory problem.) I write new scripts and classes in this project and then when they are ready, I copy them into the main project. The memory issue still has an affect, but I can work a full day on groovy files without having to restart eclipse rather than having to restart every 20-30 minutes.

2) Whenever I modify the class path of a project that includes groovy classes (thus triggering a clean build by Eclipse), the groovy class files are wiped with everything else and are not regenerated. I have to go in and touch each groovy class to get it to compile.
Work Around: I now have enough groovy classes that this isn’t feasible, so I created an ant target that compiles only the groovy classes and I trigger it from within Eclipse when necessary. Eclipse detects the newly generated class files and the remaining java classes I have that are dependent on the groovy classes finish compiling. (Why do I have to mix groovy and java so much? See #1, above and #3, below. Unless I want to take advantage of groovy syntax features a lot, I stick with Java.)

3. When I open a groovy class in the editor, it often doesn’t populate the Outline view. Makes it hard to navigate the file.
Work Around: Touch the file, or add-and-delete a space. This was a pain when using CVS for our SCM since the files were tagged as modified, but since switching to Subversion there hasn’t been a problem. I guess Subversion uses something beyond file dates to track mods.

4. When debugging, if I try to step into a class (java or groovy) that is included from another project (which I do frequently since nearly all classes are in the main project due to #1, above), I get an error window saying that the class is part of the Groovy Libraries plugin path and that the path is not modifiable. The screen doesn’t give you a button to modify the source lookup path for the directory/jar. (This one is relatively new. It started happening in some update around the beginning of the year.)
Work Around: Go into the project properties->Java Build Path->Libraries and manually set the “Source Attachment” property for the offending directory/jar. The next time you start the debugger, it will let you step in without a problem.

But then Groovy 1.6 came out and I saw that the team had added some very cool annotations to make categories even more powerful and easier to use. (See InfoQ for a great write up of the 1.6 features.) Okay – you don’t improve a dead feature, so what was I missing?

This past weekend I got a chance to ask the person from whom I first learned about Groovy, Scott Davis. His Blue Pill and Red Pill talks at NFJS were what first got my mind churning about the possibilities that Groovy opened up. Before sitting in on another of his talks at the most recent NFJS conference this past weekend, I asked him why one would use categories in a DSL rather than modifying meta classes? His answer told me that I was asking the wrong question. The strength of categories is in scoping – modifying behavior for a limited period of time. The primary example he gave was for unit tests. A category can short circuit certain behavior (like retrieving info from a DB or from a file) to make it easier to test certain behavior in isolation. This makes a lot of sense. Mocks/stubs have their place; they are useful when you need to work with an object in a very limited sense – just a few calls. Then there are times when you want a real object doing 95% of what it is supposed to do, but you want to take over in just a few situations to be able to insert custom behavior. e.g. in a unit test when you want to short circuit network or file-based activity or you want to purposefully force an error condition.

It so happens that I really need to improve the unit tests for some network-based components in the ETL system that I work with. The primary one is called GetHttpContent, so you can imagine that unit testing it can be a pain. I’ll give Categories a closer look to see if they can help me out. Unfortunately, I have to wait until we finish shifting to a our new project structure because right now there are so many dependencies in the primary project that if I work on a groovy file in that project in Eclipse I lose ~20M of memory every time I edit it. But the limitations (and benefits) of the Eclipse groovy plugin can wait for another post.