I always love the Groovy Goodness blog. So many gems in such a compact space. He just posted about a dozen tips in the past day alone, all on stuff that I could use in one project or another. The only thing I don’t like about it is it makes me want to go in and update all the places in my code where I could use the new calls right now. Major time suck… must resist… ohhhh… may just one little tweak…
We recently updated our app from Grails 1.1.1 to 1.2.2. (We wanted to move all the way to 1.3.1 but since we build with maven, we finally decided to wait until a new grails-maven plugin is released. See GRAILS-6327.) During the upgrade, we hit two particularly annoying issues related to persistence and setting up associations.
The first involved a belongsTo relation like this:
class AppUser {
UserPrefs prefs = new UserPrefs()
}
class UserPrefs {
static belongsTo = [user: AppUser]
}
This worked under Grails 1.1, but under Grails 1.2 the prefs object was not getting persisted when the user was saved. Other belongsTo relations, both one-to-many and one-to-one worked as expected. We finally discovered that using the short form of the relation notation did work:
static belongsTo = AppUser
This post provided a clue as to where the problem may be. The user property of the Prefs object is not automatically set because the relationship is one-to-one. (At least, it is not automatically set all the time. If we load an AppUser object with Hibernate, it appears that the user property is set in the UserPrefs object. Go figure.) Under Grails 1.1, having a null user was apparently fine. My guess (based on the second error we hit, below) is that any validation errors caused by the child object were not stopping the save of the parent object. Grails 1.2, on the other hand, does care about the child object validation, though I could never get it to report anything in the errors property of either the parent or child object.
In order to work around the issue, I loosened the constraints on UserPrefs a little:
class UserPrefs {
static constraints = {
user(nullable: true)
}
static belongsTo = [user: AppUser]
}
With this change, cascading persistence works. Given, it’s not an ideal solution, but it does let us have access to the user property if needed.
What made this extra confusing is that we have a similar relationship in another set of classes that worked without a problem. The only difference there is that the child class is not automatically created with a static initializer as is done in for the prefs property in AppUser. It is always explicitly set into the owning object. I’ve run out of time to pursue this one further, so I don’t have a final answer. Anyone else out there have more insight into this?
The second persistence issue related to this section of code in the update method of one of our controllers:
def update = {
def hubMap = HubMap.get(params.id);
hubMap.properties = params;
if (hubMap.save()) {
...add success message to response...
} else {
...add failure message to response...
}
...
}
Again, the code worked fine under Grails 1.1.1, but the save call failed under 1.2.2. Unfortunately, there was a hole in both our unit and integration tests for this method, so we didn’t catch it until much later in the release cycle.
Was the JSON conversion in the controller’s get method that generated the original data for the browser different? Nope.
Had the behaviour of the properties meta-method changed? Nope.
The difference is in how Grails handles any existing validation errors on a domain object when you call save(). In our case, the JSON that was being sent to the controller (via a Prototype AJAX call) contained two properties that were references to other objects. The javascript object conversion in the browser was not setting these properties in a meaningful way for the AJAX call; they were both coming across with the string value [object Object]. Since these fields are never updated in this particular workflow, we had never checked what was happening to them. Grails obviously could not convert from the string values to the proper objects, it ignored the values and set two errors on the domain object to record them. However, we didn’t check for errors after the properties call. We went straight to the save call. Under Grails 1.1, deep within the saving sequence in a random class called AbstractDynamicPersistentMethod, you come across this bit of code:
doInvokeInternal(...) {
...
Errors errors = new BeanPropertyBindingResult(target, target.getClass().getName());
mc.setProperty(target, ERRORS_PROPERTY, errors);
...
}
Any existing errors are replaced with a fresh, clean BeanPropertyBindingResult object, wiping out the error information. We never spotted it because we never expected the properties with errors to change anyway. That hole has been closed in Grails 1.2:
doInvokeInternal(...) {
...
Errors errors = setupErrorsProperty(target);
...
}
The new setupErrorsProperty call will copy out existing errors. We put in a few adjustments to not attempt to update those properties and all is well.
So there you have it. A couple of gotchas in the upgrade path for grails. Hope this saves some folks from banging their head against the wall.
The grails maven plugin honors the maven.test.skip option (which turns off compiling and executing tests), but not the skipTests option (which only turns off executing tests). This means that you cannot easily create a package with test classes in it without running all of the tests. I finally come up with a close work around:
mvn clean grails:exec install -Dcommand=test-app -Dargs="-unit DoesNotExist" -Dmaven.test.skip=true
By explicitly listing the test-app command but specifying a test that does not exist, I was able to get the unit tests to be compiled. Since I only needed some common testing files in my package, this works for me. If someone wanted to package both the unit and integration tests, I don’t think this would work. I filed a new issue in the Grails Jira to try to get a real resolution.
Testing, Uncategorized, frustration, grails, maven, unit test
Here’s a bit of maven-voodoo for you. If you have a Grails (1.1.1) app that is built with Maven (2.1) and you try to link in GeoTools (specifically the gt-xsd-core module) and you try to bring in PdfBox as well, you will get really fascinating class loading and linking problems revolving around various XML pieces. I finally found two different ways to get rid of the errors. For those who like to cut to the chase, I’ll list the solutions first and then explain a bit more what errors I was seeing.
If you don’t need commons-jxpath, you can add these three exclusions to the gt-xsd-core dependency declaration.
<dependency>
<groupId>org.geotools.xsd</groupId>
<artifactId>gt-xsd-core</artifactId>
<version>${gt.version}</version>
<exclusions>
<exclusion>
<groupId>commons-jxpath</groupId>
<artifactId>commons-jxpath</artifactId>
</exclusion>
<exclusion>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
</exclusion>
<exclusion>
<groupId>xml-apis</groupId>
<artifactId>xml-apis-xerces</artifactId>
</exclusion>
</exclusions>
</dependency>
This at least works for me given the subset of geotools functionality I am using (primarily encoding and parsing OGC Filter objects to and from XML so I can persist them with Hibernate). It is still pulling in xercesImpl and that appears to be enough, at least to get my unit and integration tests working.
If you need the commons-jxpath functionality in gt-xsd-core, add this to your dependencyManagement section of your POM:
<dependencyManagement>
<dependencies>
<!-- The gt-xsd-core tries to bring commons-jxpath 1.2 in. That messes up the maven junitreport plugin for
some reason (like links to a version of xalan or xml-apis or something. The 1.3 version doesn't have that
issue. -->
<dependency>
<groupId>commons-jxpath</groupId>
<artifactId>commons-jxpath</artifactId>
<version>1.3</version>
</dependency>
</dependencies>
</dependencyManagement>
and then keep the exclusions for the xml-api artifacts as above:
<dependency>
<groupId>org.geotools.xsd</groupId>
<artifactId>gt-xsd-core</artifactId>
<version>${gt.version}</version>
<exclusions>
<exclusion>
<groupId>xml-apis</groupId>
<artifactId>xml-apis</artifactId>
</exclusion>
<exclusion>
<groupId>xml-apis</groupId>
<artifactId>xml-apis-xerces</artifactId>
</exclusion>
</exclusions>
</dependency>
This is what I have setup in my POM right now because I think I might need jxpath soon. I also tried adding a node in dependency management to use the 1.3.03 version of xml-apis rather than 1.0b2 (which is what was pulled in by default), but that doesn’t appear to make any difference. I still had to have the 2 exclusions under gt-xsd-core for things to work.
Now, a little more about the problems I had. I had two types of errors:
Maven junitreport conflict – When I added the dependency for gt-xsd-core and then ran my unit tests using the maven grails plugin (mvn grails:exec -Dcommand=test-app -Dargs="-unit"), my tests would run fine (except for one which I’ll detail below), but then I would get this error when the junitreport plugin tried to run its XSL transform:
Embedded error: java.lang.reflect.InvocationTargetException
Provider org.apache.xalan.processor.TransformerFactoryImpl not found
Normally I would suspect that the geotools dependencies brought in some new version of the xalan library and it didn’t have an old class that the junitreport plugin required. To check this, I compared the output from both maven dependency:classpath and dependency:tree between my trunk branch and the branch where I had added the geotools dependencies. The comparision showed that only one one dependency had changed between the branches; a newer version of commons-pool was used. All other classpath mods were new libraries that were not previously included at all. In fact, xerces and xalan was not included by the trunk branch at all. I am sure that someone with more recent experiences with xml-apis, xerces, and xalan knows exactly what was happening. My guess is that by including the xml-apis-xerces dependency, some piece of the junitreport plugin thought that it could use xalan as opposed to some other XSL lib that it would use when xerces was not present. This is supported by the fact that if I added xalan as a dependency, the error went away. If I have time, I’ll look at the docs for junitreport one of these days and see if that sheds some light on the issue.
PDFImageWriter/NodeList conflict – One of the unit tests exercised our use of PDFBox. If I did not have the above exclusions and I include xalan, this error appeared during the PDFImageWriter constructor:
java.lang.LinkageError: loader constraint violation: loader (instance
of <bootloader>) previously initiated loading for a different type with
name "org/w3c/dom/NodeList"
I haven’t tracked down exactly why this linkage conflict occurred. The bizare thing is that I inserted some link-loading debugging that I used in the past to try to see where NodeList was coming from in both my trunk and geotools-enabled branches:
def clazz = org.w3c.dom.NodeList.class
def codeSource = clazz.getProtectionDomain().getCodeSource();
println ("Source Location: " + codeSource);
In trunk (i.e. no geotools and working fine), it came back as null; NodeList wasn’t loaded at all. In the geotools-enabled branch, it would load (from xerces, I think), but then had that conflict.
At this point, I have a solution (two, actually) and have spend pretty much a whole day researching various paths, so I need to wrap it up and move on. If anyone has a better solution or thoughts as to why exactly there were problems, feel free to comment.
One last note in case there is any confusion from me mentioning xalan so much – I did not end up adding a dependency for xalan to my POM. Just using the exclusion statements listed above solved the issue.
Wrapping up (I hope) my experience with custom JSON conversions in Grails… The day before we were supposed to hit a major release milestone last week, I finally caught sight of a random bug that had been driving us nuts. For some AJAX requests, seemingly at random, the object that has the transient properties was not being rendered with them. Instead, it had only the regular, persistent properties and full representations of its associated objects rather than just the normal stubs of related objects with just the type and id. In other words, it was using deep JSON rendering, and this was overriding the custom JSON ObjectMarshaller that we had set for the class. But why was it doing it seemingly at random? This is where luck and experience combined to make the solution clear rather quickly and avoided some painful slogging through request handling on the server.
The luck part was that I remembered seeing a checkin from a teammate a couple of weeks ago where he got rid of direct use of a grails.converters.deep.JSON object (which is deprecated) and instead switched to a normal grails.converters.JSON object with a use('deep') statement:
def get = {
def o = OurClass.get(params.id);
JSON.use("deep")
render(contentType: "application/json", text: o as JSON)
}
This change was in a different controller for a different domain object though. Why was it affecting our other domain class? That’s where the experience part came in. A few years ago I had to track down why hibernate transactions would fail seemingly at random in another web app when running under JBoss. I could see why an initial transaction would fail (we’d get a timeout and not rollback properly), but there was no reason why later requests would cause failures. That’s when I discovered the downside of request thread pooling. Under most web contains, HTTP requests are handled by a pool of reusable threads. That means changes to thread local variables stick around between requests unless explicitly reset. In the case of the Hibernate bug, since the session was not getting closed properly, it wasn’t getting fully reset and cleared out, so subsequent requests that used that thread were getting the old session and failing because there was no way to get it back to a good state.
For the current JSON conversion bug, the JSON.use method was calling ConvertersConfigurationHolder.setTheadLocalConverterConfiguration(...), thus making the deep converter stick around for that thread. So when future requests came in to any controller on that thread, all JSON conversion calls were made with the deep configuration.
The solution is straightforward. Use the JSON.use(String, Closure) call instead when you want to make a deep call:
def get = {
def o = OurClass.get(params.id);
JSON.use("deep") {
render(contentType: "application/json", text: o as JSON)
}
}
This sets the configuration only for the closure, saving you from thread-based side effects later on.
I figured out why my domain object was rendering as JSON differently in unit tests vs. when running the application and integration tests. Drum roll, please…No grailsApplication object is setup in unit tests.
I hear a great big, “Huh?”
First, a little background on converters. (You may want to get the Grails source code and have directories open to the org.codehaus.groovy.grails.web.converters and grails.converter packages before diving into this.) When a JSON converter is created, it obtains a ConverterConfiguration from the global ConvertersConfigurationHolder. ConverterConfiguration has several properties which influence the conversion process, but the most interesting for this discussion is its prioritized set of ObjectMarshallers. ObjectMarshallers are what do the actual work of turning an object into a JSON representation. Each ObjectMarshaller handles a certain set of classes, like arrays, maps, beans, grails domain objects, etc. When the JSON object is ready to convert a data object, it calls config.getMarshaller(data). The config iterates through its list of marshallers, calling marshaller.supports(data) on each. Since multiple marshallers may be able to handle a specific class (like GroovyBeanMarshaller and DomainClassMarshaller can both handle domain classes), whichever marshaller has highest priority (i.e. is called first) will be invoked.
When the standard ConverterConfiguration is initialized, the DomainClassMarshaller is given higher priority than the GroovyBeanMarshaller. Thus, it handles the conversion of domain objects during normal Grails application execution even though domain objects are also groovy beans. However, the DomainClassMarshaller.supports() method has a slight twist to it. It depends on there being a GrailsApplication object registered in the ConverterUtil class in order to tell it that a particular object is a domain object. Without the GrailsApplication object, ConverterUtil.isDomainClass() always returns false, causing DomainClassMarshaller.supports() to also return false. The ConverterConfiguration next checks the GroovyBeanMarshaller. Its supports() method returns true, so it handles the conversion in this case.
So why is this a big deal? For most domain object, it probably isn’t. Both marshallers render the primary properties of the domain object. However, the DomainClassMarshaller only renders properties return by domainClass.getPersistentProperties() while GroovyBeanMarshaller renders all properties, including transient ones. Because I needed some transient properties in the JSON representation, my unit tests worked great (since there is no grailsApplication setup, so the GroovyBeanMarshaller did the full rendering), but then my code failed when I ran the integration tests and the real app (since the DomainClassMarshaller handled the rendering).
How do you fix this? It’s pretty easy once you know where to look. First, I wanted to make my unit tests behave like the real app, so I had to get them to fail. Unfortunately, the conversion classes are written in java, not groovy, so you can’t just do a quick override of the ConverterUtil metaclass to get it to return what you want. You actually need to setup a GrailsApplication object. Rather than trying to create and initialize a whole one though, I determined that I could create a small stub class that overrode the two methods I needed:
import org.codehaus.groovy.grails.commons.*
class GrailsApplicationStub extends DefaultGrailsApplication {
def artefacts = [(DomainClassArtefactHandler.TYPE):[:]]
boolean isArtefactOfType(String artefactType, String className) {
return getArtefact(artefactType, className) != null
}
GrailsClass getArtefact(String artefactType, String name) {
def retVal = artefacts[artefactType] ? artefacts[artefactType][name] : null
return retVal
}
}
When initializing the unit test, I created the stub and set in the particular domain class that I wanted to be recognized. Then register the application stub with the ConverterUtil class:
def grailsApp = new GrailsApplicationStub();
grailsApp.artefacts[DomainClassArtefactHandler.TYPE][YourClass.name] = new DefaultGrailsDomainClass(YourClass);
ConverterUtil.setGrailsApplication(grailsApp);
Instances of YourClass will now be handled by the DomainClassMarshaller in the unit test. When the unit test checks for the existence of a transient property in a JSON representation, it should fail.
Next, I needed to change the marshaller prioritization so that the desired domain class was handled by a regular GroovyBeanMarshaller. Since I always wanted the GroovyBeanMarshaller to handle the class, I inserted this code into my unit test setup (and later into BootStrap so the behavior would apply to the full app):
JSON.registerObjectMarshaller(YourClass, {o, c ->
new GroovyBeanMarshaller().marshalObject(o, c)
})
This creates a ClosureObjectMarshaller that handles YourClass objects and gives the new marshaller the default priority of 1 (which puts it at the top of the priority list). When processing an object, this closure marshaller simply passes it through to a GroovyBeanMarshaller. Since the new marshaller only handles YourClass objects, the rendering behavior for all other domain objects is not affected.
This resolved my original issue. The JSON rendering of my object is now correct when I run my app. I did go down a side path while coming to this final solution. If you only want to override how a YourClass object is rendered some of the time, you could instead register a named configuration for it like this:
JSON.createNamedConfig("FullYourClassRender") {cfg ->
cfg.registerObjectMarshaller(new ClosureOjectMarshaller
new GroovyBeanMarshaller().marshalObject(o, c)
}))
}
Then when you want to use that particular configuration, you wrap the JSON object calls in a use statement:
JSON.use("FullYourClassRender") {
render(contentType: "application/json", text: myObject as JSON)
}
Other configuration options are also available. Take a look at the JSON class to see them.
I hope this helps document the conversion process a bit.
I discovered a problem this afternoon with how one of the domain objects in my Grails app was being converted to JSON in its controller. The conversion functioned as expected in a unit test, but failed in integration tests and when running the real app. “Time to see how good the debugger in Intellij 9 is,” I say to myself. So I dive into the grails.converters.JSON class to try to find where the difference comes from. Unfortunately, every time I step into the render() method, the rendering quickly fails with this exception:
org.codehaus.groovy.grails.web.json.JSONException: “Misplaced object”
It turns out that the JSON converter has a writer property that is a JSONWriter object. The act of showing the writer in the variables window of the Intellij debugger causes its toString() method to be called by the IDE (though any breakpoints set in the class do not get triggered by the call). This, in turn, causes the writer to actually do the writing of the object which, when complete, changes the writer’s state property to DONE. When the real code execution tries to process the data, the JSONWriter throws the exception because it thinks it has already finished rendering.
The trick to step through the JSON converter is to define a custom Type Renderer for the JSONWriter class that does not invoke the “toString” method. A blog post on the Intellij site defines how to setup a Type Renderer for a class. I set mine up so that the node renders simply as “JSONWriter” since I don’t need to see the details for it. One caveat though – I found that the new Type Renderer didn’t affect entries already in my variables window until I right-clicked one and selected “View As -> JSONWriter”. Then the view changed to use the new display settings.
Now back to debugging…maybe I’ll actually figure out the real problem sometime before the sun comes up.
I love Groovy’s meta-programming capabilities. Duck typing, DSLs, ExpandoMetaClass… it’s all great until you blow your arm off with them and spend two hours picking the pieces back up. Such was my experience today as I was trying to set up my first unit test for a Grails controller. I needed to pass in some parameters. As I just wrote in my last post, the grails user guide has information on integration testing controllers, but nothing more than a brief mention of the ControllerUnitTestCase. Unfortunately, searching for phrases like “grails unit test controller” on Google turn up posts from last Fall, before the ControllerUnitTestCase was released with Grails 1.1. Don’t ask me why I didn’t just search for ControllerUnitTestCase from the start. 20/20 hindsight.
Working with what I could find, I came across a blog post from March 2008 that covered integration testing with controllers, much like the grails User Guide. It at least had some explicit examples of how to pass in parameters, so I hoped the approach would work with my ControllerUnitTestCase implementation. Using the code snippets in the post as a guide, I setup my test something like this:
controller.request.params = [layoutType:'filmstrip',compKey:'testKey']
controller.create()
This looks good, right? The test scaffolding connects the injected params property of the controller with the params property of the request. Should work like a charm.
And so started my descent into WTF Land.
The tests compiled and ran fine, except that no params were getting passed through to the controller. I assumed that the old documentation for integration testing no longer exactly applied for the new unit test setups. Eventually I dug through enough of ControllerUnitTestCase, MvcUnitTestCase, and MockUtils that I figured out there was a way to get direct access to the params property of the controller. “Makes sense,” I thought, “sort of a mini-mock. Who needs the request for a unit test?” Great, so let’s try this:
controller.params = [layoutType:'filmstrip',compKey:'testKey']
controller.create()
Run again and…
groovy.lang.ReadOnlyPropertyException: Cannot set readonly property: params for class: YourController
Erk. Closer inspection of grails.test.MockUtils.addCommonWebProperties() reveals how the params property is added to the controller class with only a get method. Okay, so we can still manipulate the map once we get it. One more try:
controller.params.putAll([layoutType:"filmstrip",compKey:'testKey'])
controller.create()
Success! Finally, information is going where it needs to be. I later realized that section 9.2 of the user guide does actually show an example similar to this, except that it sets each item on params individually:
def controller = new AuthenticationController()
controller.params.login = "marcpalmer"
controller.params.password = "secret"
controller.params.passwordConfirm = "secret"
controller.signup()
“Oh well,” I say to myself, “at least I got it working. I’d better post a comment on that old blog post so that others know the new way to set parameters.” I returned to the blog page, moved down to the comments section… and saw that someone else had posted a comment just last month thanking the author for the good information. Wait a second… you normally don’t thank someone if their examples don’t work. So I scrolled up and took a closer look at the examples again:
controller.request.parameters = [sender:'me',msg:'test']
controller.create()
Anyone with a better eye for detail than me see the difference that sent me down the rat hole? It’s request.parameters, not request.params as I had entered. For the 100th time today, Doh!
But wait a second (or an hour, as the case may be)… If I was calling request.params and there is no params property, how did my test run at all? Dig, dig, dig… It turns out that the request is a GrailsMockHttpServletRequest object. Scanning through that class, I see the ever so powerful and ever so dangerous setProperty(String name, value) override. If you try to set a property that doesn’t exist, it adds the value as an entry in the request’s attributes map. So the test scaffolding was perfectly happy with my request.params call. It nicely tucked my params map away in the attributes and continued on its way.
Sigh…it almost makes me miss compile-time method checking… but not quite.
My team started on our first Grails app back in mid-August. I set out to provide good examples of using the new testing features in Grails 1.1 to encourage the rest of the team. Domain class testing went great, as did Service testing. Unfortunately, there wasn’t any documentation on the ControllerUnitTestCase class in the Grails User Guide, and I couldn’t find any blog or forum posts to help out. I tried creating an instance of it and tossing in a call to mockController(), but I didn’t get very far. I had to shift focus to other tasks before I could dive into the source code itself and figure out what it did, so progress stopped for a while.
Fast forward to today when I finally had a strong need for a controller unit test setup. I dove into the code and realized three things very quickly:
Doh! I wish I had taken 15 minutes back in August to look at the source. It would have saved quite a bit of time by encouraging us to get proper controller test cases in place faster.
To add insult to injury, after spending that time and more getting some test cases just right, I decided to try a search for more documentation online again. This time there was an answer to my query for help. A post to the grails-user mailing list less than a month ago generated several responses with useful info, including Scott Davis posting about the most recent addition to his Mastering Grails series which was published just days before the post.
So nine months after the release of Grails 1.1 there is now good documentation on this great piece. Many thanks to those who pulled it together.
I am sitting in the opening session of the International Semantic Web Conference (ISWC 2009) right now. The conference chairs normally give stats about where submitted papers are from, primary topics, etc. This time they decided to have some fun. They did a full text extraction on all the papers, removing stop words and names and then running a Porter stemmer on the text. They then came up with the list of words most used in accepted papers and those most used in rejected papers. Finally, they graphed the words by the largest difference in occurrence between the accepted and rejected papers. So if you want to get a paper accepted at a semantic web conference, don’t write about Europeans, users, or people in general. Do write about services and arguments (or, at least, words with the stems ’servic’ and ‘argu’). At least, that’s the rule this year.
Who says theoreticians don’t know how to have fun?