JSON conversion and request thread reuse

December 18th, 2009

Wrapping up (I hope) my experience with custom JSON conversions in Grails… The day before we were supposed to hit a major release milestone last week, I finally caught sight of a random bug that had been driving us nuts. For some AJAX requests, seemingly at random, the object that has the transient properties was not being rendered with them. Instead, it had only the regular, persistent properties and full representations of its associated objects rather than just the normal stubs of related objects with just the type and id. In other words, it was using deep JSON rendering, and this was overriding the custom JSON ObjectMarshaller that we had set for the class. But why was it doing it seemingly at random? This is where luck and experience combined to make the solution clear rather quickly and avoided some painful slogging through request handling on the server.

The luck part was that I remembered seeing a checkin from a teammate a couple of weeks ago where he got rid of direct use of a grails.converters.deep.JSON object (which is deprecated) and instead switched to a normal grails.converters.JSON object with a use('deep') statement:

def get = {
def o = OurClass.get(params.id);
JSON.use("deep")
render(contentType: "application/json", text: o as JSON)
}

This change was in a different controller for a different domain object though. Why was it affecting our other domain class? That’s where the experience part came in. A few years ago I had to track down why hibernate transactions would fail seemingly at random in another web app when running under JBoss. I could see why an initial transaction would fail (we’d get a timeout and not rollback properly), but there was no reason why later requests would cause failures. That’s when I discovered the downside of request thread pooling. Under most web contains, HTTP requests are handled by a pool of reusable threads. That means changes to thread local variables stick around between requests unless explicitly reset. In the case of the Hibernate bug, since the session was not getting closed properly, it wasn’t getting fully reset and cleared out, so subsequent requests that used that thread were getting the old session and failing because there was no way to get it back to a good state.

For the current JSON conversion bug, the JSON.use method was calling ConvertersConfigurationHolder.setTheadLocalConverterConfiguration(...), thus making the deep converter stick around for that thread. So when future requests came in to any controller on that thread, all JSON conversion calls were made with the deep configuration.

The solution is straightforward. Use the JSON.use(String, Closure) call instead when you want to make a deep call:

def get = {
def o = OurClass.get(params.id);
JSON.use("deep") {
render(contentType: "application/json", text: o as JSON)
}
}

This sets the configuration only for the closure, saving you from thread-based side effects later on.

eric Uncategorized , , ,

Grails JSON converter and transient properties

November 23rd, 2009

I figured out why my domain object was rendering as JSON differently in unit tests vs. when running the application and integration tests. Drum roll, please…No grailsApplication object is setup in unit tests.

I hear a great big, “Huh?”

First, a little background on converters. (You may want to get the Grails source code and have directories open to the org.codehaus.groovy.grails.web.converters and grails.converter packages before diving into this.) When a JSON converter is created, it obtains a ConverterConfiguration from the global ConvertersConfigurationHolder. ConverterConfiguration has several properties which influence the conversion process, but the most interesting for this discussion is its prioritized set of ObjectMarshallers. ObjectMarshallers are what do the actual work of turning an object into a JSON representation. Each ObjectMarshaller handles a certain set of classes, like arrays, maps, beans, grails domain objects, etc. When the JSON object is ready to convert a data object, it calls config.getMarshaller(data). The config iterates through its list of marshallers, calling marshaller.supports(data) on each. Since multiple marshallers may be able to handle a specific class (like GroovyBeanMarshaller and DomainClassMarshaller can both handle domain classes), whichever marshaller has highest priority (i.e. is called first) will be invoked.

When the standard ConverterConfiguration is initialized, the DomainClassMarshaller is given higher priority than the GroovyBeanMarshaller. Thus, it handles the conversion of domain objects during normal Grails application execution even though domain objects are also groovy beans. However, the DomainClassMarshaller.supports() method has a slight twist to it. It depends on there being a GrailsApplication object registered in the ConverterUtil class in order to tell it that a particular object is a domain object. Without the GrailsApplication object, ConverterUtil.isDomainClass() always returns false, causing DomainClassMarshaller.supports() to also return false. The ConverterConfiguration next checks the GroovyBeanMarshaller. Its supports() method returns true, so it handles the conversion in this case.

So why is this a big deal? For most domain object, it probably isn’t. Both marshallers render the primary properties of the domain object. However, the DomainClassMarshaller only renders properties return by domainClass.getPersistentProperties() while GroovyBeanMarshaller renders all properties, including transient ones. Because I needed some transient properties in the JSON representation, my unit tests worked great (since there is no grailsApplication setup, so the GroovyBeanMarshaller did the full rendering), but then my code failed when I ran the integration tests and the real app (since the DomainClassMarshaller handled the rendering).

How do you fix this? It’s pretty easy once you know where to look. First, I wanted to make my unit tests behave like the real app, so I had to get them to fail. Unfortunately, the conversion classes are written in java, not groovy, so you can’t just do a quick override of the ConverterUtil metaclass to get it to return what you want. You actually need to setup a GrailsApplication object. Rather than trying to create and initialize a whole one though, I determined that I could create a small stub class that overrode the two methods I needed:


import org.codehaus.groovy.grails.commons.*
class GrailsApplicationStub extends DefaultGrailsApplication {
def artefacts = [(DomainClassArtefactHandler.TYPE):[:]]

boolean isArtefactOfType(String artefactType, String className) {
return getArtefact(artefactType, className) != null
}

GrailsClass getArtefact(String artefactType, String name) {
def retVal = artefacts[artefactType] ? artefacts[artefactType][name] : null
return retVal
}
}

When initializing the unit test, I created the stub and set in the particular domain class that I wanted to be recognized. Then register the application stub with the ConverterUtil class:


def grailsApp = new GrailsApplicationStub();
grailsApp.artefacts[DomainClassArtefactHandler.TYPE][YourClass.name] = new DefaultGrailsDomainClass(YourClass);
ConverterUtil.setGrailsApplication(grailsApp);

Instances of YourClass will now be handled by the DomainClassMarshaller in the unit test. When the unit test checks for the existence of a transient property in a JSON representation, it should fail.

Next, I needed to change the marshaller prioritization so that the desired domain class was handled by a regular GroovyBeanMarshaller. Since I always wanted the GroovyBeanMarshaller to handle the class, I inserted this code into my unit test setup (and later into BootStrap so the behavior would apply to the full app):


JSON.registerObjectMarshaller(YourClass, {o, c ->
new GroovyBeanMarshaller().marshalObject(o, c)
})

This creates a ClosureObjectMarshaller that handles YourClass objects and gives the new marshaller the default priority of 1 (which puts it at the top of the priority list). When processing an object, this closure marshaller simply passes it through to a GroovyBeanMarshaller. Since the new marshaller only handles YourClass objects, the rendering behavior for all other domain objects is not affected.

This resolved my original issue. The JSON rendering of my object is now correct when I run my app. I did go down a side path while coming to this final solution. If you only want to override how a YourClass object is rendered some of the time, you could instead register a named configuration for it like this:


JSON.createNamedConfig("FullYourClassRender") {cfg ->
cfg.registerObjectMarshaller(new ClosureOjectMarshaller(YourClass, {o, c ->
new GroovyBeanMarshaller().marshalObject(o, c)
}))
}

Then when you want to use that particular configuration, you wrap the JSON object calls in a use statement:


JSON.use("FullYourClassRender") {
render(contentType: "application/json", text: myObject as JSON)
}

Other configuration options are also available. Take a look at the JSON class to see them.

I hope this helps document the conversion process a bit.

eric Uncategorized , ,

Fun with Grails JSON conversion

November 20th, 2009

I discovered a problem this afternoon with how one of the domain objects in my Grails app was being converted to JSON in its controller.  The conversion functioned as expected in a unit test, but failed in integration tests and when running the real app.  “Time to see how good the debugger in Intellij 9 is,” I say to myself.  So I dive into the grails.converters.JSON class to try to find where the difference comes from.  Unfortunately, every time I step into the render() method, the rendering quickly fails with this exception:

org.codehaus.groovy.grails.web.json.JSONException: “Misplaced object”

It turns out that the JSON converter has a writer property that is a JSONWriter object.  The act of showing the writer in the variables window of the Intellij debugger causes its toString() method to be called by the IDE (though any breakpoints set in the class do not get triggered by the call).  This, in turn, causes the writer to actually do the writing of the object which, when complete, changes the writer’s state property to DONE.  When the real code execution tries to process the data, the JSONWriter throws the exception because it thinks it has already finished rendering.

The trick to step through the JSON converter is to define a custom Type Renderer for the JSONWriter class that does not invoke the “toString” method.  A blog post on the Intellij site defines how to setup a Type Renderer for a class.  I set mine up so that the node renders simply as “JSONWriter” since I don’t need to see the details for it.  One caveat though – I found that the new Type Renderer didn’t affect entries already in my variables window until I right-clicked one and selected “View As -> JSONWriter”.  Then the view changed to use the new display settings.

Now back to debugging…maybe I’ll actually figure out the real problem sometime before the sun comes up.

eric Uncategorized , ,

When dynamic programming backfires

November 20th, 2009

I love Groovy’s meta-programming capabilities.  Duck typing, DSLs, ExpandoMetaClass… it’s all great until you blow your arm off with them and spend two hours picking the pieces back up.  Such was my experience today as I was trying to set up my first unit test for a Grails controller.  I needed to pass in some parameters.  As I just wrote in my last post, the grails user guide has information on integration testing controllers, but nothing more than a brief mention of the ControllerUnitTestCase.  Unfortunately, searching for phrases like “grails unit test controller” on Google turn up posts from last Fall, before the ControllerUnitTestCase was released with Grails 1.1.  Don’t ask me why I didn’t just search for ControllerUnitTestCase from the start.  20/20 hindsight.

Working with what I could find, I came across a blog post from March 2008 that covered integration testing with controllers, much like the grails User Guide.  It at least had some explicit examples of how to pass in parameters, so I hoped the approach would work with my ControllerUnitTestCase implementation.  Using the code snippets in the post as a guide, I setup my test something like this:

controller.request.params = [layoutType:'filmstrip',compKey:'testKey']
controller.create()

This looks good, right?  The test scaffolding connects the injected params property of the controller with the params property of the request.  Should work like a charm.

And so started my descent into WTF Land.

The tests compiled and ran fine, except that no params were getting passed through to the controller.  I assumed that the old documentation for integration testing no longer exactly applied for the new unit test setups.  Eventually I dug through enough of ControllerUnitTestCase, MvcUnitTestCase, and MockUtils that I figured out there was a way to get direct access to the params property of the controller.  “Makes sense,” I thought, “sort of a mini-mock.  Who needs the request for a unit test?”  Great, so let’s try this:

controller.params = [layoutType:'filmstrip',compKey:'testKey']
controller.create()

Run again and…

groovy.lang.ReadOnlyPropertyException: Cannot set readonly property: params for class: YourController

Erk.  Closer inspection of grails.test.MockUtils.addCommonWebProperties() reveals how the params property is added to the controller class with only a get method.  Okay, so we can still manipulate the map once we get it.  One more try:

controller.params.putAll([layoutType:"filmstrip",compKey:'testKey'])
controller.create()

Success!  Finally, information is going where it needs to be.  I later realized that section 9.2 of the user guide does actually show an example similar to this, except that it sets each item on params individually:

def controller = new AuthenticationController()
controller.params.login = "marcpalmer"
controller.params.password = "secret"
controller.params.passwordConfirm = "secret"
controller.signup()

“Oh well,” I say to myself, “at least I got it working.  I’d better post a comment on that old blog post so that others know the new way to set parameters.”  I returned to the blog page, moved down to the comments section… and saw that someone else had posted a comment just last month thanking the author for the good information.  Wait a second… you normally don’t thank someone if their examples don’t work.  So I scrolled up and took a closer look at the examples again:

controller.request.parameters = [sender:'me',msg:'test']
controller.create()

Anyone with a better eye for detail than me see the difference that sent me down the rat hole? It’s request.parameters, not request.params as I had entered.  For the 100th time today, Doh!

But wait a second (or an hour, as the case may be)… If I was calling request.params and there is no params property, how did my test run at all?  Dig, dig, dig… It turns out that the request is a GrailsMockHttpServletRequest object.  Scanning through that class, I see the ever so powerful and ever so dangerous setProperty(String name, value) override.  If you try to set a property that doesn’t exist, it adds the value as an entry in the request’s attributes map.  So the test scaffolding was perfectly happy with my request.params call.  It nicely tucked my params map away in the attributes and continued on its way.

Sigh…it almost makes me miss compile-time method checking… but not quite.

eric Uncategorized , ,

ControllerUnitTestCase: Documentation Has Arrived

November 19th, 2009

My team started on our first Grails app back in mid-August.  I set out to provide good examples of using the new testing features in Grails 1.1 to encourage the rest of the team.  Domain class testing went great, as did Service testing.  Unfortunately, there wasn’t any documentation on the ControllerUnitTestCase class in the Grails User Guide, and I couldn’t find any blog or forum posts to help out.  I tried creating an instance of it and tossing in a call to mockController(), but I didn’t get very far.  I had to shift focus to other tasks before I could dive into the source code itself and figure out what it did, so progress stopped for a while.

Fast forward to today when I finally had a strong need for a controller unit test setup.  I dove into the code and realized three things very quickly:

  1. You don’t need to call mockController().  It does that for you based on the name of the test class itself.
  2. You don’t want to call the constructor of your controller class in the test cases.  Use the newInstance() method instead so that all the proper scaffolding is put in place.
  3. You don’t even have to call newInstance() most of the time because the setUp() method inits a controller variable for you already.

Doh!  I wish I had taken 15 minutes back in August to look at the source.  It would have saved quite a bit of time by encouraging us to get proper controller test cases in place faster.

To add insult to injury, after spending that time and more getting some test cases just right, I decided to try a search for more documentation online again.   This time there was an answer to my query for help.  A post to the grails-user mailing list less than a month ago generated several responses with useful info, including Scott Davis posting about the most recent addition to his Mastering Grails series which was published just days before the post.

So nine months after the release of Grails 1.1 there is now good documentation on this great piece.  Many thanks to those who pulled it together.

eric Uncategorized , ,

Fun at the ISWC

October 27th, 2009

I am sitting in the opening session of the International Semantic Web Conference (ISWC 2009) right now.  The conference chairs normally give stats about where submitted papers are from, primary topics, etc.  This time they decided to have some fun.  They did a full text extraction on all the papers, removing stop words and names and then running a Porter stemmer on the text.  They then came up with the list of words most used in accepted papers and those most used in rejected papers. Finally, they graphed the words by the largest difference in occurrence between the accepted and rejected papers.  So if you want to get a paper accepted at a semantic web conference, don’t write about Europeans, users, or people in general.  Do write about services and arguments (or, at least, words with the stems ’servic’ and ‘argu’).  At least, that’s the rule this year.

Who says theoreticians don’t know how to have fun?

eric Uncategorized ,

Cruising on other people’s work

September 11th, 2009

I love it when other people do my work for me.  For example, I needed to store a few random settings in a Grails app, such as the last time a service polled an external resource.  So I went to the main Grails Plugin site and, sure enough, someone else has already written a Grails Settings plugin.  It needs a few enhancements to do what I want, but it’s better than starting from scratch.  While I was perusing the plugin list, I found a few others that can replace features I had started to sketch out (like Taggable) and was inspired by yet others that would let me easily add features that I hadn’t thought of before (like Commentable and Sparkline).

Another example is yet another good post by my friend and coworker Jeff Erikson detailing his research into problems with Grails config options getting wiped during integration tests.  I had been meaning to write up a note about how the GrailsUnitTestCase class hides the true config settings, so you don’t want to accidentally use it in integration tests.  Jeff took my little tip a lot farther with his post.  Thanks, Jeff!

eric grails, groovy, unit test

Corrections to Grails Documentation

September 5th, 2009

Here are two quick tips for anyone working with Grails 1.1.1:

1) The User Guide has conflicting info about cascading persistence.   Section 5.2.1.2 says “The default cascading behaviour is to cascade saves and updates, but not deletes unless a belongsTo is also specified.” Section 5.3.3 says, “If you do not define belongsTo then no cascades will happen and you will have to manually save each object.”

The former, 5.2.1.2, is correct. Create and Update actions are cascaded for one-to-many relations. However, hey are not cascaded for one-to-one relations.

2) Section 9.1 (Unit Testing) demonstrates how to pass in a list to the mockDomain(clazz, instanceList) call.  It then says that you can check that list to verify that persistence actions behaved as expected.  This is no longer correct.  The list is copied inside the mockDomain call.  To get access to the list of persisted objects for a class, use the MockUtils.TEST_INSTANCES map with the class name as the key.  For example:

def testInstances = []
mockDomain(Item, testInstances)
def i = new Item(name:"foo").save()
assertEquals 0, testInstances.size()
testInstances = MockUtils.TEST_INSTANCES[Item]
assertEquals 1, testInstances.size()

And in case you try the same thing as me, the mocking behavior does not support cascades.

eric Testing, frustration, grails , , ,

Hello world!

July 30th, 2009

Welcome to my new site.  I’ve had this domain for a few years and thought it was time to activate it.  Everything before this post was imported from my old blogger account, points-and-edges.blogspot.com.

eric Uncategorized

Importing an existing grails projects into Intellij

July 30th, 2009

Yesterday I had the fun experience of getting an existing grails project configured in Intellij. The process required just a few more steps than given in the Intellij documentation, so I added my notes to an existing thread in the Intellij forums on the same topic. I hope it saves someone else from denting the wall with their head.

eric Intellij, grails, import, pain