Archive

Archive for the ‘GIS’ Category

Grails+Maven+GeoTools+PdfBox = PITA

March 20th, 2010

Here’s a bit of maven-voodoo for you. If you have a Grails (1.1.1) app that is built with Maven (2.1) and you try to link in GeoTools (specifically the gt-xsd-core module) and you try to bring in PdfBox as well, you will get really fascinating class loading and linking problems revolving around various XML pieces. I finally found two different ways to get rid of the errors. For those who like to cut to the chase, I’ll list the solutions first and then explain a bit more what errors I was seeing.

Solution #1: Exclude, exclude, exclude

If you don’t need commons-jxpath, you can add these three exclusions to the gt-xsd-core dependency declaration.

    <dependency>
        <groupId>org.geotools.xsd</groupId>
        <artifactId>gt-xsd-core</artifactId>
        <version>${gt.version}</version>
        <exclusions>
            <exclusion>
                <groupId>commons-jxpath</groupId>
                <artifactId>commons-jxpath</artifactId>
            </exclusion>
            <exclusion>
                <groupId>xml-apis</groupId>
                <artifactId>xml-apis</artifactId>
            </exclusion>
            <exclusion>
                <groupId>xml-apis</groupId>
                <artifactId>xml-apis-xerces</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

This at least works for me given the subset of geotools functionality I am using (primarily encoding and parsing OGC Filter objects to and from XML so I can persist them with Hibernate). It is still pulling in xercesImpl and that appears to be enough, at least to get my unit and integration tests working.

Solution #2: Using Dependency Management

If you need the commons-jxpath functionality in gt-xsd-core, add this to your dependencyManagement section of your POM:

    <dependencyManagement>
        <dependencies>
            <!-- The gt-xsd-core tries to bring commons-jxpath 1.2 in.  That messes up the maven junitreport plugin for
            some reason (like links to a version of xalan or xml-apis or something.  The 1.3 version doesn't have that
            issue. -->
            <dependency>
                <groupId>commons-jxpath</groupId>
                <artifactId>commons-jxpath</artifactId>
                <version>1.3</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

and then keep the exclusions for the xml-api artifacts as above:

    <dependency>
        <groupId>org.geotools.xsd</groupId>
        <artifactId>gt-xsd-core</artifactId>
        <version>${gt.version}</version>
        <exclusions>
            <exclusion>
                <groupId>xml-apis</groupId>
                <artifactId>xml-apis</artifactId>
            </exclusion>
            <exclusion>
                <groupId>xml-apis</groupId>
                <artifactId>xml-apis-xerces</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

This is what I have setup in my POM right now because I think I might need jxpath soon. I also tried adding a node in dependency management to use the 1.3.03 version of xml-apis rather than 1.0b2 (which is what was pulled in by default), but that doesn’t appear to make any difference. I still had to have the 2 exclusions under gt-xsd-core for things to work.

So what was the problem?

Now, a little more about the problems I had. I had two types of errors:

Maven junitreport conflict – When I added the dependency for gt-xsd-core and then ran my unit tests using the maven grails plugin (mvn grails:exec -Dcommand=test-app -Dargs="-unit"), my tests would run fine (except for one which I’ll detail below), but then I would get this error when the junitreport plugin tried to run its XSL transform:

Embedded error: java.lang.reflect.InvocationTargetException
Provider org.apache.xalan.processor.TransformerFactoryImpl not found

Normally I would suspect that the geotools dependencies brought in some new version of the xalan library and it didn’t have an old class that the junitreport plugin required. To check this, I compared the output from both maven dependency:classpath and dependency:tree between my trunk branch and the branch where I had added the geotools dependencies. The comparision showed that only one one dependency had changed between the branches; a newer version of commons-pool was used. All other classpath mods were new libraries that were not previously included at all. In fact, xerces and xalan was not included by the trunk branch at all. I am sure that someone with more recent experiences with xml-apis, xerces, and xalan knows exactly what was happening. My guess is that by including the xml-apis-xerces dependency, some piece of the junitreport plugin thought that it could use xalan as opposed to some other XSL lib that it would use when xerces was not present. This is supported by the fact that if I added xalan as a dependency, the error went away. If I have time, I’ll look at the docs for junitreport one of these days and see if that sheds some light on the issue.

PDFImageWriter/NodeList conflict – One of the unit tests exercised our use of PDFBox. If I did not have the above exclusions and I include xalan, this error appeared during the PDFImageWriter constructor:

java.lang.LinkageError: loader constraint violation: loader (instance
of <bootloader>) previously initiated loading for a different type with
name "org/w3c/dom/NodeList"

I haven’t tracked down exactly why this linkage conflict occurred. The bizare thing is that I inserted some link-loading debugging that I used in the past to try to see where NodeList was coming from in both my trunk and geotools-enabled branches:

def clazz = org.w3c.dom.NodeList.class
def codeSource = clazz.getProtectionDomain().getCodeSource();
println ("Source Location: " + codeSource);

In trunk (i.e. no geotools and working fine), it came back as null; NodeList wasn’t loaded at all. In the geotools-enabled branch, it would load (from xerces, I think), but then had that conflict.

At this point, I have a solution (two, actually) and have spend pretty much a whole day researching various paths, so I need to wrap it up and move on. If anyone has a better solution or thoughts as to why exactly there were problems, feel free to comment.

One last note in case there is any confusion from me mentioning xalan so much – I did not end up adding a dependency for xalan to my POM. Just using the exclusion statements listed above solved the issue.

GIS, Uncategorized, frustration, grails , , , ,

Newbie Notes for GIS Web Development

July 7th, 2009

I have spent some time over the past couple of weeks getting to know the Open Source GIS arena, from spatial databases (PostGIS) to server software (GeoServer) to web display techs (OpenLayers). When I started, I decided to map my trail as I explored this space since I will have co-workers following along behind me as new projects in our group ramp up. Hopefully, these breadcrumbs will help other newbies in this area as well.

begin tangent
Have you ever felt that sometimes the worst people to write documentation for something are actually those that know the most about it? Once you are an expert in something, it’s really hard to consciously remember all the questions that came up and problems you had to solve and research you had to do to reach nirvana. That’s why with this project I tried to jot down questions that I had as I went, and kept the question list even after I figured out the answers. It helped me not forget what I didn’t know. I’ve used this technique before, and strongly encourage new hires who come onto my projects to do the same so we can fill in the holes in our developer documentation. Hmmm… sounds like another blog entry in there…
end tangent

This first entry is about the “what’s what” GIS and “who’s who” in the Open Source area. It is not intended to provide all the information you need to work on a GIS application. It is intended to tell you where to get that information. It also explains a few questions and misconceptions I had as I started down this path.

What is GIS?

First, to get your head around the GIS terms and concepts, read this short overview on GIS concepts from developerworks. One thing the article does is define what a layer is and relate that to the term feature. What it leaves out is the term FeatureType. A FeatureType defines a type of data, listing the attributes that go along with it, such as name, shape, and other meta-data (e.g. population for cities, road type for a set roads (highway, secondary, dirt, etc)). The layer concept is a way of visualizing a bunch of features of the same FeatureType. For example, in GoogleEarth, you don’t think about turning on and off the road FeatureType or the city FeatureType. You think about showing/hiding those layers. Some software (like GeoServer) uses the term FeatureType in places where we more naturally think in layers. Just get used to moving back and forth between the two terms.

What’s a Map?

Something that is missing from all the literature is a strict definition of what is a map. Tutorials describe layers, FeatureTypes,and features. Standards define ways of storing geographic information, retrieving that information, applying styles to it, and rendering it. But I couldn’t find anything that strictly defines what a map is. What this means is that each application or library has it’s own concept for what comprises a map, or it may not have a unified concept of a map at all. OpenLayers (the most popular way of rendering maps in web pages) is based around the concept of a map. It’s map has layers of data that can be shown/hidden independently. The map also has tools for zooming, paging, measuring, highlighting, and possibly even editing the data. GeoServer (an Open Source map server), on the other hand, is more focused on individual data sets (i.e. independent layers or FeatureTypes). How those sets of data are combined into a single, visual display that we would call a map is up to the consumer of the data (such as a web app using OpenLayers or a desktop app like uDig).

If you are looking for the least common denominator for the concept of a map, think of a display of layered geospatial information with one or more “base layers” comprised of static (or nearly so) data (e.g. geographic features, political boundaries, rivers, roads, cities, etc) and zero or more “live layers” comprised of data that can change with relatively high frequency (e.g. weather images, traffic patterns, earthquake epicenters, recent Elvis sightings, etc).

Who’s Who

To learn about the Open Source standards and tools in the GIS space, flip through Scott Davis’ GIS for Web Developers presentation while you listen to his GIS podcast. He has a lot of other great content on his mapmap site. If you like his presentation style, pick up his GIS for web developers book. It comes in a PDF format for instant gratification.

Now that you’ve been exposed to some of the concepts and heard mention of the major players and most popular apps, you can read what Wikipedia has to say about them. As usual, the Wikipedia pages have links to the organizations’ sites as well as important related concepts.

  • Open Geospatial Consortium (OGC) – the first thing to know about the OGC is that they publish the WMS and WFS standards. They have many other standards as well, but those two are the primary protocols by which you will get data from a map server (like GeoServer) to a UI (like OpenLayers).
  • ESRI ArcGIS – ESRI is the 800 lbs. gorilla in the GIS space. It is sort of like the Oracle of GIS. It has it’s own commercial, proprietary software suite called ArcGIS. If you have heard of Shape files, this is the company that invented that format. It’s not open source, but it’s good to know who they are. In my situation, I have existing systems that feed into ArcGIS layers, so I have to work with it as well as with other data via the OGC standards.
  • GeoServer – Highly extensible, open source WMS/WFS server. A good application to keep in mind if you want to run an application that is a single source for both “base layer” data and your app-specific data. Something that was critical for my project is that it can pull data from an ESRI ArcGIS server as well as other sources like PostGIS or raw images. It’s online user manual contains some good sections on basic concepts for serving and formating geo data over HTTP, including Styled Layer Descriptor (SLD), WMS, and WFS.
  • PostGIS – Geo-spatial extensions to PostgreSQL. This is the most popular (and powerful) OpenSource geo-enabled DB. MySql also has geo-extensions, as does MS SqlServer and Oracle (called Oracle Spatial). BostonGis has a great tutorial on installing PostGIS and the basics for using it.
  • OpenLayers – JavaScript library for displaying maps and mapping tools. If you see a map on a web site and it isn’t an embedded GoogleMap or MapQuest map, it’s probably being rendered by OpenLayers.
  • Open Source Geospatial Foundation (OSGeo) – not to be confused with the OGC, above. The OGC is a standards organization. OSGeo is a non-profit that supports open-source geo software projects and related initiatives. They support web mapping, desktop apps, geospatial libraries, and other types of projects, including GeoTools and OpenLayers.

If you thought that keeping OGC and OSGeo straight was confusing, just wait! There’s one more. OpenGeo is a company (sorry, a “social enterprise”) that integrates the most popular Open Source GIS technologies (like those listed above) into a single, supported stack or application framework. I have no experience with them, but if you need to get a GIS app up and running quickly, they sound like good people to call. I am sure there are other such organizations out there that can help write your software or train your dev team. I just now found that Scott Davis’ ThirstyHead company is offering 3-day GIS training course.

GIS Blogs

There are probably a 100 GIS-oriented blogs. Start with planetgs. That is an aggregator for many others. If you find that articles coming from a particular source are good, you can follow it directly. I happen to like Fuzzy Tolerance for it’s good content on GIS and Open Source web development in general and concise monthly roundups.

Where to Get Data

If you want to get “base layer” data (geographic, political, structural, etc) to display underneath your app-specific data, browse through these sites:


Geocoding

If you have address data or other geographic text and want to find out how to plot it on a map, there are a few free geocoding services. geocoder.us is a good starting place for testing your app if you just have data in the US. There are sister services for other countries. Google also has a geocoding service, but the license requirement says you have to use the data to display on a google map. (At least, it does in one place. In another place, it just says display on a map, without specifically stating google map.)

Getting Data Via WMS versus WFS

Before going into more detail about some of the above apps and libraries, I want to clear up something that confused me at first. How data is stored (vectors or rasters) is a separate concept from how it is distributed and displayed. When you request data from a WMS service, you will get back an image. It doesn’t matter if the data is stored as a jpeg or tiff or as a Shape file or a set of XML files or it is in a database table. Whatever the source, the map server converts that data into an image using some standard styling rules and sends that image over the wire. On the other hand, if you request the same data via WFS, you will get some form of data list, usually in an XML format known as GML. How that data is then transformed into some visual display is up to the client.

The difference between WMS and WFS has an impact on how you can combine data from different sources in the same web-based map when using OpenLayers. WFS layers can be subject to the cross domain scripting limitation. But that is the basis for another post.

Reference Apps

Finally, here are some apps that let you drool over the possibilities of what GIS tools can do for you.

Have your own cool web-based GIS app? Post a comment and I’ll add it in.

Next up… A few notes on setting up datasources in GeoServer.

GIS, GeoServer, OGC, OpenLayers, PostGIS, WFS, WMS