Architecture of online spatial modelling

With our most recent project we came another step closer to the realisation that government geospatial projects are moving out of the 90′s. We were asked by a client to build an online modelling tool for evaluating risk resulting from a natural hazard such as an earthquake. Understanding risk requires identifying the hazard (e.g. Tsunami) and exposure (e.g. population in a coastal city).

Data on hazards and exposure forms inputs to the risk and impact models. Such data is normally under the custodianship of national governments, with the responsibility of custodianship often distributed across departments and agencies. Bringing the data together into a single modelling tool is not just a technically challenging task but also requires cooperation and sharing of data across agencies. And anyone who’s worked in government knows this is often harder to do than solving the engineering challenges. We were very fortunate to have the involved custodians onboard with an eagerness to learn and deploy new technologies.

Back in the 90′s such hazard models would be run on a desktop with access to the data via a local disc copied from all the DVD’s posted by the data custodians using snail-mail.

This was a fine solution back in the 90′s but in the 21st century, quite frankly, it starts to look embarrassing. Our proposed solution was a distributed architecture which pulled hazard and exposure data using OGC web services and ran the model on a modelling server that could be scaled. The modelling server pushed the results of the run to another OGC compliant server. This is a simplified drawing of the architecture:

Once the model run was complete the client was notified via websockets. Here is the final application in action:

If you can’t access the video below – you may download the 16MB m4v file here or 10MB for the iphone.

Cool eh! With all these moving parts we needed to ensure that the various services were being monitored. Introducing Monit. If you haven’t heard of Monit check it out here: System Monitoring With Monit. It keeps tabs on your server daemons. If you have any questions feel free to post in the comments or get in touch via email.

… and I am in Berlin, if you are in town come say hi.

Plotting a feed of USGS ShakeMaps

Screen shot of the ShakeMap feed

Since Haiti I have been doing a lot of reading on disaster response and recovery. Today I needed to load a GeoRSS feed from USGS. They publish the location of earthquakes as a GeoRSS feed. They also publish a product called ShakeMaps. This is a fantastic product since it provides a map of the actual “ground shaking” a not just the simplistic magnitude and epicentre. Creating a ShakeMap is a function of local geology, epicentre, magnitude and many other variables. You can read about it here. ShakeMap are thus much more useful from a disaster response and recovery point-of-view. For example if you have the necessary data on geology and population exposure you can run a model that simulates an earthquake over a region to predict the likely effect on the population.

Getting back to the GeoRSS – the ShakeMap feed allows us to integrate this into any web mapping application thus creating interesting mashups. E.g. overlaying it with socio-economic or population exposure or vulnerability data to show the simulation results I mention above.

Okay so I’ll show you how to set up a GeoRSS feed parser using Sinatra. Here is the ShakeMap as GeoRSS Feed. You will need ruby and sinatra installed.

Here is the code for the routes in sinatra:

require 'rubygems'
require 'sinatra'
require 'open-uri'
get '/' do
  haml :shakemaps
end
get '/proxy' do
  open params["url"]
end

When the client hits the our root url (/) we will render the OpenLayers map with the USGS ShakeMap. We may also want to let people add other feeds.

So in our shakemaps.rb the first route is to the root. This is pretty simple, when the root of the app is hit the server renders the shakemaps haml template. The haml template loads our javascript to render the OpenLayers map. This javascript also calls the shakemap GeoRSS feed via a proxy. The proxy is needed because OpenLayers which is a javascript library isn’t allowed cross-site-server calls. The proxy is where the second method in shakemaps.rb comes along. It looks for the parameter url at the end of the URI and simply opens it. This is the GeoRSS XML feed. Finally there is a form allowing us to point to other GeoRSS feeds.

setup a geospatial-restful API and document it for others

UPDATE: I’ve had some criticism of the geo_scaffold code… which was really meant for my personal use. I’m tuning it up for a public release… check back in a few weeks.

Introduction
Following on from my last post where we created a mapping app using geo_scaffold I am going to show how to use cURL to play around with the RESTful location API that geoscaffold has generated for us. Geoscaffold has built us a working RESTful API that can be used with a bit tweaking. So all resources are accessable via a URL and HTTP verbs (GET/PUT/POST/DELETE). The basic API can be accessed via KML and GeoRSS flavor of XML and can simply be access by adding “.xml”, “.kml” or “.georss” to the end of your URL.

GET

curl http://localhost:3000/hotspots/1.kml

Authentication
We can add simple HTTP authentication very easily.
You will need a username and password for accessing the API. Login/pwd can be requested. Authentication is implemented as a basic HTTP authentication. in your controller

class HotspotsController < ApplicationController
  before_filter :authenticate
  ...
  protected
  def authenticate
    authenticate_or_request_with_http_basic do |username, password|
      username == "shoaib" && password == "secret"
    end
  end
end

lets try it out:

curl -u shoaib:secret http://localhost:3000/hotspots/1.kml

WRITE
In order to write to a resource we use the POST http method.

curl -u shoaib:secret -X POST -d "hotspot[name]=Merriot-San-Francisco& \
    hotspot[description]=Where20+conference+not2b+missed&lat=37.602561& \
    lng=-122.37096" http://localhost:3000/hotspots.xml

UPDATE
Updating the database requires the PUT method.

curl -u shoaib:secret -X PUT -d "hotspot[name]=My+current+location& \
    hotspot[description]=Where20+conference+is+in+progress& \
    lat=37.602561&lng=-122.37096" http://localhost:3000/hotspots/1.xml

DELETE
Delete the database requires the DELETE method.

curl -u shoaib:secret -X DELETE http://localhost:3000/hotspots/1.xml

There’s still hope for Australia’s spatial industry

The ASIBA last week release a set of recommendations for the Australian government. These recommendations were based on a economic study evaluating what the spatial industry in Australia is worth. A staggering 6-12 billion dollars is the figure mentioned.

But lets look at the recommendations: I quite like the aims of some of these: government initiatives to commercialise OZ spatial information R&D; government policy that improves the quality, quantity, currency and accuracy of spatial data; management of digital rights and information sharing and a whole-of-government approach to licensing of geodata.

There are a number of initiatives already in the pipe-line to address some of these issues for example to address constraints on information sharing, OSDM is developing a whole-of-government approach for licensing of spatial data. There is also a push to use existing standards for discovery of spatial data where applicable such as OGC‘s CSW whose most well known implementation is FAO’s GeoNetwork. There are also projects that look to make taxpayer funding research data free. The question now is how long will it be before we start seeing the fruits of all the good intentions from the government and traditional Spatial industry. How long is a piece of string?

Meanwhile neogeography projects such as OSM continue to flourish and so do the related standards such as GeoRSS. Another recommendation that will hopefully see greater support for such open and community driven initiatives in Australia is that

… priorities are user driven, not ‘producer’ driven

Hopefully this means that the OZ government’s inacceassable silo’s of geodata that’s currently begging to be freed may one day become available in formats that ease their use online and amongst the developer community. This would no doubt lead to the emergence of new industries, a responsibility that governments have towards their nation.

Mapserver output as KML

Alessandro Pasotti has authored a utility to output mapserver layers as KML for display on Google Earth. This is written in PHP mapscript so will require mapscript to be installed with your standard Mapserver installation. Check it out here.

Although its a useful tool, it would be nice to have kml support in mapserver (C++) out of the box. Apparently expected is the release of libkml in 2008. So mapserver may have to wait for google to release libkml before adding native C++ support, provided the licensing is favorable.

Geospatial @ RailsConf Europe 2007

railsconf_speaker Some great news for those interested in the convergence between Ruby/Rails and Geospatial applications. Kashif and I will be giving a tutorial at RailsConf in Berlin. There is going to be 3 hr tutorial titled Rails GIS Hacks, where we will be going through some pretty nifty stuff such as RESTful geospatial applications development. Hope to see you there!

OSGIS racing ahead: Sensor Observation Service

MapserverAs I said in my last post one of the topics of discussion at the melbourne’s Mapserver users group meeting was the uptake of OGC standards by the OSS community. Here’s an example of OSS taking lead in implementing upcoming open standards: the current CVS version has has already implemented partial support for Sensor Observation Service for enabling Sensor Webs. If you can’t wait go a head and try it. So currently its not generating an XML response to getCapabilities request but rather needs a predefined URL. This will change very soon, I’m sure.