GeoLabs

Last few years have been difficult for Pakistani citizens. However the ordinary citizen is so removed from the geopolitical forces driving the conflict in Pakistan that it is hard to imagine how an ordinary citizen may be able to contribute to resolving the conflict. In these circumstanses I was delighted to see the progress that the openstreetmap community in Multan have been making towards mapping their city using Openstreetmap. Here is a video from Aleks created from the GPS tracks that Kashif has been editing for the OSM community in Multan. Keep it up folks!

Mapping Multan from Aleks on Vimeo.

Tags: openstreetmap, multan, pakistan

OSM in Pakistan hasn’t been very active in the past. However the future is looking bright. This year during the state of the map in Amsterdam you will run into a university professor – but don’t let the absent minded-professor-look fool you. This is a man on a mission – as a professor at the largest university in Multan (Bahauddin Zakariya University) he has managed to generate a ground swell of students and faculty to start systematic mapping of Multan. He has just received a small (as in tiny) grant from his University to help kick-start the mapping of Multan.

There is a lot of work ahead and being in Amsterdam (thanks to the generous scholarship from Open Society Institute) he will be keeping his ears open to learn as much as possible from the seasoned OSMer – especially those from countries at a similar stage of mapping as Pakistan.

So get to know this face and if you see it introduce yourself and share ideas. He will be giving his lightening talk along with the other scholarship holders – add it to your calender here.

You can checkout the planned mapping parties in Multan, Pakistan. Also feel free to get in touch with Asif via email: mianasifrasul at gmail.com

Note that this post is for folks who are new to ruby. I have to admit a lot of geospatial developers have not been using ruby due to the poor support for geo libraries in ruby – so it’s understandable. In this post I will show how to start parsing the FOSS4G 2009 Schedule. Lets start by gathering our tools. I am going to assume you have ruby installed. If you don’t have Ruby install, don’t fret there are now one-click installers – yes even for windows: http://www.ruby-lang.org/en/downloads/.

Reading YAML: The first thing we will need to do is to read the ‘official’ FOSS4G schedule YAML file that’s hosted at http://2009.foss4g.org/schedule.yml – as of writing this it contains data for workshops and tutorials. We will connect to the url using OpenURI module which is part of Ruby’s standard-lib. So lets try something simple, like reading the data into a Hash:

The data from the schedule is now loaded into the hash named data. If you are not familiar with ruby hashes checkout the class docs http://www.ruby-doc.org/core/classes/Hash.html also if you listen to the surreal voice in these videos you will fall in love with hashes forever – don’t say i didn’t warn you.

Ok moving on, lets poke around this data structure containing the FOSS4G schedule. Lets say I want to iterate over each tutorial and workshop then create an ical entry that we can share with our friends – so they know which sessions they may be interested in attending. Useful eh? – OK maybe not that much but it’s a start. The code for iteration would look something like this:

Next I’ll show you how to create an iCal entry and perhaps email it to your friends.

Web is a great ideas playground. This is the last week for the submission of Presentations for FOSS4G 2009 (http://2009.foss4g.org/presentations). So far we have had a great set of workshops and tutorials submitted. Mark Leslie (FOSS4G 2009 Workshops Chairman) did a fantastic job coordinating the call, receipt and collation of workshops and tutorials as well as managing the voting process (Checkout Mark’s post about the workshops/tutorials). This has resulted in an impressive line up of expert instructors and tutorials on cutting edge and relevant topics.

This information was provided in a document to me by Mark. For the FOSS4G 2009 conference we want this information as widely shared as possible.

So the first thing I did was to take the information that Mark give us and converted it into a YAML format. This meant coming up with a simple data model for the data.
I did this by creating the a file for events called schedule.yml and one for presenters: http://2009.foss4g.org/schedule.yml and http://2009.foss4g.org/presenters.yml.

They both reference each other. After some trial and error I came up with something that gave us a good description of the event, authors/presenters and time of event. It had to be intuitive and easy to parse:

Next I want to demonstrate how easy it is to create build something useful from these YAML file. Well I am an iPhone owner and also enjoy developing in ruby.

Enter Sinatra: My first thought was to build an iPhone website where people can browse the FOSS4G 2009 – and it stuck. I decided to use Sinatra an elegant, lightweight ruby web framework. The site is up and running at http://foss4g2009.heroku.com You can get the code using git from git@heroku.com:foss4g2009.git

In the next few posts I’ll walk through and show you how I built this site using sinatra + iui and used heroku to deploy it. The site currently features:

• Browse workshops & tutorials
• Share ical event file of an event via email
• Download ical event file of an event via web

Hope it will encourage others to hack around with this data or suggest ideas for building something interesting for the FOSS4G 2009 conference.

image source ABC

So what is a SDSS and how is it different from a GIS?

In order to understand SDSS and know the difference between GIS and SDSS it is important to understand what a GIS is. GIS is a piece of software that can perform generic spatial analysis and geoprocessing methods on geographic data. It requires an GIS Analyst or an expert to operate it. In contrast a Spatial Decision Support System is a domain or an industry specific software. It doesn’t require a GIS expert to operate but rather a domain expert. As the name suggests the software provides decision support but to do so makes use of spatial analysis, geo-statistics, geo-processing or other tools from spatial information sciences. To begin with a SDSS must be designed to answer some domain specific questions that have strong elements of geography.

This is best illustrated through an example: say, in coordinating the containment of a bush fire a sector coordinator needs to decide on where to deploy bulldozers to create a containment line or a barrier to the advancing fire-front. The job of the software is to provide the coordinator with a number of alternative answers that they can choose from based on their experience. In this instance the SDSS will take into consideration a number of information sources, perform a combination of spatial analysis and use sophisticated fire modeling to determine answers.

From the above example it is clear that a Spatial DSS must have access to relevant up-to-date spatial data, contain algorithms from spatial information science and domain-specifc models to answer domain-specific questions and a method for visualizing the answers. So for the sector coordinator the relevant data required would be topography, vegetation fuel maps, weather forecast, real-time weather measurements, verbal information from lookout towers and/or air surveillance, satellite reconnaissance, information from thermal cameras. All this would need to be geo-referenced and lie with in the spatial extent of her sector.

Next component is the algorithms that will convert raw data into useful knowledge. It is important to note that data and algorithms are closely linked since some of the data sources may be derivative products. For example vegetation fuel map may have been derived by applying a Normalized Difference Vegetation Index (NDVI) algorithm to multispectral remotely sensed satellite imagery.

Another example may have an algorithm to convert topography into slope. This slope may be used to rule out areas that are too steep for bulldozers to operate. This nicely leads us to the related component of domains-specific models. In this case the information from the slope, weather conditions (such as wind direction and humidity) as well as the vegetation fuel data may form inputs into a fire-model that predicts the future course of the fire. This information may then be combine with areas where bulldozers can operate to give deployment alternatives to the coordinator.

In a SDSS the above process would form a seamless chain of inputs and give an output. While in a GIS the above would be done by an expert spatial analyst who must be aware of all the pitfalls of combining different spatial data and deal with spatial coordinate systems. But above all a GIS will lack the modeling capability to predict the future course of the fire let alone understand what a bulldozer is. This brings us to another important difference between a GIS and SDSS. SDSS must deal with semantic information. In a SDSS spatial data cannot exist in isolation from its meaning.

An eye on the drought, 2007

Originally uploaded by Old Parliament House

Today while riding by bike around Canberra I stopped over at the old parliament house to catchup with a friend. I was blown away by a stunning exhibition called “Beyond Reasonable Drought”. The exhibition documents in photos the effects of drought on the people, livestock and the land over the last few decades. The project looked to the Farm Security Administration (FSA) project from the great depression for inspiration. The FSA project is no doubt was one of the most impressive examples of photo-journalism.

In the Australian exhibition, as if the photos were not impressive enough upon checking out the website (http://www.oph.gov.au/brd/) for the exhibit I found all photos are geotagged.
http://tinyurl.com/drought-map-kml
http://tinyurl.com/drought-map-flickr

This information when overlaid on top of the major river basins in Australia provides an instant understanding of why the Darling/Murray basin is running out of water. If we take the clustering of the photographs as a surrogate for drought intensity then most of the photos are clustered in the southern end of the daring/murray basin in Victoria.

Uploaded with plasq’s Skitch!

On a related note I just finished a project with Mikel Maron to document the growing trend in the US amongst younger people to take up farming. This aims to capture profiles of young farmers by getting them to complete a survey and provide their location. Check it out here http://www.serveyourcountryfood.net/ (it was just launched and is currently in beta)

Kashif Rasul last year decided to buy himself one of those GeoChron GPS from SparkFun (they are really cool: field hardended and can take SD cards check it here) … after doing some mapping around Europe and Malaysia he decided to use it to initiate the first openstreetmap mapping party in Multan (which incidentally is one of the oldest known cities in the world – wikipedia-page). So he parted with his beloved GPS and it has started to pay off… Multan which till last week was completely blank on OSM has now got its first two roads…

To promote the idea of mapping parties Kashif’s dad who is a professor at a university in Multan organised a talk and mapping party. With just one GPS this first mapping party attracted over 45 people. This is huge! Here’s the email from Asif Rasul himself:

“How r u [Kashif]? I have given the presentation today about 45 people came. went fine. We will Inshallah start the regular work on week ends. Many student volunteered the job by motorcycle.”

Sweet! Below are the fantastic slides that Kashif and Aleks put together … they really hit the spot and make one wanna get out and map! i have to say thanks to Mikel Maron for being an inspiration (with his recent OSM related work in the Middle East) and letting us use some of his material for the slides. We also used some of Andrew Turners photos.

As you see in the slides we are currently constrained only by the number of GPS’s – no lack of motorbikes and riders here :) .. so if someone out there is looking to support a good cause then donate some GPSs for mapping a city which has a rich, unique and beautiful culture. Sadly publically available spatial data is scarce. There is a lot of work to be done and to keep the mapping momentum in Multan we really do need more GPS’s. I look forward to the day that this data can be used by the citizens of Multan for social development projects.

If you would like to get in touch about donating GPSs or are interested in osm in Pakistan feel free to leave a comment below or email us.

Google MapMaker is out. There are several geowiki projects, but when someone like google decides to have a go at the problem you have to take notice. First thing to note is that the MapMaker is open for a limited set of countries, one of which I have blogged about earlier. So lets see what the uptake of MapMaker is by looking at Karachi (a city of ~20million, metropolitan and where locals are cultured with a strong entrepreneurial spirit) it appears there is very fast uptake from the locals. Here is what has happened in the course of a few hours

Uploaded with plasq’s Skitch!

With local languages support:

Uploaded with plasq’s Skitch!

The same location under OpenStreetMap looks like this:

Uploaded with plasq’s Skitch!

This shows what great usability can accomplish. There are a number of important differences between mapmaker and openstreetmap. The concept of confidence doesn’t exist in OSM. While mapmaker allows users to state the level of confidence by indicating how well they know this location.

Uploaded with plasq’s Skitch!

The closest OSM gets to this idea is the tip about only mapping places you have been to.

Uploaded with plasq’s Skitch!

Mapmaker has a distinct peer-reviewed moderation process. Users have the ability to state the quality and confidence level of the data. This can allow the higher quality data to bubble up into googlemaps while work goes on on the lower accuracy/confidence data.

Another important difference is the separation of editing and browsing the map in mapmaker. In mapmaker note that you can select the data and pan around without the fear of accidently editing the features while in openstreetmap panning and editing are mixed.

MapMaker Browsing:

Uploaded with plasq’s Skitch!

OSM Browsing/Editng:

Uploaded with plasq’s Skitch!

OSM has a “play” option to let you get confident before making the edits but moving between play and edit mode means an all or nothing choice. While in mapmaker you can browse, select a feature and choose to make edits if you want.

Mapmaker Pick Editing:

Uploaded with plasq’s Skitch!

Another interesting feature is the ability to add events and specify their type and significance. This is an interesting paradigm shift in stating an event by going via its geographic attributes first.

Uploaded with plasq’s Skitch!

Now the question is will google play nice and make their data available under a liberal licensing scheme? Will there be a REST API? Can OpenStreetMap learn from this and get even better? What does this mean for national mapping agencies and commercial data providers? Is google now a competitor? Is it time for other to build even better tools for crowd sourcing or should we be sharing and have a coordinated mapping effort? Lots of questions very interesting to see how it all plays out over the next few months.

UPDATE: I downloaded the full postal codes listing for Australia from Australia Post’s website. I then wrote a script to geocode these postcodes by querying the Geonames postcode API. Here is a file with all the postcodes I couldn’t find in Geonames. Next I will spatially look for the electoral divisions for the postcodes I have found.

Incase you are interested here is code (a spec – don’t you love Ruby/RSpec – a specification that actually works :) )

require 'open-uri'
require 'csv'
require 'json'
 
class GeocodeAustralianPostcodes
  @@base_url = "http://ws.geonames.org/postalCodeSearchJSON"
 
  def find(postcode)
    url = @@base_url+"?postalcode=#{postcode}&maxRows=5&country=AU"
    JSON.parse(open(url).read)
  end
 
  def postcode_hashes(file_location)
    csv = CSV::parse(File.open(file_location, 'r') {|f| f.read })
    fields = csv.shift
    csv.collect { |record| Hash[*fields.zip(record).flatten ] }
  end
 
  def write(infile_location, outfile_location)
    phashes = postcode_hashes(infile_location)
    outfile = File.open(outfile_location, 'w+')
    bad_outfile = File.open('data/postcodes_notfound.csv', 'w+')
    outfile.puts("POSTCODE, LAT, LON, adminCode1, adminName1, adminName2, placeName")
    phashes.each do |h|
      response = find(h['Pcode'])
      if response["postalCodes"].empty? 
        bad_outfile.puts("#{h['Pcode']} not found") 
      else
        response["postalCodes"].each do |r|
          outfile.puts "#{h['Pcode']}, #{r["lat"]}, #{r["lng"]}, #{r["adminCode1"]}, #{r["adminName1"]}, #{r["adminName2"]}, #{r["placeName"]}"
        end
      end
    end
    outfile.close
  end
end
 
describe GeocodeAustralianPostcodes do
  before do
    @gap = GeocodeAustralianPostcodes.new
    @phashes = @gap.postcode_hashes('data/postcode_sampledata.txt')
  end
  it "should read the postcodes from the text file and convert them to an array of hashes" do
    csv_reader = CSV::Reader.parse(File.open('data/postcode_sampledata.txt', 'r')) 
    csv_reader.shift
    @phashes.each do |h|
      h['Pcode'] == csv_reader.shift[0]
    end
  end
  it "should find the postcode on each line and geocode the postcode using the Geonames JSON API" do
    @phashes.each do |h|
      resp = @gap.find(h['Pcode'])
      if resp["postalCodes"].empty? 
        #puts("#{h['Pcode']} not found") 
      else
        resp["postalCodes"].each do |r|
          #puts "#{h['Pcode']}, #{r["lat"]}, #{r["lng"]}"
        end
      end
    end
  end
  it "should write back to an outpufile with Lat, Lon, Postcode if geocoded, othewise postcode# not found" do
    @gap.write('data/pc-full_20080529.csv', 'data/aus_post_output.csv')
  end
end

I noticed the Postcode Areas (POA) data at the ABS is missing Postcodes because they have been created by aggregating Census Districts (CDs) resulting in some postcodes dropping out because the CDs which they covered had already been allocated to other postcodes. Strange a one-to-many associations don’t seem to be supported for CDs and POA. I might need to get in touch with Australia Post to see if they have any point or polygon data for postcodes that has been mapped independent of Census Districts. Then do a join with the Electoral Divisions from the ABS. Anyone out there already done this?

Back in Feb at the Sydney Ruby meetup Matthew Landauer gave a talk on the Open Australia project. Today the site went live and it’s looking great! (OpenAustralia.org) The project has been developed by volunteers. You can enter your postcode, see who your local representative in parliament is, watch their activities in parliament. I love the ability to comment on each statement spoken by a parliamentarian. I actually had a very-teeny-tiny-little-bit to do with the postcode functionality. I sniffed out some of the data for postcode to commonwealth electoral division mapping. Matt informs me the first bug reports have to do with missing postcode data *blush*… yep I screwed up!… I didn’t spend enough time looking at the data before I sent it, thus overnight turning my geo-guru status in Matts eyes to a mere geo-goat!

Now having a closer look at the ABS site I think I may have spotted the problem… the data for the Electoral Divisions and Postcode are not in the same hierarchy. In other words one dataset cannot be aggregated or de-aggregated to create the other. So what’s the solution? Yep the *real* geo-gurus are thinking full outer spatial-join.

For the geo-goats (like me) it will help to check out this diagram that shows how Australia is chopped up into shapes by the very competent folks at the ABS.

What’s going is that the two regions are derived separately and in separate branches. I am currently looking at fixing this by getting a hold of the two data sets and freshly deriving the postcodes.

So I got the data from ABS and here are the results, looks like its as though the electoral divisions may have more than one postcode. Now to generate a new dataset… This is not a problem at the application level. The problem might occur if there is a postcode overlaps more than one electoral division … still checking the data for that

Where2008

This is a post that’s long over due … I really should have been blogging on a more regular basis while in the US but I was having so much fun experiencing the culture, cuisine, nature and the sheer creative energy of the Bay Area that it left me too exhausted at the end of each day to actually blog. Now back in the quite country town of Canberra, I am recollecting the experiences of the last 4 weeks. First of all let me say how useful Where2.0 and WhereCamp were for me personally and as the buzz around the web indicates for all those who were involved. I was touched by the generosity of Mikel Maron (an SF local) and Andrew Turner (mostly local). They showed me around the city and introduced me to the geo-scene in SF and despite their very busy schedule made time to catch up. Thank you also Brian Hamlin another SF local who helped me out at the Where2.0 tutorial. I’ll write more about the exciting 4 weeks but here are some words that summarize the experience: Open source, Commercial, Intellectual-Fu, Social Causes, Multiculturalism, Downtrodden, Natural Beauty, Minimum Wage, Healthcare, Entrepreneurial Drive and Good Old Mate-ship … Yes even Americans have mateship albeit known by a different name.