Pulling data from OSM using XAPI, bringing it to ArcGIS

I was recently asked by a friend if I knew of an ArcGIS-friendly data set representing the highway interchanges within New Jersey. I knew the information was already available in OpenStreetMap, but how could I quickly pull just the highway interchanges out of OSM and bring them into ArcGIS?

Using the XAPI, I was able to pull just the relevant information from OSM. In this case, I only wanted nodes that contained a highway key with the value of motorway_junction. I used MapQuest‘s XAPI service and quickly generated a URL using a nifty map mashup. I had to manipulate the XAPI URL manually to change the bounding box; New Jersey was too large to search with a statewide bbox, so I broke the state into four portions by changing the latitude ranges. Making smaller requests for the part of the state between latitude 39 and 40, then 40 and 41, allowed my request to go through without timing out.

I was then left with four XML files that contained the highway interchanges. ArcGIS doesn’t know what to do with this, so I whipped up some Python to convert all four tables into one tab-delimited file. This code (included below) can then be used to add the points to ArcMap using the Add XY Data functionality.

All said and done, this project took me about an hour. I was able to send the data off in about 30 minutes, but then spent another thirty minutes cleaning up the script, modifying it to spit out any available key, not just the “ref” tag. The script is GPL, free for you to use as you see fit. Just remember to give credit where it’s due, attributing your data back to OSM.

Taking the time to do this for a friend is a good reminder of why I push our students to do more than just the GIS curriculum. Being familiar with OSM gives you a different point of view on GIS data, schema, and community than what you would gain from most GIS courses and entry-level positions. GIS is already about problem solving; being able to do some basic programming will enable you to tackle much larger problems efficiently. And providing technical assistance to others just makes you a better person.

This entry was posted in OpenStreetMap, Technology, Tools and Scripts and tagged , , , , , . Bookmark the permalink.

One Response to Pulling data from OSM using XAPI, bringing it to ArcGIS

  1. Harry Wood says:

    Great stuff! Here’s a couple of extra things to note:

    XAPI returns .osm formatted XML which may or may not contain only nodes. In the case of this highway=motorway_junction tag, it’s hopefully only matching on nodes, not ways.

    So actually rather than saying “Script to merge and convert a series of OpenStreetMap XAPI results” you can say more generally it’s a script which converts any “.osm formatted XML” into TSV …assuming that you’re only interested in nodes. Nice! I did a similar thing in ruby here

    Of course there’s probably some more pointy-clicky ways of getting .osm formatted data into ArcGIS. Would the OpenStreetMap ArcGIS plugin load in this XAPI output? Hopefully.

    You said the XAPI queries were hitting timeouts when working with all of New Jersey. Another approach to this problem, for the first step of getting filtered OSM XML, would be to take a New Jersey planet extract (a download of all the data) and then use the osmosis tool to perform filtering on your local machine.

    All good fun, but I hope we can find ways to make this stuff a bit easier