Pulling data from OSM using XAPI, bringing it to ArcGIS

I was recently asked by a friend if I knew of an ArcGIS-friendly data set representing the highway interchanges within New Jersey. I knew the information was already available in OpenStreetMap, but how could I quickly pull just the highway interchanges out of OSM and bring them into ArcGIS?

Using the XAPI, I was able to pull just the relevant information from OSM. In this case, I only wanted nodes that contained a highway key with the value of motorway_junction. I used MapQuest‘s XAPI service and quickly generated a URL using a nifty map mashup. I had to manipulate the XAPI URL manually to change the bounding box; New Jersey was too large to search with a statewide bbox, so I broke the state into four portions by changing the latitude ranges. Making smaller requests for the part of the state between latitude 39 and 40, then 40 and 41, allowed my request to go through without timing out.

I was then left with four XML files that contained the highway interchanges. ArcGIS doesn’t know what to do with this, so I whipped up some Python to convert all four tables into one tab-delimited file. This code (included below) can then be used to add the points to ArcMap using the Add XY Data functionality.

All said and done, this project took me about an hour. I was able to send the data off in about 30 minutes, but then spent another thirty minutes cleaning up the script, modifying it to spit out any available key, not just the “ref” tag. The script is GPL, free for you to use as you see fit. Just remember to give credit where it’s due, attributing your data back to OSM.

Taking the time to do this for a friend is a good reminder of why I push our students to do more than just the GIS curriculum. Being familiar with OSM gives you a different point of view on GIS data, schema, and community than what you would gain from most GIS courses and entry-level positions. GIS is already about problem solving; being able to do some basic programming will enable you to tackle much larger problems efficiently. And providing technical assistance to others just makes you a better person.

Posted in OpenStreetMap, Technology, Tools and Scripts | Tagged , , , , , | 1 Comment

New Jersey Bike Map public meeting

detail of the NJ Statewide Bicycle Map.

This Thursday, May 10th, I will be attending the final public meeting to discuss the draft New Jersey Statewide Bicycle Map and Resource Guide. I attended the first meeting at Cumberland County College, but will be again providing comments at this final meeting to be held at NJ DOT Headquarters. I’m very much a single-use public commenter, as I really do not have many concerns about the map at all; frankly, it’s beautiful and I plan on getting copies as they are data-rich and rendered by an excellent set of cartographers. My concern is the data-richness – this map is detailed, and for DOT or the consultants to not release the underlying data along with the maps would be a sin.

Let me take a step back and explain how I first found out about this project. Dave Heller from SJTPO posted an email to the New Jersey Geospatial Forum discussion list, asking the group if there was any bicycle facility data that was publicly available and more recent than 2007.

Hello:

We have a bicycle/pedestrian GIS layer from 2007.  I wanted to find out if people here are aware of anything more recent.  Specifically, we are interested in South Jersey.  Please let me know if you are.

Thanks.

David Heller

Being the great resource that it is, within 10 minutes another member of the NJGF mailing list, Robert Blash, pointed Dave to the Statewide Bicycle Map project web page.

Take a look at http://bikemap.com/njbike/

NJDOT has a project to map out biking areas in NJ.

Robert Blash

I was in the middle of writing an email to Dave & the list about using OSM data for bicycle facilities when Robert’s email came in, so I took a look at the site. While I was glad that DOT is looking to create a statewide cycling map, I was a little dismayed by what I saw on the site. My email to the list follows:

CloudMade has OSM-sourced data available as ESRI Shapefiles.
http://downloads.cloudmade.com/americas/northern_america/united_states/new_jersey#downloads_breadcrumbs

You can see what bike/ped facilities are in place using OpenCycleMap: http://opencyclemap.org/
If you take a look at Glassboro – http://opencyclemap.org/?zoom=15&lat=39.70289&lon=-75.10368&layers=B00 – You will see the Glassboro-Monroe bikeway and the pedestrian connections throughout campus. Some of our bike racks are on OSM, but not all of them.

Regarding the bikemap.com site, I contacted DOT a few years ago regarding the old Bike Maps that were released, looking for the underlying data. The maps were produced by the RBA Group. DOT informed me that the PDFs were the only deliverable and that DOT did not have ANY geospatial data related to bicycle lanes, routes and facilities. bikemap.com is run by RBA Group. Will this project be no different and community-sourced bike data be locked up by a private consulting firm?

You can register for the site and leave comments. Please take a few minutes and let them know that the GIS data behind the map should be made publicly available. Comment #2 is what deliverables are important; click “electronic file”. For question #4, comments, I left the following:

“Please make sure that all data collected through bikemap.com relevant to the production of the cycling routes and facilities is made available to the citizens of New Jersey by placing the data online under an open data license or released as public domain.”

If this project is funded by both NJ DOT and US DOT, the results of it should be open and accessible.

Peace, love, and open source,
John

Merrilee Torres, from Burlington County, replied to the list mentioning that Steve Spindler Cartography (who owns bikemap.com) is involved in the project and that echoing my concern for public release of the data, as the project managers, RBA Group, solicited the counties for bicycle facilities data.

My understanding is that Steve Spindler Cartography is doing the mapping for the state bike map but yes, RBA is managing it.  I do know that they were trying to collect as much source data from Counties as possible which gives even more credence to the idea that the data should be publicly available.

Merrilee J. Torres, GISP

I then replied to Merrilee and the list, expanding on my position on government and open data:

I don’t want to suggest that the State (if it’s even legally allowed to do so) adopt some sort of license like the GPLv3 where derivative works are required to be released under the same terms; open and redistributable. If that were the case, it could be argued that anything sourced off of aerial photos would then need to be released. There’s a massive amount of value in having the State’s data open, however it does seem like the public is getting short shafted when public data gets vacuumed up and then locked away. There is some value-added in performing the work that merits compensation to the consultant, but if the State is the one footing the bill, it should be stipulated that the product is owned by and distributable under whatever terms the purchaser/the State deems fit.

The paper NJ Bike maps are going to be great to have. Let’s face it, while many GIS geeks seem to also be biking/outdoor enthusiasts, the reverse is far from true. The public is going to need the paper maps. If the data is released, there’s great opportunities for the GIS geeks to perform some analysis that either improves the data further or could help influence policy relating to bicycle use in the state. Open government data allows many great ideas and projects to come to fruition; producing data, research and applications far beyond the scope of the original project.

And if the data was already there and freely accessible, Dave Heller would have found it and y’all wouldn’t have been subjected to my open data rants! 🙂

John Reiser

The conversation then took a couple turns back on track, to locate some additional information for Dave, then died out. Then a month or so later, I received notice of the public meetings. On April 24th, I rode my bike to work, checked out a ZipCar and went down to the CCC campus for the public meeting.

The meeting was conducted by two staff members from RBA and a staff person from Baker. After explaining the process and the methodology used to rate the state’s roadways for bicycle suitability, they opened it up to comments and questions. I asked about the intent of the map; “Who is your audience?” was met with a “Yes,” followed with an explanation that they intend the map to be useful to anyone.

I voiced my concern with such a sentiment. Paper maps are not going to serve everyone. While most people cycling for recreation are almost guaranteed to benefit from the map, it’s still in paper form. There’s a cost to print and deliver. And even if you provide PDFs, there’s few opportunities for individuals to print large-format maps without a visit to a printer.

Cycling is a key part of commuting, whether it is the sole mode or used to make connections, like above. About 50 bikes were locked outside of Harrison's Path station on a rainy weekday.

Beyond that, what about those that bike because it is their only available mode of transportation? Are the maps and guidebooks going to be available to them? Are the books only going to be available in English? Take a visit to nearly any suburban chain restaurant and you will see a few bikes locked up in the rear, surrounded by a sea of automobile parking. Those bikes don’t belong to a suburban family of four, they belong to those cooking the meals and washing dishes.

If the data behind the maps are made available, the utility of this endeavor increases exponentially. Paper maps serve a purpose and are nice things to have, but in terms of providing for cyclists, planners, social workers, public advocates, and geeks like me are not going to be able build upon a PDF map. Providing the data in a free (libre, not gratis) form enables so many ancillary players to tap into the resource provided and expand its benefits to a larger audience.

Jersey looks devoid of regional connections on OpenCycleMap. I've personally added some smaller tours and bike paths, but there's much more left to add.

I’m attending the meeting at DOT mainly due to the response I received from the individual from Baker regarding the previous set of bicycle maps. When asked about why the data behind the previous set of maps was not made available, the answer I received was that the maps “were done in Illustrator.” Yes, the set of maps (available to download at no-cost from the link above) were likely modified and aesthetically improved in Illustrator. But only Illustrator? Considering the state has GIS data for the millions of miles of roadway in the state, why was only Illustrator used? Fine, forget the old maps for now – the new maps have the tours available on them, along with all of the roadway condition information. There is no possible way that all of that information was managed effectively outside of GIS. And if it was an entirely Illustrator produced product, why is the State paying consultants to be inefficient?

I was told that the data behind the new maps is definitely GIS data, but that the staff of DOT will ultimately decide if the data will be released. First, of all, I intend to submit an OPRA request for the data if it is not released when the project is deemed complete and the paper maps & PDFs are released. Secondly, why wouldn’t DOT want the data released? They already provide GIS data and maps in multiple forms for any use.

I spoke with Elizabeth Cox from RBA after the meeting and she encouraged me to provide written feedback. I will likely print this blog post (how quaint) and will submit it at the public meeting. I may decide to write something more formal for submission and will update this post to include it if written. At the least, I intend to attend the meeting and directly ask the individuals at DOT what are the future plans for this data. I also encourage you to provide feedback via their contact form. I’ll provide updates after the meeting Thursday.

Posted in Cycling, Data, Government, OpenStreetMap, OPRA, Planning, Technology, Transportation | Comments Off on New Jersey Bike Map public meeting

Announcing LearnWebMapping.com

I am pleased to announce that I have started another blog, LearnWebMapping.com. This summer, I am offering a Web Mapping & GIS Services course through Rowan University Online. To promote the course and to help keep myself mentally focused on web mapping concepts and technologies, I set up Learn Web Mapping. I will post as often as possible – aiming for daily – with tips, announcements, and articles related to web mapping. At this time, I intend to keep the blog ad-free; actually, the only advertisement being for my online course.

Go check it out. The first article is on the Digital Blue Ridge Parkway project from the University of North Carolina. I give an overview of the map itself, and then some details on the technology used to the make the web map.

I welcome any feedback you have, either here in the comments or via Twitter.

Posted in Updates, Web Mapping | Tagged , , , | Comments Off on Announcing LearnWebMapping.com

Zipcar is here!

Chelsey and Mike approve of Zipcar

Chelsey and Mike approve of Zipcar

The Rowan University Facebook account recently posted some not-so-clear messages announcing the arrival of Zipcar on campus. I was thrilled to hear the news! Having a cheap rental car nearby has prompted me to reconsider owning two vehicles. I live just over a mile from Rowan and often ride my bike or walk to work. Why should I pay for a car I don’t use? Continue reading

Posted in Cars, Transportation | Tagged , , | Comments Off on Zipcar is here!

Finals season is upon us.

Fall Final Exam week is now here and I have been dealing with an unending stream of last-minute, panicked questions about the software and the assignments. This semester, I have two sections of GIS I which is required for a few majors. Until our students make it through GIS II (not required except for GIS majors, so usually only the Kool-aid drinkers get that far), they have a love-hate relationship with ESRI’s flagship product. At the start of each semester, until they have had a significant amount of experience with the software, students react with groans as each new assignment is given.

"What's in the box?" "ArcGIS 10 for Desktop Standard."

After a long 15 weeks, students will gain the experience and confidence needed to tackle some difficult, real-world problems with GIS.

"Mastering GIS is akin to riding your first sandworm."

Continue reading

Posted in Uncategorized | 1 Comment

Colleague in Need – Atanas Entchev facing deportation

Update: The GIS community has come out in support of Atanas and his family. Calls for letters of support have spread across Twitter and has been picked up by several other blogs:

Despite this, the Entchev’s future in the US remains uncertain. If you have written a letter of support, thank you. Please write one if you are able.


Atanas Entchev, a familiar face to many on Twitter and throughout the GIS circles in the Northeast is currently facing deportation to Bulgaria. After emigrating here with his family over twenty years ago to escape political persecution, the US Government is now planning to deport the Entchevs. Atanas, his wife and his two children have been living in Central New Jersey and integrated into the local community. He has been involved with GIS in New Jersey for years, having worked in government at the Department of Environmental Protection and at several firms before starting Entchev GIS Architects back in 2005.

Last week, I received an email from Mayia, his wife, asking for letters of support as he is facing deportation. Mayia’s email is as follows:

I am writing you on behalf of my husband, Atanas Entchev, and our family. As you may know, Atanas and I emigrated to the U.S. from Bulgaria 20 years ago. Today, our immigration case is at a crossroads, and it is pertinent that we obtain letters of recommendation from the community. I am reaching out to you because you have been a colleague, client, or valued business contact of Atanas’. I am hoping you may be able to write a letter of recommendation for him. Time is of the essence, and it it crucial that we gather these letters as soon as possible, by today would be best.

If you are willing and able to write a letter of recommendation, your contribution will be invaluable to our case, and would be greatly appreciated by our family.
Please include your name, signature, address and phone number

Best would be if you can email the letters to:
mentcheva@hotmail.com & ericmarkesq@gmail.com
If not possible please fax to (201) 262-7640.

If you would like to get in touch with me, my contact information is listed below.

Thank you very much in advance, you help is greatly appreciated by our family.

Best,
Mayia Entcheva

I have known Atanas for several years now and I consider him a friend. While I knew he was not originally from here, I never thought of him as an immigrant. He is – in my mind – a fellow US citizen and outstanding professional colleague. He’s not a Bulgarian; he’s a New Jerseyan –  which is far different from what you see on TV.

If you have worked with Atanas, please take a few minutes to write a letter stating your support. These letters will be incredibly helpful to him and his family. Please send any letters to the email addresses or fax number above. If using email and if you have access to a scanner, they should be signed with your signature. I’m including below my letter of support for you to use as a guide. Continue reading

Posted in In the News, Meta, NJGeo.org | 9 Comments

RULost? I hope not.

Fellow Rutgers grad Jim Barry shared with me a link to a press release on the RULost iPhone app. While I was initially excited to see Rutgers come out with an app to aid students in navigating the huge New Brunswick campus (which is actually 5 campuses split between New Brunswick, Piscataway Township, Edison Township and a small portion in North Brunswick Township) and its extensive bus system, after plunking down 99¢, I’m not so impressed. Continue reading

Posted in Data, Mass Transit, Technology, Transportation, Web Mapping | Tagged , , , | 1 Comment

GIS & Live Maps for Hurricane Irene

New Jersey is in the path of Hurricane Irene. The last time a hurricane actually made landfall in New Jersey was the 1903 Vagabond Hurricane. The mandatory evacuations are underway (NJ OEM) and to my knowledge, it is the first time the Garden State Parkway has instituted contraflow traffic (NJTA, PDF). Below are links to several trackers and GIS resources to keep you up-to-date on the storm.

Posted in Data, In the News, Web Mapping | Tagged , , , | 1 Comment

Statewide 1974 aerial imagery available via WMS

OIT has made the 1974 statewide aerial photography available via their WMS service.

WMS: http://njwebmap.state.nj.us/njimagery

Civil Solutions was the contractor that assembled the source imagery and prepared the seamless dataset for OIT.

The northeastern quadrant of Rowan's Glassboro Campus in 1974.

The northeastern quadrant of Rowan's Glassboro Campus in 1974.

Same extent, but from the 2007 statewide imagery.

Same extent, but from the 2007 statewide imagery.

What I find amazing about the data is how clear the imagery is down to very large scale.

Detail of the parking lot north of Mimosa Hall

Detail of the parking lot north of Mimosa Hall.

And just for fun, here’s an animation of the two images tweened, so you can see the amount of change on our campus in 33 years.

I checked EarthExplorer and there were several single aerial images from the 1950s of the area around our campus, but they aren’t georectafied and mosaicked. I may try to stitch them together so we could have images of the campus and ‘Boro from the 1930s, 1950s, 1970s and 1990s along with our more recent imagery.

Posted in Aerial Photography, Data, Government, Technology | Tagged , , , | 1 Comment

Map Maker for carto-masochists, part 2

After failing to provide data straight from GIS – which is possible using OpenStreetMap or ESRI’s Community Base Map programs – I sat down with Matt, my student intern that is working on developing the campus data in Map Maker. I instructed him to use our Facilites GIS database as a reference for populating the attributes for Map Maker and to have a polygon as a reference to supplement the Google aerial photograph. He began adding in features, but quickly ran into issues with the review process behind Google Map Maker edits.

Map Maker has a moderation process for all of the edits made by Map Maker users. There are two big issues I see with the manner in which Google has implemented moderation. First, you cannot re-edit (or even touch) new features until they have been moderated. That means that if you are adding a development (or a series of roads on a college campus) you cannot link up the roads that have been previously added and awaiting moderation. The moderation puts a serious crimp in attempts to push out many edits in a small area. Mike Dobson has an incredibly detailed write up on the issues he faced in trying to correct data using Google Map Maker. A very interesting read on how difficult it is to contribute minor fixes.

The second, bigger issue is the moderation itself. Crowdsourced moderation relies on people from potentially anywhere in the world to weigh in on edits. Now, it is likely that most of the edits occurring in the United States are being performed by individuals that are familiar with the area; they’re locals. Locals are the best source of information; they have intimate knowledge of the area and they are able to readily field-verify the information in the system. So we have a pool of local users making edits, but the moderators may not have any prior knowledge on an area in which they moderate.

Here’s a perfect example of the issue with this form of moderation. Google provides an attribute field for buildings that records the construction material. The field is restricted to an enumeration set of four allowable values: “Concrete”, “Mud/Stone”, “Steel” and “Wood”. Several of our campus buildings were held in moderation limbo, as a reviewer felt that my choice of “Steel” was incorrect and should instead be “Concrete.” Why would a reviewer feel the need to question an attribute and hold up the data creation process? Mind you, the values in this field does not impact the manner in which the building is displayed on the map, so it is not critical to know before the feature is drawn. What could a moderator glean from an aerial photograph that would justify questioning the local editor? I know the buildings are constructed using a steel structure and they are faced in brick and stone – I can walk up and touch the building; go inside. I spend hours inside these buildings each work day, but I need to provide a moderator elsewhere some justification or proof to have the edit approved.

Matt drew in Rowan Boulevard Apartments, a building with a rather complex footprint, as the building has a series of setbacks dividing up the residential suites inside. He drew it in Map Maker as close as possible, similar to how it appears in OpenStreetMap. This edit was then altered and poorly generalized. The moderator informed Matt that it should not reflect the roof line. There are no stepbacks and the building is near the nadir of the photo, so the roof line is awfully close to the actual building footprint. He spent a considerable amount of time drawing in a feature to have it simply blown away by some one that likely has never been near the actual structure.

Compare the photo to what Matt was finally able to have approved.

The moderation system in Google Map Maker is broken and will continue to be if local users are not given the benefit of the doubt regarding their edits. OpenStreetMap has the ability to revert vandalism or poorly edited features. Know why people don’t vandalize OpenStreetMap? It’s often a pain in the ass to commit large amounts of vandalism on OSM. Unless you’re a programmer, committing large bogus edits to OSM isn’t easy to do. And on the occasions when vandalism occurs on OSM, it is often corrected by local users. Google’s fear of having bad data on Google Maps (even though so many errors pop up all the time through erroneous business entries scraped from search indexes) is hindering the ability of users to contribute to their platform. Google stole several concepts from OpenStreetMap but not the ones that can elevate it to a platform that could truly produce credible maps from locally-sourced volunteered data. The only thing going for Map Maker is that eventually your edits will appear on the main Google Maps site.

I got involved in Map Maker solely to get Rowan’s campus data on Google Maps. I cannot pull data back out of Map Maker, so after I achieve my goal, I will likely never use Map Maker again. Why deal with all the frustration and reap none of the rewards?

Update: Google Maps routes park visitors down a residential street that doesn’t actually provide access to the park, frustrating the local residents. Also, Google Maps Fail on Tumblr.

Posted in Data, Google, Google Maps, OpenStreetMap, Uncategorized, Web Mapping | Tagged , , | 1 Comment