Using the XAPI, I was able to pull just the relevant information from OSM. In this case, I only wanted nodes that contained a highway key with the value of motorway_junction. I used MapQuest‘s XAPI service and quickly generated a URL using a nifty map mashup. I had to manipulate the XAPI URL manually to change the bounding box; New Jersey was too large to search with a statewide bbox, so I broke the state into four portions by changing the latitude ranges. Making smaller requests for the part of the state between latitude 39 and 40, then 40 and 41, allowed my request to go through without timing out.

I was then left with four XML files that contained the highway interchanges. ArcGIS doesn’t know what to do with this, so I whipped up some Python to convert all four tables into one tab-delimited file. This code (included below) can then be used to add the points to ArcMap using the Add XY Data functionality.

All said and done, this project took me about an hour. I was able to send the data off in about 30 minutes, but then spent another thirty minutes cleaning up the script, modifying it to spit out any available key, not just the “ref” tag. The script is GPL, free for you to use as you see fit. Just remember to give credit where it’s due, attributing your data back to OSM.

Taking the time to do this for a friend is a good reminder of why I push our students to do more than just the GIS curriculum. Being familiar with OSM gives you a different point of view on GIS data, schema, and community than what you would gain from most GIS courses and entry-level positions. GIS is already about problem solving; being able to do some basic programming will enable you to tackle much larger problems efficiently. And providing technical assistance to others just makes you a better person.

© John Reiser for new jersey geographer, 2012. | Permalink | No comment | Add to del.icio.us
Post tags: github, mapquest, OpenStreetMap, python, scripts, xapi

detail of the NJ Statewide Bicycle Map.

This Thursday, May 10th, I will be attending the final public meeting to discuss the draft New Jersey Statewide Bicycle Map and Resource Guide. I attended the first meeting at Cumberland County College, but will be again providing comments at this final meeting to be held at NJ DOT Headquarters. I’m very much a single-use public commenter, as I really do not have many concerns about the map at all; frankly, it’s beautiful and I plan on getting copies as they are data-rich and rendered by an excellent set of cartographers. My concern is the data-richness – this map is detailed, and for DOT or the consultants to not release the underlying data along with the maps would be a sin.

Let me take a step back and explain how I first found out about this project. Dave Heller from SJTPO posted an email to the New Jersey Geospatial Forum discussion list, asking the group if there was any bicycle facility data that was publicly available and more recent than 2007.

Hello:

We have a bicycle/pedestrian GIS layer from 2007.  I wanted to find out if people here are aware of anything more recent.  Specifically, we are interested in South Jersey.  Please let me know if you are.

Thanks.

David Heller

Being the great resource that it is, within 10 minutes another member of the NJGF mailing list, Robert Blash, pointed Dave to the Statewide Bicycle Map project web page.

Take a look at http://bikemap.com/njbike/

NJDOT has a project to map out biking areas in NJ.

Robert Blash

I was in the middle of writing an email to Dave & the list about using OSM data for bicycle facilities when Robert’s email came in, so I took a look at the site. While I was glad that DOT is looking to create a statewide cycling map, I was a little dismayed by what I saw on the site. My email to the list follows:

CloudMade has OSM-sourced data available as ESRI Shapefiles.
http://downloads.cloudmade.com/americas/northern_america/united_states/new_jersey#downloads_breadcrumbs

You can see what bike/ped facilities are in place using OpenCycleMap: http://opencyclemap.org/
If you take a look at Glassboro – http://opencyclemap.org/?zoom=15&lat=39.70289&lon=-75.10368&layers=B00 – You will see the Glassboro-Monroe bikeway and the pedestrian connections throughout campus. Some of our bike racks are on OSM, but not all of them.

Regarding the bikemap.com site, I contacted DOT a few years ago regarding the old Bike Maps that were released, looking for the underlying data. The maps were produced by the RBA Group. DOT informed me that the PDFs were the only deliverable and that DOT did not have ANY geospatial data related to bicycle lanes, routes and facilities. bikemap.com is run by RBA Group. Will this project be no different and community-sourced bike data be locked up by a private consulting firm?

You can register for the site and leave comments. Please take a few minutes and let them know that the GIS data behind the map should be made publicly available. Comment #2 is what deliverables are important; click “electronic file”. For question #4, comments, I left the following:

“Please make sure that all data collected through bikemap.com relevant to the production of the cycling routes and facilities is made available to the citizens of New Jersey by placing the data online under an open data license or released as public domain.”

If this project is funded by both NJ DOT and US DOT, the results of it should be open and accessible.

Peace, love, and open source,
John

Merrilee Torres, from Burlington County, replied to the list mentioning that Steve Spindler Cartography (who owns bikemap.com) is involved in the project and that echoing my concern for public release of the data, as the project managers, RBA Group, solicited the counties for bicycle facilities data.

My understanding is that Steve Spindler Cartography is doing the mapping for the state bike map but yes, RBA is managing it.  I do know that they were trying to collect as much source data from Counties as possible which gives even more credence to the idea that the data should be publicly available.

Merrilee J. Torres, GISP

I then replied to Merrilee and the list, expanding on my position on government and open data:

I don’t want to suggest that the State (if it’s even legally allowed to do so) adopt some sort of license like the GPLv3 where derivative works are required to be released under the same terms; open and redistributable. If that were the case, it could be argued that anything sourced off of aerial photos would then need to be released. There’s a massive amount of value in having the State’s data open, however it does seem like the public is getting short shafted when public data gets vacuumed up and then locked away. There is some value-added in performing the work that merits compensation to the consultant, but if the State is the one footing the bill, it should be stipulated that the product is owned by and distributable under whatever terms the purchaser/the State deems fit.

The paper NJ Bike maps are going to be great to have. Let’s face it, while many GIS geeks seem to also be biking/outdoor enthusiasts, the reverse is far from true. The public is going to need the paper maps. If the data is released, there’s great opportunities for the GIS geeks to perform some analysis that either improves the data further or could help influence policy relating to bicycle use in the state. Open government data allows many great ideas and projects to come to fruition; producing data, research and applications far beyond the scope of the original project.

And if the data was already there and freely accessible, Dave Heller would have found it and y’all wouldn’t have been subjected to my open data rants!

John Reiser

The conversation then took a couple turns back on track, to locate some additional information for Dave, then died out. Then a month or so later, I received notice of the public meetings. On April 24th, I rode my bike to work, checked out a ZipCar and went down to the CCC campus for the public meeting.

The meeting was conducted by two staff members from RBA and a staff person from Baker. After explaining the process and the methodology used to rate the state’s roadways for bicycle suitability, they opened it up to comments and questions. I asked about the intent of the map; “Who is your audience?” was met with a “Yes,” followed with an explanation that they intend the map to be useful to anyone.

I voiced my concern with such a sentiment. Paper maps are not going to serve everyone. While most people cycling for recreation are almost guaranteed to benefit from the map, it’s still in paper form. There’s a cost to print and deliver. And even if you provide PDFs, there’s few opportunities for individuals to print large-format maps without a visit to a printer.

Cycling is a key part of commuting, whether it is the sole mode or used to make connections, like above. About 50 bikes were locked outside of Harrison's Path station on a rainy weekday.

Beyond that, what about those that bike because it is their only available mode of transportation? Are the maps and guidebooks going to be available to them? Are the books only going to be available in English? Take a visit to nearly any suburban chain restaurant and you will see a few bikes locked up in the rear, surrounded by a sea of automobile parking. Those bikes don’t belong to a suburban family of four, they belong to those cooking the meals and washing dishes.

If the data behind the maps are made available, the utility of this endeavor increases exponentially. Paper maps serve a purpose and are nice things to have, but in terms of providing for cyclists, planners, social workers, public advocates, and geeks like me are not going to be able build upon a PDF map. Providing the data in a free (libre, not gratis) form enables so many ancillary players to tap into the resource provided and expand its benefits to a larger audience.

Jersey looks devoid of regional connections on OpenCycleMap. I've personally added some smaller tours and bike paths, but there's much more left to add.

I’m attending the meeting at DOT mainly due to the response I received from the individual from Baker regarding the previous set of bicycle maps. When asked about why the data behind the previous set of maps was not made available, the answer I received was that the maps “were done in Illustrator.” Yes, the set of maps (available to download at no-cost from the link above) were likely modified and aesthetically improved in Illustrator. But only Illustrator? Considering the state has GIS data for the millions of miles of roadway in the state, why was only Illustrator used? Fine, forget the old maps for now – the new maps have the tours available on them, along with all of the roadway condition information. There is no possible way that all of that information was managed effectively outside of GIS. And if it was an entirely Illustrator produced product, why is the State paying consultants to be inefficient?

I was told that the data behind the new maps is definitely GIS data, but that the staff of DOT will ultimately decide if the data will be released. First, of all, I intend to submit an OPRA request for the data if it is not released when the project is deemed complete and the paper maps & PDFs are released. Secondly, why wouldn’t DOT want the data released? They already provide GIS data and maps in multiple forms for any use.

I spoke with Elizabeth Cox from RBA after the meeting and she encouraged me to provide written feedback. I will likely print this blog post (how quaint) and will submit it at the public meeting. I may decide to write something more formal for submission and will update this post to include it if written. At the least, I intend to attend the meeting and directly ask the individuals at DOT what are the future plans for this data. I also encourage you to provide feedback via their contact form. I’ll provide updates after the meeting Thursday.

© John Reiser for new jersey geographer, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

© John Reiser for new jersey geographer, 2011. | Permalink | One comment | Add to del.icio.us
Post tags: HTML5, iPhone App, RULost, Rutgers

WMS: http://njwebmap.state.nj.us/njimagery

Civil Solutions was the contractor that assembled the source imagery and prepared the seamless dataset for OIT.

The northeastern quadrant of Rowan's Glassboro Campus in 1974.

Same extent, but from the 2007 statewide imagery.

What I find amazing about the data is how clear the imagery is down to very large scale.

Detail of the parking lot north of Mimosa Hall.

And just for fun, here’s an animation of the two images tweened, so you can see the amount of change on our campus in 33 years.

I checked EarthExplorer and there were several single aerial images from the 1950s of the area around our campus, but they aren’t georectafied and mosaicked. I may try to stitch them together so we could have images of the campus and ‘Boro from the 1930s, 1950s, 1970s and 1990s along with our more recent imagery.

© John Reiser for new jersey geographer, 2011. | Permalink | One comment | Add to del.icio.us
Post tags: Aerial Photography, historic, imagery, OIT

Way, way back in March of 2010, I contacted Google through their Map Content Partners form to submit Rowan University’s GIS data to be included on Google Maps. I heard nothing from them – not even an automated response – for several months. In August of 2010, someone in Administration noticed that Google had relabeled Rowan University as Glassboro State College, a name not used since 1992. I was instructed to try and remedy this as soon as possible. I explained that Google had given me the cold shoulder, but I would try some different avenues. The “report a problem” feature in Google Maps did get “Glassboro State College” off the map, but Rowan remained a featureless blotch within Glassboro.

After giving up on contributing data to Google Maps, I was finally contacted by “Brandon” from the Cities in 3D Team. Brandon requested a sample of the data for review. “Finally!” I thought, “now I can submit our data for Google Maps!” I’ll admit, I got excited. In hindsight, the excitement was premature and irrational. OpenStreetMap has been updated to reflect the campus since June of 2009, when I used my Export to OSM script to dump the facilities data into .osm files. I should be focusing my efforts on a custom, Mapnik-rendered set of OSM tiles that look good on top of aerials, a “Google Maps Hybrid” layer of my own. I can then sandwich in custom tiles instead of overlaying my data on top of the annotation. (Compare anno-on-top to data-on-top.)

Anyway, I eagerly await word back from Google that our data will appear on Google Maps. The notice I receive was far from what I wanted to hear. Instead of accepting the data, I’m instructed that due to the small number of objects to be added, I should use Google Map Maker instead. I was pissed.

I find Google Map Maker to be an incredible disappointment on two major fronts; first, I refused to use the service as I felt Google was in the wrong for keeping VGI-sourced base map data under lock and key. The second, which I will elaborate upon tomorrow, involves the Map Maker tool itself. For a company such as “do no evil” Google to create a framework where users can volunteer time to build its GIS repository with no compensation – not even data retrieval – I find completely unconscionable. Compare their data download form with getting data from OpenStreetMap. To non-techies, OSM’s XML format might be intimidating, but it’s data – no forms to fill out or licenses to sign. Sure, it’s not a shapefile… but you can get them too, without signing any agreement. I loved hearing how a group of OSM mappers hijacked a Google Mapping Party (the concept itself lifted from OSM) and got the assembled group to work on OSM. Many others have asked “why Google, why not OSM?” The common thread to most Map Maker criticism and OpenStreetMap’s major issue with Map Maker is regarding access to the data that has been contributed. Why make data created of the third world (and likely by individuals in the third world) a for-sale product for a first world company? Even within your own neighborhood, why volunteer your time if you can’t reap most of the benefits?

Back from ideology to practicality, I wanted the University on Google, solely because Google is the new Mapquest; it’s the go-to map service for the average person. (Quick aside, Mapquest has Rowan University completely mapped out through its OSM-sourced open.mapquest.com.) Brandon’s earlier suggestion of Map Maker wasn’t acceptable because of my open-access advocacy leanings, it was out of time constraints. “You want me to redraw everything?” That wasn’t going to happen and I let Brandon know:

Brandon,

Thanks for the info. My major concern has been getting an “official” set of data on to Google Maps. When Google switched to their own data and pulled place names from GNIS, Rowan University reverted to Glassboro State College, a name last used in 1992. As “the map guy” for the University, I was asked by several offices within Rowan to “fix Google.” While reporting the antiquated name as a problem did get “Rowan University” restored on Google Maps, it brought the lack of campus data on Google to the attention of many in the University community. It has been frustrating for me, as I have attempted to contact Google to provide official data for use on Maps for a considerable time, while also attempting to squash bugs in the Google Maps data on our campus. Being a University, there have been “creative” additions to the campus on Google Maps, such as the fact that one of our largest academic buildings appeared as a Bed and Breakfast for about a month.

I would like to provide Google with a set of data out of our Facilities GIS DB that is the same source for our printed campus maps. The previously sent data, reformatted to match your requested schema, is from either survey data or from a 3″ planimetric image. The address information included in this layer matches Gloucester County’s 911 data. This is as official as it gets.

Google Map Maker is not a viable option for me. The data for Rowan University that is available through OpenStreetMap  was generated through an export utility that I wrote. It is the same source as I emailed you previously, except that it dates from the summer of 2009. Very little of what is on OSM was hand redrawn. Not only would it be a considerable time sink for me, the tools in Map Maker would not be able to reproduce the high quality data that already exists.

Through the Geovisualization course I teach, we are offering students the opportunity to build SketchUp models of the campus as a final project. Several of the buildings have been completed and I will be uploading them soon. The entire campus is not complete, so if you would like a shapefile with extrusion data, I’d be happy to resubmit the data with those additional attributes.

Again, I would like to thank you for taking the time to respond to me. I understand that the Cities in 3D team might not have the 2D maps as their major focus, but if you could direct my request to the correct group, I would be incredibly grateful. I initially filled out the Data Provider form in March 2010 and have resubmitted the form several times after receiving no response. I even pestered Ed Parsons over Twitter to no avail. So, I am truly appreciative of your assistance in this issue.

Thanks,
John

Brandon responded with two points that made me bristle. First, Rowan was too small for Google: “the number of buildings on campus provide a relatively low priority for Google.” This is a 215 acre campus with 50+ structures, it’s not huge, but it’s also not insignificant. Second, I was encouraged “to do as much as you can on your own, or as part of
your class with the students help.” No one volunteers my cheap labor for projects other than me, buddy.

Through the reconfiguration of our program offerings, Geography and Environmental Studies students are required to have an internship or directed study as part of their capstone research experience. I was able to find a student that really excelled at contributing to OpenStreetMap and asked him if he would be interested in working on updating three map services: Google (via Map Maker), OpenStreetMap (via JOSM) and ESRI’s Community Base Map via (ArcInfo for Advanced Desktops). He agreed and began working on updating Google Maps. I finally bit the bullet, signed into Map Maker and started working with him on some of the edits.

Tomorrow, how Map Maker can only make bad maps.

© John Reiser for new jersey geographer, 2011. | Permalink | 5 comments | Add to del.icio.us
Post tags: Google Map Maker, Google Maps, OpenStreetMap

One interesting project that has popped up recently is NJ’s Broadband Map. You can dig through the data to find which areas of the State are served with broadband Internet. I think the color scheme needs to be rethought, as dark green to light purple is better suited to divergent or bimodal data, not to a linear progression of 1 to 8 broadband providers.

I’ll be at the ESRI DevMeetup tonight in Philadelphia, giving a lightning talk on the Land Change Viewer app I developed. Next week at the NJGF, I’ll give a 5 minute talk on the ArcGIS license monitor application I’m writing. I’ve got a couple other tidbits in the pipeline and hopefully I’ll be able to share them here in the next few weeks.

© John Reiser for new jersey geographer, 2011. | Permalink | No comment | Add to del.icio.us
Post tags: Broadband, DevMeetup, New Jersey Geospatial Forum, Rowan University

© John Reiser for new jersey geographer, 2011. | Permalink | 5 comments | Add to del.icio.us
Post tags: ArcGIS, Development, DevSummit, ESRI, GIS, mobile, Software

It’s been a crazy few weeks, juggling several research projects, two classes and compiling everything needed for my recontracting at Rowan. The last two weeks have been especially tasking; even without the occasional curve ball, I still lacked the hours in the day to get everything done. I need a break, but before that can occur, I need all the boxes checked on my to-do list. Tomorrow and Friday are going to be a blur of activity in attempt to get everything wrapped up by the week’s end.

Next week, I’m fortunate to be heading out to Palm Springs for the ESRI DevSummit. I’m really looking forward to learning more about the latest web mapping technologies. When it comes to server-side GIS, the Windows/.Net-based “ESRI Stack” is still a little foreign to me and it’ll be good to see how I could incorporate some ArcGIS-based technologies into my open source-based map services.

I’m also hoping to meet up with a few GIS folks I’ve followed on Twitter. While individuals at ESRI have been incredibly helpful with specific issues, I’ve found their online documentation to be lacking. There are several ESRI staff and third-party developers whose blog posts about an issue or a technology have clarified things for me and I’d like to let them know that in person.

Also, the DevSummit is the week before Spring Break. I’m not one to cancel class, so I’ve put materials online for my students and will likely Skype back in during class time. The question is, do I video call from my hotel room or do I conduct class poolside under a clear blue sky? Decisions, decisions.

© John Reiser for new jersey geographer, 2011. | Permalink | One comment | Add to del.icio.us
Post tags: ArcGIS Server, DevSummit, ESRI, Teaching

© John Reiser for new jersey geographer, 2011. | Permalink | 3 comments | Add to del.icio.us
Post tags: creative commons, presentations, Teaching

More on NearMap from Slashgeo.

© John Reiser for new jersey geographer, 2011. | Permalink | No comment | Add to del.icio.us
Post tags: Aerial Photography, Australia, Brisbane, flooding, Queensland