Mapping Jersey City

When I went to college I would be away for a few months at a time and I would come home for a holiday to some inevitable new neighborhood feature. I'd be grateful for some of the changes: a local coffee shop and a bar or two. These welcome changes came with unwelcome ones like traffic and an obtrusive red, white and blue "LOSE WEIGHT TODAY" plastered across the side of the formerly endearing plumbing supply building. Typical bus shelter ads for abogados or liquor-on-demand were replaced one by one with smiling white faces of realtors promising to make this great-commute unbeatable-views neighborhood your home. Come discover Jersey City.

I became embittered to the initially welcome changes. I would head home for a summer ready to be reunited with familiar views and musty smells and sounds of "la proxima" before the next stop on the immy. Instead I would find that the overlook where I used to be guaranteed gazebo solitude on weekend mornings was bustling with sun-saluters and farmers' market shoppers. I hadn't seen any of their faces back when the park was treacherous with dog poop and empty plastic Smirnoff bottles. But places don't belong to people.

One thing I came to love about the city I grew up in was its stoop culture. As soon as it wasn't winter anymore every block would have a handful of stoops with the whole building sitting outside. Kids might be chasing each other and when one ran into the street they would be dragged back inside by their shirt-backs. If a boy called something after me while I walked home from the bus stop there would be an immediate slap on the side of his head from some aunt or mom or grandma or sister - the same matriarch who always had pieces of advice to shout after me. To a bloody skinned knee after soccer practice on sharp turf: "better wash that knee good!" Hoping for spring in late February with a thin denim jacket: "better put a real jacket on!" Stumbling down the block after midnight over the summer before college: "better hope your daddy's asleep when you get in!"

My family never sat out on a stoop (we also never had one). We had two cars. We set an alarm before leaving the house in the morning. I would take home friends from college for long weekends and they would comment on how their childhood homes seemed so safe by comparison. Parents friends, with good intentions, would remark how bold and progressive my parents were for raising us in such an urban environment.

But Jersey City taught me the lessons that the Ivy League never did. They were not always rosy. A summer night mugging left me skeptical of men with hoods dropped low over their faces at night, but ever more so of the police who insisted that I was covering up a case of domestic violence. However uncomfortable, these lessons always stuck with me, like how I was startled to find that my friend's parents who couldn't find work that wasn't behind a cash register both received doctorates in medical imaging before immigrating. The kid in my geometry class who always seemed obtuse spoke three languages, of which English just happened to be the last one. And one of the prettiest, most enviable girls in my year shared a bedroom without a door with her two younger brothers.

I've struggled for a long time to solidify my thoughts on my changing neighborhood and the stakes are always higher with pen on paper. It frustrates me to see so many new faces that don't reflect the diversity that defined my childhood. I don't know the names of my old neighbors but I do know that their late-night summer laughter, catcalls and old engine groans always sound like home. I know that my memories of my childhood are rich with pungent dishes, tiny apartments with hardworking families, fire hydrant sprinklers, and laughing at obscene graffiti scrawled across our garage door.

I wonder if I am entitled to resent my new neighbors who look just like me. Am I wrong for romanticizing and reminiscing chain link and cracked sidewalks and abandoned properties? Did my neighbors ever feel all this about me?

I could research increasing rents. I could find cases where residents have been pushed out to make way for condos. I could count the number of spots in my public magnet high school that now go to kids whose families might have sent them straight to private school if they weren't accepted, but my frustration isn't that tangible. It just saddens me that the landscape from which I like to think I extracted my favorite pieces of myself - grit and a little delinquency - is changing.


Unfortunately the data on Jersey City is sparse. The US Census Bureau and American Fact Finder have a wealth of data on demographics and housing, but it's all aggregated up to the tract level and I'm looking for something more granular. I don't have a hypothesis to test here. I just want to be able to capture my neighborhood as it is now (or was not too long ago) at the block and building level so that I can track how things change over time. 

Even with that simple goal in mind, this is just a first step. I've digitized all of the building footprints in the Heights and a select number of other features like sheds, pools, parking lots, and parks. I'm gradually classifying the buildings by their use, with the help of some of my dad's planning surveys (he's an architect who's been helping the local neighborhood organization for a number of years). I'd like to be able to classify all of the buildings by height and age too, and then join the buildings with assessor data for ownership and evaluation information, but that will be the next step. For now, I've simply digitized all of the buildings, put them into Open Street Map so anyone can use (and improve) them, and done some basic classification by zoning and land use. I'm sure this project has a long way to go, and I've tried to document the process below so anyone can do this for their neighborhood. More soon.

See the full map. | Get the data.

I'm going to be quick with a lot of the details here because most of this stuff is already well documented. The tools that I worked with are OSM's iD-editor, JOSM, the JOSM BuildingTools plugin, QGIS, and the qgis2web plugin. I'll describe how I used these tools at each step of the process and provide lots of links, but these tools have great supporting resources already so I won't reinvent the wheel. With these tools you'll be able to draw as many features in as many places as you want and stick those features in a responsive web map. However this process is laborious. Even with helpful plugins that speed-up the drawing process you would have to be crazy to trace the buildings footprints for an entire city. I've been working on some computer vision algorithms that automate this part of the process and I did use them here. Unfortunately, I am an amateur so the models are weak and I did a good amount of manual work on top of them. They also perform much better on some imagery sources than others. I hope to improve and release that code soon, as soon as its good enough to digitize the entire city and be applicable to other areas.

1. WORKING WITH OSM & JOSM

Open Street Map (OSM) is the Wikipedia of geospatial data. It's just a database filled with delicious data under the Open Data Commons license so anyone can contribute and anyone can use the data, unlike stingy Google Maps. Only certain features belong in OSM and everything that goes in must be verifiable and observable with the eye. So street centerlines and building footprints are appropriate but parcels are not. There are two ways to contribute to OSM that I used for this project: the iD-editor and JOSM. The iD-editor is in your browser and adding data is as easy as drawing on top of satellite imagery (once you make an account). Here's a presentation I made for a Crisis Mapping workshop that I gave at Stanford on GIS Day 2015, with lots of good links for getting comfortable with editing using the iD-editor.

The other option is JOSM, a desktop application for editing OSM. The OSM Wiki is really good, and if you've never OSM'ed before you should check out their intro. You can download JOSM here. You'll need the latest version of Java too. When you've created an account and downloaded JOSM open up the application. Click the green arrow button to download the existing data (if there is any) for the place you want to map. Pick a small area (about a block to start). When you're data is downloaded from the server, click on Imagery at the top of the application. Check out which imagery looks most crisp for your area of interest and select it. I've been going with Bing exclusively for my US mapping projects. This could be entirely coincidence.

Use the draw tool to make some lines by dropping nodes. You can read more about drawing here. I used the BuildingTools plugin, which will dramatically up your speed. To do this, go to Edit -> Preferences, and select the Plugins tab.

2. USING THE TRACE BUILDINGS FEATURE

Click on Download List to see all available plugins. Check BuildingTools, click Download, and restart JOSM when it's complete. Download data and imagery for your area of interest again.

Most of the buildings in the Heights are simple rectangles with a small extrusion, like a vestibule or bay windows. The BuildingTools plugin has an simple tool for drawing rectangles rapidly. Here's a handy Mapbox blog post on tracing buildings fast. The default tag for buildings you create will be building=yes, but you can change this in the "Advanced" dialog.

Click the up-facing green arrow (next to the download button) to upload your edits periodically. If you can't make out a shape, try switching the satellite imagery up by source. Do anything other than checking Google. Literally anything.

I went back and forth between JOSM and the iD-editor for a bit (mostly because I found iD easier on the go). I used the iD for some of the first pools I traced. Some helpful shortcuts in iD after you've drawn a polygon are: O - creates a circle from your points; S - squares your corners; R - rotates on a centroid; M - moves object.

Related: I discovered that Pascal Neis created an awesome way to track your OSM contributions. I looked up my contributions and found some fun stats. No big surprises here that on account of this project the vast majority of my edits are buildings. Also, my very first node wound up (again, by no coincidence) smack in the middle of this project area. And clearly I have a night mapping problem.  And it's not looking great for my social life that I map the most on Thursday and Friday nights. The good news to report to my boss is that my OSM mapping low is between 8am and 4pm. Moving on: 

osm-contributions2

When you're done digitizing your area of interest, get your data out using one of three services. OPTION 1: In JOSM just save your project. It will export as a .pbf (protocolbuffer binary format) which is the OSM shapefile type. OPTION 2: On the OSM webpage, go to export, draw a bounding box around your area of interest and click export. This will also get you a .pbf. OPTION 3: Go to bbbike.org and get either a .pbf  or an .shp of your area. The only downside with bbbike is they only grab OSM data every few days, so your most recent edits won't be reflected. The great news is that QGIS lets you directly import .pbfs as layers, so this shouldn't be much of an issue. I only use bbbike when QGIS is being fickle about importing .pbfs (this happens sporadically - more below).

3. QGIS

QGIS is open source mapping software. It's just like ArcMap (almost) but free! The download is a bear, but it's worth it. Find the right download package for your machine here. You can find the downloaded for Mac here, but note that you'll also need to download all of the Python Modules listed here before running QGIS. Also, you want NumPy and SciPy anyway if you ever hope to have fun with data and python. 

Import your .shp or your .pbf - here's a 50 second video on how to do it. If you're working with a .pbf you'll be prompted to select whether you want to load your points, lines, or polygon layers (that are all stored in this one shapefile bundle). Select polygons, but note that at this point all of your polygons will be saved in one layer. You can extract these later using the "select by expression" tool. For now, load them in (ignore the lines and points in your .pbf for now). Right click on the layer in the layers toolbar and click Save As. Save your layer as an ESRI shapefile. For some reason QGIS it really picky about working with .pbfs like I mentioned. Just save yourself the hassle and load in your new .shp - or follow this GIS StackExchange thread and figure out the answer - and select add layer to map.

When this is done, right click on your new layer in the layers panel and click Attribute Table to inspect its fields. Identify which features you'd like to extract by the columns and their values. In my case, to extract building polygons (to have them as one layer separate from other polygons like land use, parks, etc.) I used the select by expression tool (View -> Select -> Select By Expression) and queried all objects where "building" = 'yes' OR "building" = 'commercial' OR "building" = 'retail' OR "building" = 'residential' OR "building" = 'church', etc. Similarly, I later used a query that looked like "leisure" = 'swimming pool.' You can also manually select multiple features for your new layer by freehand. Just hold the ctrl key while selecting. When these features are selected, right click your layer and save a new shapefile for buildings or pools or whatever, but be sure to click the "only save selected features" and "add layer to map" buttons. Repeat this as many times as necessary for all of the feature layers you'd like in your map. Style your layers to taste.

I took things one step further and downloaded the city zoning map here and as a shapefile here. Layering the zoning map behind buildings looked zany (as you can see above), so I decided to color individual buildings according to the zone they fall into. There are a number of ways to do this. You can select by freehand and just grab the buildings in each zone. I first generated different layers for each zone I was interested in by querying the "zone" attribute, for example:  "ZONE"  =  'C' OR "ZONE"  =  'C/A' . I saved separate layers for each zone and redevelopment plan area. I then went to Vector -> Research Tools -> Select by Location in order to create a new selection of all features in my building layer that intersected features in the zoning layer, and then saved these again as new building layers. My way was cumbersome but I wanted separate layers for each zone.  Other solutions to the same problem would be to use the QGIS intersect tool or to do a query in a Spatialite database.

4. QGIS2WEB

qgis2web makes it easy to turn your shapefile layers in QGIS into a responsive webmap built on Leaflet without any coding. I first learned about it on GIS Lounge. It's a plugin and plugins in QGIS work the same way as in JOSM. Go to Plugins -> Manage and install plugins. Check qgis2web and click install. Once installed, go to Web -> qgis2web. Set your preferences. You can select which layers to display and which attributes (if any) you want to visible in a pop-up for each layer. You should then set your precision (recommended 10+ for building level objects, and still this leaves some resolution to be desired), set the size of the canvas (I went fullscreen), and what other map features to add. I added a layer list and a Stamen toner basemap. Set your export file path and click export. Opening the index.html file that is generated in Chrome should be sufficient to inspect your map. 

I edited my map a little further from here. I opened the index.html in my preferred free code editor - Sublime Text 2 - and made a few changes. I edited the text in the layers list and in the popups to make them more reader-friendly. I also deleted some of the attributes that I decided didn't need to be listed in each tooltip. I added a little link to my site in the attributions. I reordered the features in my legend and then I added links to the local redevelopment plan areas that affect the zones in my map. Then I took the whole file produced by gqis2web and drag-and-dropped it into a new github repository. And this point, it looks like this. Step one complete.


A few problems

This map looks weird in a lot of ways for a lot of reasons. For example, it looks like some of the buildings I drew are out of place (see below). Notice how the polygons overlap with Cliff Street. I looked into this and Cliff Street was incorrectly drawn in OSM. In the real world, Cliff stops at JFK Boulevard and does not connect with Collard Street. I corrected this in JOSM and uploaded my edits but I'm not sure how often (if ever) Stamen updates their toner basemap tiles to reflect OSM updates, so this could look weird for a while. 

I saw the same issue with Perry Street and thought there might be a similar issue (below), but looking into it through different sources of imagery it became clear that Perry Street really does run all the way through to St. Paul's, and that my building polygons don't appear to overlap with Perry as I've drawn them. It's hard to say whether this is an issue caused by Stamen basemap tiles that were generated using outdated street data, or if there's a more insidious misalignment issue happening here. 

Yes, my wide double screens are the bomb. On the left is a mockup of the map. On the right is the same area zoomed in on with the OSM id-editor. Perry Street is selected.

Finally, I mentioned that precision has been an issue. I set mine at 15 - the maximum in qgis2web - but there are still some resolution issues to say the least. On the left are a few pool objects shown at zoom level 19. Two of the same swimming pools are shown at zoom level 22 on the right. Clearly, the features have much better resolution (and proper circular shape) when they are loaded to their fullest capacity. 


Next steps

My next steps are to find assessor data for the city or county (and hopefully I won't have to digitize it). I can then join buildings onto their parcels to map the Heights by valuation, ownership patterns, and whatever other data is available. I'm hoping that data like this can help track new development, how patterns of ownership and leasing are changing, how density is being affected, how uses are shifting to reflect the new population, and more.

I'm also focused on getting the model that digitizes building footprints from satellite imagery up to par so that I can release it. Once it's good enough I'll use it to generate buildings for the whole city and then open source it. In a dream world, I would then build that into a tool for JOSM and QGIS. Finally, when the data is good enough, I'm hoping to share it with the city. I don't think the data I've generated could possibly be less desirable than the reams of .pdfs on the open data portal currently. 

I hope this was helpful, I appreciate feedback, and I'd love any recommendations for where to take this project next.