Pages

Showing posts with label google maps. Show all posts
Showing posts with label google maps. Show all posts

Redistricting: A Question and a Query

Wednesday, January 15, 2014
Alert: Both policy AND technology will be discussed below. Bring both your brains. You've been warned.
Gotta love the classics

The Question

My city is on the tail end of the country's redistricting cycle. Somehow while Texas was marginalizing Latinos and Dennis Kucinich was getting scribbled out of his constituency, we here in Burlington were waffling over how to deal with triparty politics and the implications of our changing neighborhoods.

I was unrealistic early on about how much of a role technology could play in the process. A survey of neighborhood geographic identity and a do-it-yourself redistricting app were helpful but not game-changing. After years of wrangling, the parties mostly just hashed it out in committee (though the final plan is derived from one made with our DistrictBuilder app), and now in about 6 weeks the voters will approve or reject it.

Which brings me to the question: What would the proposed plan change things for the average BTV citizen? It's a pretty easy one to answer in the aggregate - all the information is out there: new boundaries, old boundaries, population distribution, polling locations. But these bits are spread out across various sites and articles. What if there was a way to just ask the question: How will this change things for me?

Under the umbrella of CodeForBTV - the local CodeforAmerica brigade - I put together the basic answering tool. Built for phones (via Bootstrap), the app lets the user plug in their address and get back a list of redistricting plan implications:




The Query

The question of individual impact can be answered with a simple spatial query: return some pieces of information based on the polygons that intersect the user's location. This pointed to some basic building blocks:

  • The city redistricting plan - both the old wards and the new ones lumped into one geojson file
  • A robust search engine - I went with the Google Maps API because of the amazing viewport-biased geocoding and the built-in typeahead, but given more time I would use Leaflet and a state-hosted geocoder plus typeahead.js
  • Spatial query capability - fortunately CartoDB offers all the magic of PostGIS at a URL endpoint, with great styling available as well. 
The user-selected address gets converted to lat/lon, then sent to the CartoDB API in an ST_Intersects() query. It returns a map overlay of the old and new district boundaries at the site, and a set of answers to some FAQs about redistricting.

It's a simple app, relying on some robust APIs. I hope it's useful in getting my neighbors oriented to the landscape of the proposed redistricting. At the very least it'll save people some time peering at a large paper map in city hall, and at best it'll head off 40,000 individual "how will this affect me" emails directed at the city councilors and GIS manager.

I also hope it'll be of use to anyone else who wants to present geographic change over time at a user-selected location. Get the code on github.
Read more ...

On Reverse-Engineering a Map Stack

Saturday, January 11, 2014


I'm going to come right out and say this:

You should probably just use Mapbox.

How I came to that conclusion is a bit of a longer story.

The Scene

As a cartographer, I am an unabashed fan of Mapbox. I've been using Tilemill for years, and I love the fully-realized design of the Mapbox Streets basemaps. Even before Google Maps brought the paywall hammer down I was already migrating my clients' projects to the open-source ecosystem based out of a D.C. garage. Controlling so much of the stack in an open-source environment meant less risk to my clients, and in most cases it came out a lot cheaper. The choice was pretty easy, actually. 

Looking around this section of the market I don't believe that Mapbox and Google Maps have anyone to compete with but each other (mobile is another story). They are the only two companies currently offering a rigorous javascript API tightly integrated with attractive map services built for the web. (Debate this in the comments, ESRI, Nokia, and Bing users) There are plenty of a la carte options out there - Stamen's wonderful map tiles, the sheer power of the OpenLayers API - but as a lazy developer I've come to really like working with an integrated, open stack. Mapbox is my current choice, though many use cases all but require Google's services.

The Challenge

An open-source stack doesn't mean a free stack, and Mapbox's map tile charges can rack up quickly if you're not paying attention. Exhibit A: I offered Brandon Martin-Anderson one of my Mapbox tilesets to use as a reference for his Census Dotmap. Several viral weeks later I was looking at overages the size of my annual budget.

An open-source stack may not always be free, but in theory it can be copied and hosted by others. I've done this quite a bit on the javascript API side, mixing and matching tile providers with various client libraries. However, the costs that had begun to concern me were on the basemap end. So I set out to do what Mapbox flat-out encourages you to do with its wide-open codebase:

I would make and serve my own damn basemap. How hard could it be?

The Tools


I started with the open-source Tilemill template OSM-Bright, noting that there are some good examples of it in use out there. I grabbed the current OSM data, piped it into my local PostGIS database (note the hazards of adding many extracts) and spent some time turning it into "Geosprocket-Bright" in Tilemill:


Next came the heavy lifting. I exported a slew of regions to .mbtiles format; I would have loved to build a map of the entire world down to street zoom level, but I thought I'd start more realistically with a global map down to zoom level 9, then a handful of cities down to zoom level 17. I planned to put them on an Amazon S3 instance and tap them directly from the client library, doing an end-run around Mapbox hosting. Sounds pretty smooth, right?

Let's keep track of the time investment, shall we?

  • 4 hours to export everything from Tilemill
  • 6 hours to chop the .mbtiles into 1.6 million PNGs
  • 3 hours to consolidate them all into a single directory structure (because I was too dumb to do that in the last step)
  • 56 hours to push the tiles to an S3 bucket on a 20mbps connection
Obviously these are CPU hours, not billable hours - but it was still more than two days between when my map style was ready and when I could actually hook the map up to a browser.


The Verdict

This is a narrow case where I needed to roll my own tiles and serve them. My map included texture and custom fonts, which is beyond the reach of Mapbox Streets. The total filesize of my exports - even as .mbtiles - was 15GB; that translates to the Premium Mapbox hosting plan, and a whopping $6,000 flat fee per year. That's for five cities - the tiniest fraction of a world of tiles. It's not quite Google Maps Enterprise money, but damn. By contrast, it cost me nine bucks to get all of my map tiles into an S3 bucket, where I'll get billed something like half a cent per 1,000 map views.

But this process will be moot pretty soon anyway; the promise of Tilemill 2 is that you only need to bring your cartography to the table - Mapbox will do the rest of the work to get your style to the browser with vector tiles. They've only done this for a few testers while they're hashing out the details, but odds are there will be a public version in the first half of 2014. 

In keeping with other examples of open-source underpinning software-as-a-service, Mapbox has a sound business model. Despite the huge amount of intellectual capital they've open-sourced, it is still easier and - when time spent is considered - VASTLY cheaper to just use their hosted map services. I suspect they've been totally aware of this even as they release service-liberating tools like Tilemill, OSM-Bright and mbutil. The scale and efficiency of Mapbox make their hosted maps too good to avoid.

I hope my experience here has been instructive to others.

The Product

I did get a map out of my experiment, and it's free to use. If you happen to be mapping in Warsaw, Sochi, Santo Domingo, LA or the Bay Area, I hope it proves useful. Just use this XYZ tile scheme in your client implementation:

http://s3.amazonaws.com/geosprocket/tiles/{z}/{x}/{y}.png

As per the usual OSM license, be sure to include "© OpenStreetMap contributors". Happy Mapping!

Read more ...

5 Reference Recipes

Thursday, October 17, 2013


At the risk of beating a dead horse, I'm coming back to reference overlays. I've noticed that there a still a fair number of interactive maps online that permit a bullying thematic layer to stomp all over the poor, defenseless map labels and context information. This is not cool:



So I've hashed out the code for getting past this using five common basemap services:

Feel free to reuse them as you see fit. All but the Google example use Leaflet, but the z-index and canvas manipulation tactics are applicable to other APIs as well. If anyone wants to add adaptations for OpenLayers or whatnot I welcome pull requests. Also worth a note is that - in the leaflet examples - the thematic layer is rendered as vector from topojson, which is arguably the fastest vector format available to us today. Give it a shot as a thematic mapping tool.

Go forth and bring context back to your interactive maps!


Note: Thanks to Bobby Sudekum, Josh Livni and Jason Sanford for the starting blocks.

Read more ...

Tiled Basemaps Survey 2013 - Results!

Tuesday, August 6, 2013
From Stamen, OSM Contributors and National Geographic

The results are in!

123 helpful participants submitted their thoughts on their favorite basemap services, and without further ado I present some of the results:

Which tiled basemap is most cartographically appealing to you?



Google, National Geographic (served via ESRI), and Mapbox are the winners here, with support for Stamen's options as well. "Other" in this case included OSU's Crinkled Watercolor and OpenCycleMap. Between the Stamen, OSM-Mapnik and Mapbox selections here, tiles based on OpenStreetmap data were ahead of the others. But this is a popularity contest based on looks. Moving on to substance . . .

Which tiled basemap do you use most frequently in your projects?

Google wins on ubiquity. Almost half of the respondent pool uses Google's tiles first; though given the widespread adoption of the API this is not a great surprise. More interesting is that about half of these Google users find a different basemap to be more attractive. Which brings us to . . .

What factors drive the choice of basemap for your projects?


Respondents could choose more than one here, but it's clear that not many balked at the technical challenge or vendor lock-in concerns. Mostly it seems like choices were driven by a balance between aesthetics and quality/detail, with cost and licensing hovering in the back of respondents' minds.

What do you like best about your "cartographic favorite"?

Unpacking these results is best done with a series of [gasp!] word clouds for the top three choices. You might call it a word storm . . .

National Geographic: 

Mapbox: 

Google: 

Interpret these as you will - subjective assessments have a way of bleeding from one choice to another. For instance, it looks like everyone likes the "colors".

Whoa there . . .

This is not a scientific survey. And despite my best efforts to reach out to all the communities involved, it's probably not even terribly representative. But it's interesting to inspect the choices made by this particular group of mappers, and fantastic that so many were willing to share their thoughts. Check out the unvarnished results here - mostly interesting to see the full explanations of what folks like about their preferred aesthetic.

As always, drop me a line if you have any questions or concerns about these results. Thanks to all who participated!


Read more ...

Tiled Basemaps: the Survey

Monday, July 8, 2013


The time has come once again for a taking of the geopulse! In years past I've asked broad questions about platform-level technology use, but this year I'm getting more specific:


Yes, I realize that the very concept of tiles is about to go the way of the Avenue script, but the present moment is still flush with amazing cartography served up in square chunks. For this survey I've chosen a selection of "Streetmap" styles from the major players, since imagery is another can of worms entirely.

I've built a comparison app here (with thanks to Dair Grant for the idea). Check it out and vote for your favorite.

Note: It's not possible to get a perfect comparison, since some maps tiles are locked to a particular API (what up, Apple), and others were too hard to parse into a single web map (Was a quadkey scheme really necessary, Bing?). Feel free to add these in if they serve as your go-to basemaps anyway.

Thanks for participating, and check back here for results in a few weeks!
Read more ...

5 Examples of an Inflection Point for Maps on the Web

Monday, June 3, 2013
Some recent thoughts by Brandon Rosage reminded me that webcartographics (I should trademark that one) are the midst of a change. Since the dawn of Mapserver, we've been building maps into our websites in a segregated fashion, often treating the map as little more than an <iframe> keyed to a CMS. This works well enough in a short blog or location description, but there are more engaging stories to tell with cartography, and better ways to deal with UX complexity than just adding more markers.

Fortunately some great developers are steadily building tighter integration between geography and web design. My favorite recent examples of this follow:

1. Chasing the Boston Bombers

A maps-in-the-background piece on the Boston bombings from the New York Times Graphics Department uses a Google static map as a location-based table of contents to a scrollable storyline highlighted with more maps and photos.

2. The Royal Navy in WWI

From Vizzuality, a moving record of Allied ship traffic in WWI was derived from their work crowdsourcing the transcription of old ships logs. This is more of a show than an interactive piece, which works perfectly to illustrate the flurry of activity around major geopolitical events.

3. The Adventure of the Bruce-Partington Plans

Young Hahn's Mapbox-based adaptation of a Sherlock Holmes storyline changes the click-and-drag paradigm of the web map by tying map navigation exlusively to a scrolling text storyline. Unlike the NYTimes piece above, the map does move - but only as the narrative progresses. Bonus points that it's responsive, making this an equally-rich read on mobile.

4. Crowdmap's New Layout

Crowdmap has some great potential here, though it's still being pulled in several directions by varying stakeholders. The previous version was a codeless window into Ushahidi's functionality, and that was great for easy access to a map-within-a-CMS. But with the current iteration they've tried to bring the focus to a map driven by its content, rather than content accessed via a map.

5. Hurricane Sandy Coastal Impacts

Another entry in the the journalism category, this tour-de-force from Derek Watkins treats the coastal axis (North-to-South) as a narrative, attaching aerial photos seamlessly to a thematic map built with the D3 javascript library. Perhaps the coolest part of this one is what it implies - D3 is one of the best emerging tools for breaking maps out of "Plugin" status and letting them interact seamlessly with the architecture of web design. This is looking more and more like a window into the future of maps on the web.


Read more ...

We Are Still Talking About These Things: Dispatches from the Dawn of Crowdsourcing

Thursday, May 23, 2013


I was just having a conversation with some colleagues about the nuts and bolts of participatory mapping with some local farmers. We were all:
"Man, it totally depends on if you have internet at the site. If you do, just pull up Google Maps or CartoDB and have them digitize right into the iPad. If not it's a huge hassle; you'd have to annotate a PDF and georeference it later."
"Yeah, but either way the stylus is key. Non-technicians always want to use a pen and paper."
This talk of getting a skeumorphic data-entry device in the hands of the common folk reminded me of something from the past. This, specifically:
Even taking as a fact of life that community GIS will always require some mediation by the more technically skilled, Al-Kodmany's Chicago neighbourhood projects are not a little bit extreme. In fact, by the time he describes the community members being given 'coloring the map' participation exercises as a way to actually participate, there is a rather patronizing air about the project. 'Participants were broken into small groups and were given a map and felt-tipped markers,' he writes. Felt so as not to inadvertently poke out their own eyeballs, no doubt.
This from Christopher Miller, writing in 2006 about how participatory GIS (nee "GIS/2") was being hamstrung by academic condescension, cultural barriers and the failure of imagination that is traditional GIS ("GIS/1"). Man, 2006 sounds like a long time ago from here, but we're still spinning our wheels in some ways on this stuff.


Based on Andy Woodruff's Boston-oriented project, last year I built an app to get locals defining the boundaries of neighborhoods in the city of Burlington. When I showed up to the first hearings on city redistricting to present the results, their credibility was questioned by some who hadn't heard of the project, even though it had been announced in every local news outlet as well as digital channels. These were people who showed up to meetings, because that was how you participated. Not learning my lesson, I adapted Azavea's Districtbuilder app for the redistricting process and made it public, hoping "participatory technology" would open a traditionally-closed political world. It turns out that the technology wasn't really the problem.


In his snark-laden, brilliant 2006 paper, Miller goes on to describe this neat little app called Scipionus, built in the aftermath of Hurricane Katrina on the back of a newly-opened Google Maps API. Here's the thing: it let anyone with a web connection report their location and status on a Google Map! Fast-forward three years and Ushahidi comes along. Fast-forward another three years and Google Maps gets collaborative, Crowdmap is everywhere and Twitter becomes a news source. Fast-forward to last month and the most reliable information on a civil war is from a participatory map, and Ushahidi can take in reports from every technology short of smoke signals.

We still don't have good retorts to Miller's challenges. We still haven't made GIS accessible to the public, and I'm not really sure that's a valid goal. More importantly, we seem to be driving deeper into the gulch between Internet-Worshippers and Tech-Hostile Curmudgeons that Miller warned us about. Whether we're community-building, solving disputes or reporting the news, we've got to do a better job building bridges between the public and our maps.

In many cases, "Paticipation" is still a felt-tipped marker jabbed into the eyeball.

Read more ...

Geoflow: A Hasty Review

Friday, April 19, 2013

Remember those Microsoft guys?

Yeah, me neither. They're not really geo-involved in any loud way. Their contribution to GIS has consisted of a few excel plugins and a dedicated deference to the ArcGIS platform. They made a halfhearted attempt to enter the web mapping space with Bing maps, but it's safe to say the developer community hasn't expressed much interest:


But there are some geo-progressive (How many times am I allowed to use it as a prefix?) corners of the Windows empire, notable in the hiring of OpenStreetmap founder Steve Coast, and in a generous licensing of high-resolution imagery for OSM mappers. Add to that a new tool for Excel called Geoflow, which launched to some fanfare Last week as we all looked up from our maps with a collective "Microsoft? Really?"

I've been fiddling with it a bit since and I'm glad to have given it a try. It can probably be best described as the CartoDB/Fusion Tables of the desktop, and while that may sound like a backhanded insult, I think it will be extremely useful to the large and dedicated population of Excel users. This is not GIS. This is geovisualization, and it's not half bad.



The premise is simple - put your spreadsheet records on a map - and it executes with little difficulty. The user just needs a column with placenames (or coordinates), and the rest of the columns become thematic options in a 3D map without having to leave excel. The defaults are novel and in some cases visually appealing - which is fortunate because there's not a lot of tweaking you can do (Until Microsoft exposes a Visual Basic UI. Heaven help us.). The orthographic perspective is nice and the application did a better job of figuring out my data structure than excel usually does.

A quick roundup:

  • Only available on Office 2013 for now.
  • It's the antithesis of open-source but they get a temporary pass because excel is ubiquitous. I look forward to the LibreOffice Calc plugin.
  • The Nokia-based geocoding is solid, with only one misfire to null island.
  • The map UI isn't intuitive, but I'm not exactly sure if there is such a thing as intuitive 3D navigation. Maybe this is one for the Leap Motion developers. 
  • Not portable. Unless there's some feature in the Office365 version that I'm missing, this is Geoflow's biggest shortcoming. The one nod to the desire to show your map somewhere other than in your cubicle is the "Copy Screen" button. Which I already had on my keyboard.
  • The tour function is clunky, roped to a jerky graphics engine that tries to load every zoom level of tiles even after you've arrived at your stop. This happens every time, suggesting there's no tile caching at all.
  • There's a nice selection of default map themes from Bing Maps, but they have trouble in the orthographic environment. Labels and imagery are slow to load, and text keeps its north orientation even if you want the Hobo-Dyer projection.

Overall: not bad, Microsoft. I like novelty, and they've somehow managed to combine some of the better elements of Google Earth with a spreadsheet and make it look compelling. I wouldn't say it's buggy, just in need of some UX rethinking. Ultimately it's good for all of us when a new map technology emerges from unexpected quarters.


Read more ...

Navigation on Planks

Wednesday, February 6, 2013
We're not there yet . . . (Skier dude by Saman Bemel Benrud)
The news yesterday was that Google has added the trails of 38 ski resorts in the US and Canada to its Maps database. Here in Vermont, my stashes are safe from further encroachment, since the only two Green Mountain resorts on the list are perennial tourist-bait favorites Okemo and Stowe. That said, this is a cool move by Google in its desire to permeate our mobile world. I'm used to leaving my phone behind while I ski, but this sort of information would be useful to have at my fingertips if I were in unfamiliar terrain at a new resort. It could also be useful in tandem with the ski-run-tracking apps that are proliferating these days.

However, one item is missing from the new functionality: ski navigation. You can't request the shortest (or gnarliest) route from the top of the Sensation Quad to the bottom of the Alpine Double. These new lines - while color-coded according to difficulty - are just background images in Google's otherwise-rich network of information. The navigation engine suggests that the shortest route down the Liftline at Stowe is to pop off my cables and trudge down the Toll Road or the Long Trail.

The longest way around
I can see why Google would be nervous about offering directions in this context - There are two problems that are unique to alpine skiing as a navigation paradigm:

Obey the Rope.

Trail conditions (especially in the East) change more rapidly than road conditions. However, they don't change more rapidly than traffic conditions, and Google's got that covered in near-realtime in major metro areas. 
Solution: GTFS for ski resorts - While it's not immediately realistic to ask every resort to come up with a trails API, it's possible to work out a common format for trail reports (Y'know, those things you look at in the morning to see where it's not bulletproof) much like Google does for public transit. It would be an afternoon task for a data ninja at Mountain View to figure out a way of scraping and parsing that information on a daily basis.

Gravity's a . . . Precondition.


Leaving nordic and backcountry skiing aside, ski navigation would require consideration of elevation change. No uphill turns allowed. 
Solution: Spatial Analysis -  This is one of those bread-and-butter geoprocessing problems that GIS undergrads are given: calculate a flow accumulation surface, add turn attributes to the trail data accordingly. This is actually kind of a fun one.

Lots of folks are never going to look at a mobile device while skiing, maybe myself included. But for the ones who do, it'd be pretty danged cool to have navigation assistance built into the vertical territory that Google is now adding to its already-thorough map system. Just a few hitches to overcome, and I think they're up to it :)

"If only there were some way I could technify this experience . . ."

Read more ...

Context For Cheap - The Map Reference Overlay

Wednesday, January 23, 2013
I used to hate building reference layers in my maps. Labeling placenames was painful, but nothing compared to the chest-hair-waxing misery of scaling transportation symbology by road class. No out-of-the-box defaults were ever cartographically pleasing, and hours were incinerated in the fires of annotation placement. [Sorry, QGIS and Illustrator - you're just as much to blame here as the Redlands upstart.]

But that's basically all in the past. Not long after I slogged my way into web interactive maps, the crew at Mapbox released the finest preconfigured basemap I've ever seen, "Mapbox Streets". They quickly followed with a dozen attractive starter styles and an impressive customization pallette.

Mapbox Streets

The idea of the pre-baked map was nothing new - Google, ESRI, GeoIQ and others had entered the web map age with variations on the Reference-Canvas idea: "You provide the overlay data and the story to tell, we'll provide the geographic context." And this works really well for monodimensional Point of Interest (POI) data, as demonstrated by the Google pushpins that these days rain from the sky to skewer every interesting set of coordinates on the planet. Pins and icons don't get in the way of labels and reference features.

The siren call of the martini glass . . .

But what about polygons and the world of the choropleth? Not that it stopped anyone from trying, but polygons on top of a reference canvas either obscure the features beneath or require too much transparency to make a thematic point. I bombed out on early attempts to work with this essential truth:

NYC Metro, covered in bubble bath

The solution to this problem is under our noses. I first noticed this technique when John Keefe and Steven Melendez at the WNYC Data Desk posted a Mapbox-based interactive looking at NYC's proposed wards; the streets were curiously visible above the color-coded ward polygons.

They had introduced me to the reference overlay.

Leveraging the customization options of Mapbox streets, they had
  • Winnowed out the layers representing land and water from their basemap, leaving just roads, land use and text,
  • Set these to a modestly transparent level (maybe 30-40%),
  • Using the compositing of the Mapbox API, laid this semi-transparent layer on top of the thematic polygon layer, inverting the standard reference canvas model
After I recovered the pieces of my brain that had exploded out my ears (maybe I'm easily impressed), I set to applying this tactic to my own maps. I also realized that this could be expanded to allow for a sort of map sandwich, with land and water below, thematic data next and reference data on top:


And it's not just a Mapbox thing:
Hell, if you can do better than AJ Ashton and company, build your own mostly-transparent reference overlay and cache it in a tile server for future projects.

Subtle Context on the Census Dotmap

While I realize this is all old hat in the GIS world (yes, of course you place your labels above your polygons dude), it's usefulness in web mapping can't be overstated. The reference overlay saves us serious time, solves the "mashup" problem, and lets us focus on our data and what it has to say.

Update, Wednesday Night:
After conversations with some of the Stamen and former-GeoIQ folks, I figured it'd be worth comparing what happens when three different teams build reference overlays from the same (OpenStreetmap) data. Check it out here.







Read more ...

HTML5 and the Limits of Imagination

Tuesday, December 11, 2012
I just (days later than everyone else) came across this awesome Google Maps application built by Darren Wiens using the buckled-on graphics capabilities of paper.js. It strikes me as a perfect example of good imagination at work on complex geographic problems. Specifically, this is a mixture of pure javascript, HTML5 browser-boosting and advanced trip routing, all to make a cool visualization of where traffic bottlenecks will arise.

The application calls on paper.js to generate an endless series of random point-to-point trips within the map view, then passes them to Google's routing API and renders the result visually as a trip on the local road network. Over a short time, the traces of the trips build up into a density map of highest traffic:


So as is my wont, I grabbed the code (Thanks again to Mr. Wiens) and adapted it to my home territory of Burlington, VT, with a bit of map styling for contrast (modern browser required):



I intend to spend a lot more time figuring out the ways that new graphics modules can interact with the maps I build, but as always I'm limited by my own imagination. It seems like the technology is increasingly unlimited in capability.

Read more ...