I'll be wrapping up a series of existing projects, but then Geosprocket will be on hold. Thanks to all of you who have been such great clients, collaborators, mentors and friends. I look forward to keeping the conversation going!
Moving to a New Gig
Monday, February 10, 2014
I'll be wrapping up a series of existing projects, but then Geosprocket will be on hold. Thanks to all of you who have been such great clients, collaborators, mentors and friends. I look forward to keeping the conversation going!
Redistricting: A Question and a Query
Wednesday, January 15, 2014
Alert: Both policy AND technology will be discussed below. Bring both your brains. You've been warned.
I was unrealistic early on about how much of a role technology could play in the process. A survey of neighborhood geographic identity and a do-it-yourself redistricting app were helpful but not game-changing. After years of wrangling, the parties mostly just hashed it out in committee (though the final plan is derived from one made with our DistrictBuilder app), and now in about 6 weeks the voters will approve or reject it.
Which brings me to the question: What would the proposed plan change things for the average BTV citizen? It's a pretty easy one to answer in the aggregate - all the information is out there: new boundaries, old boundaries, population distribution, polling locations. But these bits are spread out across various sites and articles. What if there was a way to just ask the question: How will this change things for me?
Under the umbrella of CodeForBTV - the local CodeforAmerica brigade - I put together the basic answering tool. Built for phones (via Bootstrap), the app lets the user plug in their address and get back a list of redistricting plan implications:
It's a simple app, relying on some robust APIs. I hope it's useful in getting my neighbors oriented to the landscape of the proposed redistricting. At the very least it'll save people some time peering at a large paper map in city hall, and at best it'll head off 40,000 individual "how will this affect me" emails directed at the city councilors and GIS manager.
I also hope it'll be of use to anyone else who wants to present geographic change over time at a user-selected location. Get the code on github.
 |
| Gotta love the classics |
The Question
My city is on the tail end of the country's redistricting cycle. Somehow while Texas was marginalizing Latinos and Dennis Kucinich was getting scribbled out of his constituency, we here in Burlington were waffling over how to deal with triparty politics and the implications of our changing neighborhoods.I was unrealistic early on about how much of a role technology could play in the process. A survey of neighborhood geographic identity and a do-it-yourself redistricting app were helpful but not game-changing. After years of wrangling, the parties mostly just hashed it out in committee (though the final plan is derived from one made with our DistrictBuilder app), and now in about 6 weeks the voters will approve or reject it.
Which brings me to the question: What would the proposed plan change things for the average BTV citizen? It's a pretty easy one to answer in the aggregate - all the information is out there: new boundaries, old boundaries, population distribution, polling locations. But these bits are spread out across various sites and articles. What if there was a way to just ask the question: How will this change things for me?
Under the umbrella of CodeForBTV - the local CodeforAmerica brigade - I put together the basic answering tool. Built for phones (via Bootstrap), the app lets the user plug in their address and get back a list of redistricting plan implications:
The Query
The question of individual impact can be answered with a simple spatial query: return some pieces of information based on the polygons that intersect the user's location. This pointed to some basic building blocks:- The city redistricting plan - both the old wards and the new ones lumped into one geojson file
- A robust search engine - I went with the Google Maps API because of the amazing viewport-biased geocoding and the built-in typeahead, but given more time I would use Leaflet and a state-hosted geocoder plus typeahead.js
- Spatial query capability - fortunately CartoDB offers all the magic of PostGIS at a URL endpoint, with great styling available as well.
It's a simple app, relying on some robust APIs. I hope it's useful in getting my neighbors oriented to the landscape of the proposed redistricting. At the very least it'll save people some time peering at a large paper map in city hall, and at best it'll head off 40,000 individual "how will this affect me" emails directed at the city councilors and GIS manager.
I also hope it'll be of use to anyone else who wants to present geographic change over time at a user-selected location. Get the code on github.
On Reverse-Engineering a Map Stack
Saturday, January 11, 2014
I'm going to come right out and say this:
You should probably just use Mapbox.
How I came to that conclusion is a bit of a longer story.
The Scene
As a cartographer, I am an unabashed fan of Mapbox. I've been using Tilemill for years, and I love the fully-realized design of the Mapbox Streets basemaps. Even before Google Maps brought the paywall hammer down I was already migrating my clients' projects to the open-source ecosystem based out of a D.C. garage. Controlling so much of the stack in an open-source environment meant less risk to my clients, and in most cases it came out a lot cheaper. The choice was pretty easy, actually.
Looking around this section of the market I don't believe that Mapbox and Google Maps have anyone to compete with but each other (mobile is another story). They are the only two companies currently offering a rigorous javascript API tightly integrated with attractive map services built for the web. (Debate this in the comments, ESRI, Nokia, and Bing users) There are plenty of a la carte options out there - Stamen's wonderful map tiles, the sheer power of the OpenLayers API - but as a lazy developer I've come to really like working with an integrated, open stack. Mapbox is my current choice, though many use cases all but require Google's services.
The Challenge
An open-source stack doesn't mean a free stack, and Mapbox's map tile charges can rack up quickly if you're not paying attention. Exhibit A: I offered Brandon Martin-Anderson one of my Mapbox tilesets to use as a reference for his Census Dotmap. Several viral weeks later I was looking at overages the size of my annual budget.
An open-source stack may not always be free, but in theory it can be copied and hosted by others. I've done this quite a bit on the javascript API side, mixing and matching tile providers with various client libraries. However, the costs that had begun to concern me were on the basemap end. So I set out to do what Mapbox flat-out encourages you to do with its wide-open codebase:
I would make and serve my own damn basemap. How hard could it be?
An open-source stack may not always be free, but in theory it can be copied and hosted by others. I've done this quite a bit on the javascript API side, mixing and matching tile providers with various client libraries. However, the costs that had begun to concern me were on the basemap end. So I set out to do what Mapbox flat-out encourages you to do with its wide-open codebase:
I would make and serve my own damn basemap. How hard could it be?
The Tools
- Openstreetmap Data
- QGIS
- osm2pgsql
- PostGIS
- Tilemill
- OSM-Bright Template
- mbutil
- s3cmd
- A big ol' S3 bucket on Amazon Web Services
I started with the open-source Tilemill template OSM-Bright, noting that there are some good examples of it in use out there. I grabbed the current OSM data, piped it into my local PostGIS database (note the hazards of adding many extracts) and spent some time turning it into "Geosprocket-Bright" in Tilemill:
Let's keep track of the time investment, shall we?
- 4 hours to export everything from Tilemill
- 6 hours to chop the .mbtiles into 1.6 million PNGs
- 3 hours to consolidate them all into a single directory structure (because I was too dumb to do that in the last step)
- 56 hours to push the tiles to an S3 bucket on a 20mbps connection
Obviously these are CPU hours, not billable hours - but it was still more than two days between when my map style was ready and when I could actually hook the map up to a browser.
The Verdict
This is a narrow case where I needed to roll my own tiles and serve them. My map included texture and custom fonts, which is beyond the reach of Mapbox Streets. The total filesize of my exports - even as .mbtiles - was 15GB; that translates to the Premium Mapbox hosting plan, and a whopping $6,000 flat fee per year. That's for five cities - the tiniest fraction of a world of tiles. It's not quite Google Maps Enterprise money, but damn. By contrast, it cost me nine bucks to get all of my map tiles into an S3 bucket, where I'll get billed something like half a cent per 1,000 map views.
But this process will be moot pretty soon anyway; the promise of Tilemill 2 is that you only need to bring your cartography to the table - Mapbox will do the rest of the work to get your style to the browser with vector tiles. They've only done this for a few testers while they're hashing out the details, but odds are there will be a public version in the first half of 2014.
In keeping with other examples of open-source underpinning software-as-a-service, Mapbox has a sound business model. Despite the huge amount of intellectual capital they've open-sourced, it is still easier and - when time spent is considered - VASTLY cheaper to just use their hosted map services. I suspect they've been totally aware of this even as they release service-liberating tools like Tilemill, OSM-Bright and mbutil. The scale and efficiency of Mapbox make their hosted maps too good to avoid.
I hope my experience here has been instructive to others.
The Product
I did get a map out of my experiment, and it's free to use. If you happen to be mapping in Warsaw, Sochi, Santo Domingo, LA or the Bay Area, I hope it proves useful. Just use this XYZ tile scheme in your client implementation:
http://s3.amazonaws.com/geosprocket/tiles/{z}/{x}/{y}.png
As per the usual OSM license, be sure to include "© OpenStreetMap contributors". Happy Mapping!
The District Plows Tonight
Thursday, January 2, 2014
It will surprise no one in Vermont if I say it's cold tonight.
On the fun side, during the last storm in D.C. Jeremy Bowers of NPR wrote a handy script ("Mister Plow") to scrape location data for the city's plow fleet and offer it up as a live feed. Since this sort of screamed for a map, I built one, and now it can serve for this storm:
It's lacking a good temporal symbology, but it'll do to see if the plow has been by your street recently. The mechanism is Leaflet, Mapbox Satellite, and a dose of PHP to parse the feed. Grab it here if you dare.
I FOUND THEIR NEST! DEAR GOD, THE PLOWS ARE SWARMING!
We've got Barrow, Alaska beat at 10pm. It's been that way most of the day and should continue tomorrow. #btv pic.twitter.com/EiVDjEbGwt
— Channel 3 Weather (@wcaxweather) January 3, 2014
So I'm indoors, and bemused to note that it's also snowing down in the nation's capital, where such things tend to bring out the crazy in the more troglodyte-like elected representatives.On the fun side, during the last storm in D.C. Jeremy Bowers of NPR wrote a handy script ("Mister Plow") to scrape location data for the city's plow fleet and offer it up as a live feed. Since this sort of screamed for a map, I built one, and now it can serve for this storm:
It's lacking a good temporal symbology, but it'll do to see if the plow has been by your street recently. The mechanism is Leaflet, Mapbox Satellite, and a dose of PHP to parse the feed. Grab it here if you dare.
I FOUND THEIR NEST! DEAR GOD, THE PLOWS ARE SWARMING!
What a difference a decade makes in satellite imaging
Tuesday, November 26, 2013
It seems like there's an armada of new earth-imaging satellites headed for orbit every day. As we consider the changes afoot in an industry where entry costs have traditionally been measured as a percent of GDP, it's worth a look at the tools of today's remote sensors.
Something has sure as hell been disrupted here. We're about to find out what.
This is a commercial, high-resolution earth imaging satellite before launch in 2001:
 |
| QuickBird 1 being prepped in 2001. Image courtesy of DigitalGlobe |
And this is a roomful of them in 2013:
 |
| The PlanetLabs crew and their fleet of Doves, 2013. |
Something has sure as hell been disrupted here. We're about to find out what.
How to build a Mobile-First Map Form
Friday, November 8, 2013
 |
| Find me, small screen. |
I've spent a lot of time thinking about the incredibly rich experiences we're getting these days from a digital map paradigm basically invented by Google in 2005.
A rich experience everywhere but on mobile devices, that is.
The dichotomy is pretty intense - on our larger screens, we have data visualizations of artful grandeur. We have amazing cartography in tiles and vectors, and we have smooth analytical tools. We're even approaching "GIS on the Web" that doesn't suck.
But then you turn on your phone and the best applications use oblique-angle maps the same way they did in Car SatNavs a decade ago.
I don't have an answer to this problem, I just mean to say I've been thinking about it. And it turns out that a bunch of the technologies I use regularly can be pretty easily combined to offer some possible fixes for how a user experiences maps on mobile. I'll walk you through them while trying to tackle a basic application: a location-based survey.
1.) Bootstrap
I get it. I overuse Bootstrap for the architecture of my sites. But in the latest release (3.0) they went "Mobile First", so it makes many of my other tasks easy by providing the responsive classes out of the box:2.) Google Forms
I still think of Google Forms as an underutilized gem for easy data collection: pipe data from a form to a spreadsheet with an API; one barely needs to try anymore :) However, tying a custom Bootstrap page to a specific google form requires a few things available in the source code of the input form's page:- Form identifiers:
- Field-specific IDs:
- A Redirection iframe:
3.) Mapbox.js
Most of the necessary functions here are actually native to Leaflet, but I like the look of the Mapbox implementation, and they provide "Retina Tiles" that scale better for mobile. The basic tasks for the map in this application are:- Figure out user location without asking
- Let the user fix the inevitably-incorrect positioning
This works with a quick call to a free IP registry, centering the map on the result:
Then the user can drag the map around under the crosshair to where they actually are - this is a tactic I borrowed from OpenPlans after I got a reminder:
@mourner @vtcraghead an alternative would be to just let people drag the map and have the center be the target
— tschaub (@tschaub) November 7, 2013
4.) CartoDB
Using a new tool they've made available to sync tables, there's almost no code involved in this step. All I needed to do was go to "File-->Publish" in the form target spreadsheet, then copy the .csv URL into the "New Table" creation dialog in CartoDB. Then a bit of easy styling created a custom-mapped results page:
The page itself is pretty simple, and you can grab it as a gist here. It'd be pretty easy to fork it and replace the form parameters with one of your own. As always, please share if you use it; I'd love to see where others take this.
The Caveats and Alternatives:
- Geolocation by IP is pretty awful, unless you happen to be Google (but in that case you're currently being sued for the method by which you collected your registry). The service I used put me in the correct city on my laptop, the correct state on my tablet, but according to my phone I'm in Buffalo. There are better ways to get location, but they require user permission, which is a UX hurdle I didn't want to jump for this.
- I subscribe to the gospel of responsive design because I'm lazy. It would probably be smoother to build this as a native Android/iOS app, but no way would it be this easy.
- Since location is being passed to the table from the initial map, there's no geocoding and you could theoretically just tap the Google Drive API to put points on a results map. But again: EASY. CartoDB is that :)
#Facepalm
In the first run I forgot the mobile-specific meta tag. I was reminded :) Thanks, Vladimir!
@vtcraghead yeah, it's not marker's fault, fix the page scale with a correct meta tag :)
— Vladimir Agafonkin (@mourner) November 7, 2013
Seamless Textures for Tilemill
Thursday, October 24, 2013
As evidenced by AJ Ashton's announcement this week, Tilemill keeps getting cooler. With vector tiles and Tilemill 2 on the horizon it seems like anything will be cartographically possible, but I've been having fun while we wait.
Tilemill allows you to fill a polygon or feature with an image instead of a plain color, which makes for some cool pattern and feel options:
However, you need to be careful not to run afoul of the tiling system that repeats any such image in gridded 512px intervals. You can get some funky edges if you're not paying attention. The solution to this is to use a seamless image - i.e. an image where every edge matches up with its opposite. There are quite a few options out there for seamless backgrounds, but not very many that are both square and 512px on a side. So I followed the lead of the talented artists at Stamen and reluctantly opened photoshop to make some seamless texture images of my own for Tilemill.
Tilemill allows you to fill a polygon or feature with an image instead of a plain color, which makes for some cool pattern and feel options:
However, you need to be careful not to run afoul of the tiling system that repeats any such image in gridded 512px intervals. You can get some funky edges if you're not paying attention. The solution to this is to use a seamless image - i.e. an image where every edge matches up with its opposite. There are quite a few options out there for seamless backgrounds, but not very many that are both square and 512px on a side. So I followed the lead of the talented artists at Stamen and reluctantly opened photoshop to make some seamless texture images of my own for Tilemill.
There are 12 of them, and you can grab them here, licensed CC-by-SA.
I'm personally inclined to overuse things like this once I've made them available (I used them last weekend at StoryHack, actually), so it's going to take some time before I find the right subtle level. I'll let you know when I get there.
Have fun! And I'd love to see any examples of these in the wild once you put them to use . . .
5 Reference Recipes
Thursday, October 17, 2013
At the risk of beating a dead horse, I'm coming back to reference overlays. I've noticed that there a still a fair number of interactive maps online that permit a bullying thematic layer to stomp all over the poor, defenseless map labels and context information. This is not cool:
So I've hashed out the code for getting past this using five common basemap services:
Feel free to reuse them as you see fit. All but the Google example use Leaflet, but the z-index and canvas manipulation tactics are applicable to other APIs as well. If anyone wants to add adaptations for OpenLayers or whatnot I welcome pull requests. Also worth a note is that - in the leaflet examples - the thematic layer is rendered as vector from topojson, which is arguably the fastest vector format available to us today. Give it a shot as a thematic mapping tool.
Go forth and bring context back to your interactive maps!
Note: Thanks to Bobby Sudekum, Josh Livni and Jason Sanford for the starting blocks.
Subscribe to:
Posts (Atom)