GeoSprocket 2011 in Review

It's been a good year. Lots of new open-source tools for geospatial applications have come on-line; large organizations have been opening up their data stores and letting us run wild with the possibilities. It's hard to keep up with everything - to leverage every new technology - but here are a few projects we've been occupied with this year:

Read more ...

Place and Sentiment From Many Voices

Place is powerful in a crisis.

Place brings the responder to the emergency. Place measures risk and damage. Place is often the most important piece of situational context, particularly when a hurricane takes a wrong turn on the New Jersey Turnpike at the end of an otherwise-unremarkable August.

Crisis maps were invaluable in the reaction and recovery surrounding the drunken lope of Hurricane Irene up the eastern seaboard a few months ago. The rapid deployment of tools like Ushahidi, Tweak the Tweet and #VTResponse created a centralized, crowdsourced voice for the crisis. Google's Crisis Landing was used to augment the capabilities of local, state and federal agencies as they organized reports of damage and assessed priorities, disseminating the information directly to the public. [Patrice Cloutier has exhaustively documented the crowdsourced Irene response in this report] All of these tools involved a place component -  "Where is this happening?", "Where is this needed?" - and could draw on semi-precise geolocation assets like online address searches and smartphone GPS sensors.

However, there was also a huge dialog occurring without real geolocation, on Twitter, Facebook and any number of smaller social media outlets. These messages numbered in the hundreds of thousands, but on their own were lacking the context of place.

This is where new developments in "Big Data" analytics come in handy. With some computational heavy lifting from Kate Starbird at the University of Colorado and Chris Danforth at the University of Vermont, Geosprocket was able to bank millions of Twitter posts from the days surrounding the storm. Then the assistance of metaLayer Inc. was instrumental in putting Irene-related tweets on the map. Using a series of digital sifting processes, they were able to mine the archived Twitter data for placenames and keywords. Where a town or street name was included in a post, that post could be placed at a set of geographic coordinates. Were words like "washout" and "devastated" were used, fine-tuned algorithms could assign a scaled value for the sentiment of the post.

Add a bit of cartographic styling and serving with the open-source MapBox toolkit, and we've got an interactive mapped timeline of Hurricane Irene, as told by Twitter:

No analysis of this type is going to be perfect - the data is inherently messy, noisy, inconsistent - but now that the storm is long past this kind of view can tell us a lot about how we responded to the crisis. It can give us ideas for preparedness. And as platforms like metaLayer and Swiftriver become more robust, we can apply the power of place and more in real-time when the next crisis hits.

Read more ...

Creating Geographic Variation Where it Doesn't Exist . . .

Info-Vortex Over Burlington

Cartography has a well-earned but still-largely unrecognized reputation for messing with human perceptions of real phenomena. Got heathens on the other side of your national border? Just shade them out and pretend they don't exist. Want to make it seem like you're a global underdog? Show yourself surrounded and menaced by powerful foes. This sort of crap makes cartography - and geography more broadly - look like something practiced by schemers in tophats and handlebar mustaches.

But manipulation of space is the bread and butter of cartography, stemming from some central questions: How do you make round things flat? How do you make the impossibly-large small enough to see on a canvas or on a glowing smartphone screen? Mapping inherently involves reduction and obfuscation, so the problem I recently came across is a novel one:

How do you expand an effect on a map?

When Hurricane Irene hit the U.S. Northeast, hundreds of thousands of people took to the Twitterverse (and to Facebook and traditional civil response agencies, not to discount them) to report damage and request help. A few thousand folks mostly in the NYC area sent their tweets with geolocation included. Good on them, but they represented somewhere in the neighborhood of 1% of the hurricane-related traffic. Tens of thousands more sent messages with placenames in the text, some more specific than others.

Modern geocoding services like Google, Yahoo and Nominatim provide great geolocation based on addresses and fragments, but what happens when you've got a single placename representing a general area? What if it recurs thousands of times? Do you map everything at a single location, the centroid of a state boundary?

The word "Vermont" appeared in over 6,000 Twitter posts in the four days immediately after Irene washed through the state. With nothing else to add specificity, this means that over 6,000 events occurred at 43.8717 N, 72.4517 W: the geographic center of the state. While this is a bit whimsical, it still presents the problem of showing a huge number of unique messages at a single place.



A spiral seems to fit the bill; the points spill outward and occupy their own space, but they clearly originate from the central coordinates they share. It's not spatially exact, but it shows both association and information volume. Smaller spirals begin everywhere multiple posts share the same placename, for example in Rochester and South Burlington.

Using LibreOffice Calc (or Excel if you must) and CartoDB, this visualization is actually a snap. It begins with a set of Twitter posts and Lat (Y1) & Long (X1) coordinates taken from placenames in the text (a service provided by metaLayer - more on that in the next posting).

1. Sort the table by unique location
2. Add fields representing:
1. Location-magnitude (i)
2. Bearing of spiral point (ang)
3. New Lat (Y2)
4. New Long (X2)
3. Calculate as follows:

a = 0.005

b = 0.009

i = if X1 is not unique

  i + 1

 else 0

ang = (Pi/45)*i

Y2 = Y1 - (a * SIN(ang)) * (2.718281828 ^ (b * ang))

X2 = X1 + (a * COS(ang)) * (2.718281828 ^ (b * ang)) 

This may seem a bit obtuse, but it can be adapted to VBA, Python, JavaScript or whatever suits your purpose. And the 'a' and 'b' variables can be tweaked to make a spiral to your own specifications; 'a' controls the distance offset from the central point, and 'b' controls the rate at which the spiral expands. It works best if both are kept between 0.001 and 0.1. In any case, once the table has the new coordinates, upload it to your CartoDB account and render some tiles! 

There are certainly a host of other ways to attack this problem, but a spiral is a basic solution in a world where crowdsourced data increasingly muscle into the geographic sphere, begging to be placed on a map.

Read more ...

CartoDB and the Manifold Uses of Tiled Maps

Javier de la Torre was kind enough to give me a demo account for the new CartoDB project from Vizzuality. Described by some as the "Open-source fusion tables", CartoDB is interesting to me for its novel tiling functionality. Usually, tiled maps are lightning-fast on the web, but they're set in stone and take forever to render on the server side. CartoDB uses some cool tweaks to serve map tiles fast as usual, but with added feature editing capability. When a feature is selected from the tiles, CartoDB instantaneously serves up a vector version for editing.

While this can lead to some interesting hiccups - multi-feature editing, for example - this is still a huge step forward for web cartography and GIS. It's got the speed of tiles and the interation of desktop vector. The future looks bright . . .

Here's an example from CartoDB, serving up a Geosprocket database of ecoregions clustered into Biodiversity Driver Zones (BDZs):

Read more ...

Renewable Energy Potential Mapped

I've been fiddling with some data from the National Renewable Energy Laboratory (NREL), and the solar potential in particular has a great spatial angle to it. Using QGIS for analysis, I fused NREL's Direct Normal Irradiance (DNI) database with U.S. Zipcodes. Then I used Developmentseed's Tilemill engine to give a picture of how insolation changes over the course of a normal year. If you click on any of the zipcode polygons, you can tease out the regional variations in specific months, and maybe even make a longshot prediction about how viable it might be to install some rooftop photovoltaic panels.

Sorry, Pittsburgh. It doesn't look good for you or Seattle, honestly. This may not be news to you.

Read more ...

Crowdsourcing the Guatemalan Election Cycle

Photo: Associated Press

Ushahidi has just been deployed in Guatemala for monitoring of the election cycle there, with a really slick interface. It looks like they have a pretty robust participation and reporting as the national race goes to a runoff between two right-wing, "ley y orden" candidates. Fun tidbits include the Partido Patriota buying votes for about $38 a head in Xela and election workers being assaulted in the capital:



Guatemala is a mess, but they don't need some hack gringo from the subarctic to say so. What they need is a "Functioning-Civil-Society-in-a-Box" and the instantaneous establishment of rule of law. They were stripped of these things over years of civil war that we in the U.S. abetted, but I have hope of their return when I see a crowdsourcing application like this bring a little bit of transparency to an otherwise-depressing democratic exercise.

Read more ...

Crowdsourcing the High Water

Now that the floodwaters have receded, it's really important to document the extent of the water before the traces change. Emergency management agencies still have their hands full, but they've recommended we collect geolocated photos of high water marks around the state - these help with mapping the effects and preparing for the future. If you're in a position to do so, we'd be psyched if you'd help out by:

1.) Taking photos of any high water positions around you (sides of buildings, washouts, etc.), then entering them with location info in the map below or here: Submit a Report to the Crowdmap

2.) Downloading the free Ushahidi App on your Android or iPhone and recording high water marks live from the field. Once it's installed,

  - Add your info,
  - Click on "Add Deployment",
  - Then enter this crowdmap address: "vtirene.crowdmap.com"

  and you're set to enter info from the field using your smartphone's GPS and camera. Reports will go directly     to the crowdmap.

3.) If you've got a high-resolution GPS unit like a Trimble or Garmin, collect position info on any photos you take and add them to the map using either of the above methods (They both accept latitude/longitude coordinates). Here's a guide to getting the greatest accuracy while mapping high water marks this way

With any of the above methods, be sure to check the "High Water Mark" box to clarify what type of info you're entering.

Be sure to defer to first responders and residents at all times. Some towns have restricted access, and it's important to respect that as they dig out. Also, keep an eye on the weather at all times and stay out of danger if more rain is threatening. But overall, thank you all for the work you've been doing; we're lucky to have such a great community in Vermont.

Read more ...

In Which I Miss the Point . . .

This storm appears headed straight for Iceland! Egads! High winds and rain in the North Atlantic! Unprecedented! I wonder if Icelandic TV has an endless feed of a meteorologist standing on the Reykavik waterfront with excessive rain gear on.

Read more ...

Republican Fundraising 2012 - the Early Days

They're coming! Holy crap, they're coming!

It's always instructive to see the spatial dimensions of politics. Development Seed in D.C. just set up this snappy way of viewing individual donations to candidates so far this election cycle [it's already a cycle? Jeez, slow down, Hoss.]:



The regional clustering of some candidates - Romney owns the coasts, the midwest is split - is a fairly good representation of how our society fractures and gathers, even on the conservative side of things alone.

Mostly surprising is the number of times Ron Paul's name shows up. Dude just won't quit. Or make a coherent argument for a tight monetary policy.

Read more ...

Oh for the love of . . .

Finally took a look at ArcGIS Online, released in May this year. I now feel a wee bit like Mike Doonesbury circa 2000:

So I've been working for a few months on an open-source workflow that runs between a web map and an editable database with embedding capability, actually pretty psyched that ESRI wasn't in that space yet. I wish I'd tried this out months ago:



View Larger Map

It looks like you still need some ArcSDE and ArcGIS for server chops to pull off a completely-editable interface - to actually practice vector editing in a webpage window - but this was frightening. I threw the above map together in 30 minutes using ArcGIS Online, while coding something similar from scratch took closer to 40 hours (though I'm no programmer).

ESRI strikes again.

Read more ...

Shapefiles to Fusion Table Maps

So you need a web map on the fly?

Google's fusion tables toolkit is something of a killer app for cartographers. As many have already discovered, it's sort of a miniature, compacted version of the process by which a relational database gets viewed as a web map. By tossing it all into the same browser-based, code-free blender, Google has removed a whole slew of otherwise-tedious steps from the process. Fusion tables can do the following:

• Clean data
• Serve data
• Centralize authentication and access
• Merge/Relate Datasets
• Map and style geographic data
• Serve maps on the web.

That's anywhere from three to six different types of software using traditional methods. Hot damn.

The one downside to fusion tables has been the use of KML as the standard accepted geometry type for complex features like lines and polygons. For those of us who work with the shapefile format, that's an extra step. Fortunately, Josh Livni at Google hacked together a way to send shapefiles directly to fusion tables with geometry intact for mapping:

http://www.shpescape.com/ 

It's a sweet add-on that lets you send - say - a shapefile showing Vermont towns directly to a styled web map, with queryable information via Google charts.

I love how this stuff just keeps getting easier . . .

Read more ...

Cartodynamism

Herodotus laid down his world on parchment with ink. Samuel de Champlain traveled with paper and pencil. They - and every mapmaker in between and since - created map objects that were imbued with authority.

Maps have always been trusted for complex reasons; social, cultural, political. But for millennia map-making and even map-reading were the province of an elite few, mirroring the dynamic of priests who owned the word of god when the world was 5% literate. Only in our great-grandparents' lifetimes have maps been democratized, their use become common for road navigation and capital-memorizing.

Broken Survey - Incomplete topography in Wyoming


The same way it shook everything else in our lives, the internet has brought maps a step further. Common map interfaces are available in every browser and show only the most basic of facades - these maps contain the world within, and the whole of geography in a 4"x3" container on a glowing screen. Cartographers are only just starting to grasp the possibilities here - this technology isn't yet a decade old - but now that maps of everything are available to everyone, the size of the world has changed.

Washington D.C. live - dark dynamic cartography

Read more ...

Multi-Attribute Labels in QGIS

We frequently have to render feature labels on maps that have more than one type of information in them, for instance both parcel ID and parcel area. This quick run-through shows the process for multi-attribute labelling in QGIS, eveyone's favorite open-source alternative to ArcMap.

Read more ...

Agriculture, Food and Poverty

A recent post from the world of crowdsourced mapping has me thinking about food availability and the nature of poverty in the United States.  The Brooklyn Food Coalition has begun a community food survey, collecting location and stocking information for food stores throughout the borough.  The aim is partly to identify zones where - for reasons of distance, transportation or pricing - it is difficult to locate healthy food.

The general term used to describe zones like these throughout the country is "Food Deserts."  This also refers to the economic state of people; a household is said to live in a food desert if it doesn't have access to a car or public transportation, and lives more than a mile from a grocery store.  According to the USDA, there were 2.3 million such households in 2009.  Here's what that looks like spatially, broken down by county:

Percent of households a.) without access to a car, and b.) living more than one mile from a grocery store (Data: USDA)

While there's lots of local variation, the emerging theme is one where relative poverty and food deserts coincide spatially, particularly in the South and in Appalachia.  Huge chunks of rural America lack access to fresh foods, relying instead on smaller stores with a greater abundance of processed foods.  The Brooklyn Food Coalition is also noting this trend on the community scale, and with greater detail about the specific character of food availability.

There is a dichotomy here, and it will require further investigation: How do food deserts line up spatially with agricultural production?  Here is a tiny look in that direction, using a geographically-weighted regression:

Percent of county land area in crop cultivation (Data: NASA)

And what I'll call the "Map of Socio-Agricultural Irony", or where existing food deserts coincide with high levels of total crop cultivation:

Geographically-weighted local R-squared: percent area in cultivation vs. percent of population in a food desert per county

Amber waves of grain everywhere, and not a kernel to gnaw on . . .

Read more ...