Open Cadastral Data: a Github Test Case

A bolt of GIS-Manna from Heaven

A dedicated and talented GIS developer at the VT Agency of Natural Resources just finished compiling the most-comprehensive cadastral dataset to which our small state has ever had access. Coincidentally he sent out the announcement while I was annoying a few githubbers about their new geodata capabilities. What luck: a test case.

The dataset - as a shapefile - clocks in at 250MB, representing 289,000 parcel polygons in 191 towns. [The 46 VT towns not included tend to have populations under 200 and a blithe commitment to dusty paper records.] It turns out that this is over the 100MB limit imposed on individual files, so I pulled the dataset, split it by town and converted the outputs to geojson, then pushed them to a repository I just set up. The largest of these is juuuuust over the maximum file size to be renderable in Github's snazzy new Mapbox-based interactive maps, but most of the town parcel maps pop up just fine.

The Implications

Okay, great - another slippy map somewhere on the web. Who cares? These folks, actually:

• A landowner who wants to look up the parcel numbers of her abutting properties without visiting the steam pipe distribution venue in the basement of her local town hall.
• A town GIS manager who has a hard drive overflowing with parcel shapefiles, converted CAD drawings and map requests from lawyers.
• Probably a lot of lawyers. Don't get me started.    
• The overworked data gurus at the Vermont Center for Geographic Information (VCGI), who - while they're probably the only agency that can handle it - are really reluctant to take on management of a statewide, rolling, versioned database.

What Github provides in this test case, for free:

• Fast hosting of modestly-large geospatial data
• Relatively-simple version control of said data
• Edit access to anyone with a github account and a GIS platform (FOSS or Arc'ed)
• An embeddable client view for the public for files up to 10MB
• A robust API for client views bigger than that (for example)
• A muthaflippin' download button (well, "save page as")

For those of us who have been ranting about geoportals in recent months, this pretty much covers the bases. For free. Github is walking the #opendata walk.

What's Next . . .

There are a whole bunch of caveats here. Biggest is the file size issue (though I've seen Bill Dollins and Sophia Parafina starting to work around that in the past few days), since 10MB is a fine limit for municipalities in the second-smallest state, but Manhattan is a teensy bit bigger. Another hitch is data quality; this dataset is top-of-the-line, but like any other it's missing SPAN numbers, dates and acreages here and there. The vendors will tell you that quality can't be beefed up in a free collaborative environment, without data value-add. I don't know if they're right or wrong.

But I'd love to see a town GIS manager throw a pull request and get this ball rolling. Who's up for it?

The data hub is waiting right here


Update: 7/25/13
I've heard a bunch of awesome questions about the practicalities of using Github this way, so while I am by no means a power user I recorded a quick demo video.

Read more ...

We Are Still Talking About These Things: Dispatches from the Dawn of Crowdsourcing

I was just having a conversation with some colleagues about the nuts and bolts of participatory mapping with some local farmers. We were all:

"Man, it totally depends on if you have internet at the site. If you do, just pull up Google Maps or CartoDB and have them digitize right into the iPad. If not it's a huge hassle; you'd have to annotate a PDF and georeference it later."

"Yeah, but either way the stylus is key. Non-technicians always want to use a pen and paper."

This talk of getting a skeumorphic data-entry device in the hands of the common folk reminded me of something from the past. This, specifically:

Even taking as a fact of life that community GIS will always require some mediation by the more technically skilled, Al-Kodmany's Chicago neighbourhood projects are not a little bit extreme. In fact, by the time he describes the community members being given 'coloring the map' participation exercises as a way to actually participate, there is a rather patronizing air about the project. 'Participants were broken into small groups and were given a map and felt-tipped markers,' he writes. Felt so as not to inadvertently poke out their own eyeballs, no doubt.

This from Christopher Miller, writing in 2006 about how participatory GIS (nee "GIS/2") was being hamstrung by academic condescension, cultural barriers and the failure of imagination that is traditional GIS ("GIS/1"). Man, 2006 sounds like a long time ago from here, but we're still spinning our wheels in some ways on this stuff.

---

Based on Andy Woodruff's Boston-oriented project, last year I built an app to get locals defining the boundaries of neighborhoods in the city of Burlington. When I showed up to the first hearings on city redistricting to present the results, their credibility was questioned by some who hadn't heard of the project, even though it had been announced in every local news outlet as well as digital channels. These were people who showed up to meetings, because that was how you participated. Not learning my lesson, I adapted Azavea's Districtbuilder app for the redistricting process and made it public, hoping "participatory technology" would open a traditionally-closed political world. It turns out that the technology wasn't really the problem.

---

In his snark-laden, brilliant 2006 paper, Miller goes on to describe this neat little app called Scipionus, built in the aftermath of Hurricane Katrina on the back of a newly-opened Google Maps API. Here's the thing: it let anyone with a web connection report their location and status on a Google Map! Fast-forward three years and Ushahidi comes along. Fast-forward another three years and Google Maps gets collaborative, Crowdmap is everywhere and Twitter becomes a news source. Fast-forward to last month and the most reliable information on a civil war is from a participatory map, and Ushahidi can take in reports from every technology short of smoke signals.

We still don't have good retorts to Miller's challenges. We still haven't made GIS accessible to the public, and I'm not really sure that's a valid goal. More importantly, we seem to be driving deeper into the gulch between Internet-Worshippers and Tech-Hostile Curmudgeons that Miller warned us about. Whether we're community-building, solving disputes or reporting the news, we've got to do a better job building bridges between the public and our maps.

In many cases, "Paticipation" is still a felt-tipped marker jabbed into the eyeball.

Read more ...

Geoflow: A Hasty Review

Remember those Microsoft guys?

Yeah, me neither. They're not really geo-involved in any loud way. Their contribution to GIS has consisted of a few excel plugins and a dedicated deference to the ArcGIS platform. They made a halfhearted attempt to enter the web mapping space with Bing maps, but it's safe to say the developer community hasn't expressed much interest:

Despite my last Win8 Metro map project working flawlessly using @leafletjs, I decided to give @bingmaps SDK a try. Don't do that mistake.
— Marcel Šulek (@veproza) April 19, 2013

But there are some geo-progressive (How many times am I allowed to use it as a prefix?) corners of the Windows empire, notable in the hiring of OpenStreetmap founder Steve Coast, and in a generous licensing of high-resolution imagery for OSM mappers. Add to that a new tool for Excel called Geoflow, which launched to some fanfare Last week as we all looked up from our maps with a collective "Microsoft? Really?"

I've been fiddling with it a bit since and I'm glad to have given it a try. It can probably be best described as the CartoDB/Fusion Tables of the desktop, and while that may sound like a backhanded insult, I think it will be extremely useful to the large and dedicated population of Excel users. This is not GIS. This is geovisualization, and it's not half bad.

The premise is simple - put your spreadsheet records on a map - and it executes with little difficulty. The user just needs a column with placenames (or coordinates), and the rest of the columns become thematic options in a 3D map without having to leave excel. The defaults are novel and in some cases visually appealing - which is fortunate because there's not a lot of tweaking you can do (Until Microsoft exposes a Visual Basic UI. Heaven help us.). The orthographic perspective is nice and the application did a better job of figuring out my data structure than excel usually does.

A quick roundup:

• Only available on Office 2013 for now.
• It's the antithesis of open-source but they get a temporary pass because excel is ubiquitous. I look forward to the LibreOffice Calc plugin.
• The Nokia-based geocoding is solid, with only one misfire to null island.
• The map UI isn't intuitive, but I'm not exactly sure if there is such a thing as intuitive 3D navigation. Maybe this is one for the Leap Motion developers. 
• Not portable. Unless there's some feature in the Office365 version that I'm missing, this is Geoflow's biggest shortcoming. The one nod to the desire to show your map somewhere other than in your cubicle is the "Copy Screen" button. Which I already had on my keyboard.
• The tour function is clunky, roped to a jerky graphics engine that tries to load every zoom level of tiles even after you've arrived at your stop. This happens every time, suggesting there's no tile caching at all.
• There's a nice selection of default map themes from Bing Maps, but they have trouble in the orthographic environment. Labels and imagery are slow to load, and text keeps its north orientation even if you want the Hobo-Dyer projection.

Overall: not bad, Microsoft. I like novelty, and they've somehow managed to combine some of the better elements of Google Earth with a spreadsheet and make it look compelling. I wouldn't say it's buggy, just in need of some UX rethinking. Ultimately it's good for all of us when a new map technology emerges from unexpected quarters.

Read more ...

Results of the Geo Toolkit Poll 2012

Many thanks to everyone who participated in this survey of the geospatial industry. I tried to get a platform-agnostic look at the tools that are most-frequently used in our community, and after 250 responses I think we have a useful glimpse of the scene.

My reason for running this survey: I'm tired of speculating. As a multi-platform user I don't know where I stand with my clients and competitors, let alone in this sprawling industry, and I've heard lots of similar curiosity from colleagues. I call this an independent survey in that no vendor funded it (nobody funded it, actually) and it isn't for marketing use. This is a community poll, nothing more. Onward . . .

---

Methods

This is worth a cursory glance, since the results are inevitably colored by the collection routine. I relied almost entirely on social media to get the word out, specifically:

• Numerous twitter solicitations to my ~500 followers, retweeted to a combined audience of over 11,000 utilizing hashtags for both the ESRI User Conference and OSCon - probably annoying the crap out of everyone in the process
• Google+ and Facebook posts
• Listserv posts to ESRI, OSGeo, OSM and Google user groups
• Posts on the three largest geospatial groups on LinkedIn

Given the warren-like distribution network, I do not know how many people saw this poll. Thus the sample size is 250 out of an unknown population, and no big-picture conclusions should be drawn. Also let it be known that the balance - even if measured properly - changes from month-to-month.

---

Results

Question 1: Which of the following geospatial technologies have you used on at least one project in the past year? [Note - I Included pre-purchase GeoCommons on its own out of morbid curiosity; I otherwise would have included it with FOSS4G Web Tools]

Some of the technologies that went into the "Other" column include FME, MicroStation, ENVI/IDL, GIS Cloud, AutoDesk, Maptitude, Idrisi, ERDAS, MapProxy, R-Spatial, Garmin Basecamp, Oracle XE, Ushahidi and Geocortex. Sorry to have ignored those, but it's a big ecosystem out there.

---

Question 2: Which of the above technologies did you use most frequently in the past year?

---

Platform Gregariousness: Do you cross over from your primary platform? e.g. ESRI is your main platform but you've also used Google Maps/Earth at least once in the past year. [A venn diagram would be cooler but the chart API was inscrutable]

---

Use by Business Sector:

---

Use by Country: (Click here for fullscreen glory - we're all cartographers here)

---

There's a lot to see in these distributions - an ESRI lean among U.S. respondents, a FOSS4G lean among Europeans. Also interesting to see how the sectors use these tools. See anything of note? Anything obviously-spurious? Do tell - I think there's a good discussion to be had here.

I'll say it again: this was not a scientific, controlled survey. It's a snapshot or an anectodal collection; take your pick. But it is nonetheless interesting to see what this group of mappers uses to get the job done. Thanks again to you all for pitching in, and maybe we'll try an expanded version next year.

If you're interested in the raw, messy results (stripped of unique identifiers of course), hit this link for an XLS download, and happy parsing!

Read more ...

GeoTools 2012 Poll - Round 2

Expanding the search around the Geo Community:

Read more ...

Results of the Geospatial Technology Users' Poll 2012

Update 7/23/12:  The poll is now reopened and live results are appearing at a new post. The figures and discussion below should be considered preliminary

Thanks to all those who hit yesterday's poll of technologies at work in the geospatial field. I've got some interesting results below.

First a note on experimental design: This crap is not scientific. First I tweeted, facebook-posted and Google+'d, so I got in contact with the core community of geogeeks with whom I regularly interact. Then I sent it out via the Vermont GIS listserv, the ESRI user conference hashtag and the O'Reilly open-source conference hashtag, hoping for balance. There is surely a geographic skew toward the U.S. Northeast, but I'm pleased with the general distribution of respondents. n = 117, which seems pretty good to me. Hit me on Twitter or on the GeoSprocket contact page if you'd like a copy of the raw survey results.

Here's a look at the participants using the generalized locations of reported companies/institutions (lots were left blank, so who knows):


The results of question 1:

Note: Some of the technologies that went into the "Other" column include FME, MicroStation, ENVI/IDL, GIS Cloud, AutoDesk, Maptitude, Idrisi, Mapserver and Geocortex. Sorry to have ignored those, but it's a big ecosystem out there.

And the results of question 2:
Ayup, ESRI Desktop is the big winner in this circle. But a surprising number of Google Maps folks there too. Also intriguing is the even split among the open-source toolset types, contrasting with the topheavy ESRI lean toward desktop.

Here is primary toolset use by overarching category:


Things get interesting when we parse out some conditional results:

• 40% of users whose primary tool is an ESRI product have also used an open-source geo platform in the past year.
• But a whopping 80% of users whose primary tool is open-source (desktop, web or DB) have also used an ESRI product in the past year.
• Same with Google - 80% of respondents who primarily use Google Maps have also used an ESRI product in the past year.
• That favor is largely returned - 75% of primary-ESRI users have used Google Maps.
• OpenStreetmap and GeoCommons had plenty of casual users, but very few used them/built them as their primary tool (1% each).

      There's a venn diagram to be had in there somewhere, but I'm not up to it.

Without leaping to conclusions, I would say that it's still an ESRI world. Even the folks whose day-to-day revolves around open-source or Google tools still fire up an Arc license every now and then. The converse is not equivalent; fewer than half of ArcJockeys use any of the open-source tools, though they are partial to Google Maps.

There are a lot of potential reasons for that, but it seems safe to say that open-source geo is still developers' territory, and Google mapmaking tools are more comfortable ground for ESRI's users. I recall that specific path when I was making my own way from ArcGIS to GDAL and Javascript.

There's a lot to read here; what are your thoughts? Anything surprising?

Read more ...

Browser Cartography: A Manifesto

Hear me out for a minute . . .

"Toner" tiles by Stamen Design

. . . I'd like you to make maps intended for online viewing. 

This is partially a selfish impulse; I'm going to throw a rock through the window if I see another "PDF Download" link masquerading as a web map. I'd also like you to do this to save a few trees from their 24"-by-36"-poster fate, but mostly it's because we're at an inflection point in cartography . . .


View Larger Map

Routing application with Google Maps

. . . I think the public - the folks who show up to input forums for development projects, and who want to know how far it is to the next lean-to on the trail - are now fully-literate in maps. Mapquest started this process and Google advanced it, but now there are tools beyond those to help you engage your audience on computers and mobile devices (No, Trimble, I am not referring to the Juno. Screw that.). Many of these tools happen to be open-source . . .

Ecoregions of New Zealand, hosted by Github

. . . If you currently describe yourself as a "GIS Analyst/Technician/Monkey" - as I did not long ago - your day largely consists of desktop GIS, and 95% of you are working entirely with some combination of ESRI products. I frequently slag ESRI for bugginess and closed-ness, but they do just fine making a desktop platform for geospatial analysis. However they have never known what "Cartography" means (that's right, you take it to Illustrator if you want your map to look good), and they're just starting to realize that they don't know what "Web Mapping" means either. . .

Population Change in South American Cities, using Mapbox 

. . . The "open-source alternatives" are not replacements for ArcGIS. But through advances in cartography and and browser-based distribution, open tools are unearthing the truth that the rest of us forgot about while sorting through ArcToolBox: that maps tell stories. That your data can be weighted, decoupled, buffered and regressed twenty ways and it means nothing if your cartography is crap. That you can drop cash on a printed poster with a sweet imagery background and it's lost to the ether if two people show up to your hearing, then you post it publicly as a PDF.

These are issues that bedevil public and private sectors, academic and amateur alike. But I say that you can make accurate yet artistic maps and deliver them in a way that is almost universally accessible (at least in the developed world). Let's make maps that people care about.

With the 'Why' of it behind, my next post will be an introduction to the 'How.'

Read more ...