Fun with the Open Data Kit

Posted by & filed under Uncategorized.

I’ve recently rediscovered the Open Data Kit project and have been having a lot of fun playing around with this suite of tools designed for building mobile data collection solutions. ODK is a Google sponsored project, which is supported by the University of Washington’s Department of Computer Science and Engineering. It appears mainly aimed at humanitarian projects in the developing world, but looks like it may be creeping into the business sector as well.

The basic ODK workflow:

  • Design a data collection form
    • ODK forms follow the XForms XML standard and can be built using ODK’s web-based build application. There are several other tools available for building these forms, and they can also be built manually with a simple text editor.
  • Install the ODK Collect app on your Android smartphone
    • This is a generic mobile data collection app that transfers data between your phone and your database server.
  • Configure the ODK Aggregate server to manage your collected data
    • This is a Java based application, which can be deployed on Google’s App Engine infrastructure, or configured locally to connect with a MySQL or PostgreSQL database.

GIS Potential:

ODK has the potential to fill a crucial gap that has existed for years between low-end recreational GPS units (think eTrex) and high-end GPS data collectors (think Trimble). Recreational GPS units are dirt cheap and fairly accurate these days, but they lack any sort of sophisticated data collection capabilities. You get waypoint name and description fields, and a selectable symbol if you are lucky, but this just won’t cut it for most GIS projects where data integrity is paramount. Most recreational GPS units are built on closed platforms (Garmin, Magellen) while the industry standard for high end units seems to be  Windows Mobile (ArcPad, TerraSync).

Smartphones however, offer reasonably cheap hardware paired with sophisticated open operating systems. Most have decent cameras and GPS sensors and they are connected to the web by default. There are tons of GPS and mapping apps already out there, but what we really need is a generic application that works with our custom data dictionaries or forms- and this is where ODK shines!

Tips, Tricks, and Hacks:

As a Google project, ODK works seamlessly with App Engine and Fusion Tables, which is cool in and of itself, but hardcore GIS requires more tweaking. A Java application that uses PostgreSQL just screams out for PostGIS and GeoServer integration! Here’s what you need to know to get this system up and running…

Prerequisites:

  • Android smartphone (for ODK Collect)
  • Java servlet containter (for ODK Aggregate)- tested on Jetty and Apache Tomcat 6
  • GeoServer 2.1.0 or higher for SQL Views (can use database views with < 2.1.0)

Lessons Learned:

  • Play around with the App Engine install first to see what you can do with ODK before you dive into configuring locally
  • Download the configuration wizard from the Google code repository: http://code.google.com/p/opendatakit/downloads/list
  • ODK Aggregate can run in the Jetty servlet container that ships with the GeoServer windows service installer, but I would recommend deploying on Tomcat…
  • Make sure you create your intended ODK postgres database using the PostGIS template!
  • Setup your ODK database as a GeoServer data store and make your layers as GeoServer SQL Views
  • Use the following bit of PostGIS magic in your SQL View to transform the lat, lng coordinates in the ODK database into GeoServer friendly WKB geometries: ST_GeometryFromText(‘POINT(‘||”FORM_FIELDNAME_LNG”||’ ‘||”FORM_FIELDNAME_LAT”||’)', 4326) AS “the_geom”
  • You will have to manually enter the layer bounding box values as you probably won’t have the full extent of your data populated in your table yet. To be safe, you can enter: -180, -85.051, 180, 85.051, which covers the globe avoiding the poles.

ODK + PostGIS + GeoServer + Leaflet:

An exercise in open source mobile GIS data collection: http://geoserving.net/odk_geoserver

17 Responses to “Fun with the Open Data Kit”

  1. Sindile

    Wow this is cool. But trying your application returns the following:

    Connecting to server gives this error Error getting form list form listing failed. null

    I am using ODK Collect 1.1.5

  3. Tumaini

    Hey Bryan, am really new to ODK (couple of days old actually) and though your project above sounds something I would REALLY like to try, it seems aimed more at those with enough experience to follow, e.g. “Make sure you create your intended ODK postgres database using the PostGIS template!”… but I would love to know how to even get ODK working with Postgres to begin with, so for us complete newbies (well, just me I guess :-) ) could you please maybe explain how to set up ODK Aggregate locally and working with Postgres, and then maybe flesh out some of the bullet point instructions above? It would be tremendously useful for people just getting into ODK, and there are not (or dont seem to be, or disjointed efforts in creating) enough documentation to guide the absolute novice through these processes.

    Thanks for understanding, and look forward to hearing from you. Am on tkilimba at ihi dot or dot tz, you can hit me up there maybe if you have the time?

  4. Andy

    I am also trying to you use ODK to collect data using an Android device. I have that working but would like to map those GPS collections in real-time using a Google mashup or Geoserver. Neither I am have any experience with. Any help would be appreciated.

  6. Bryan

    Sebastian,

    If you are doing editing, you may want to avoid the view completely and write a stored procedure or function that inserts the ST_GeometryFromText bit into your geometry field. I have never messed around with the primary key metadata table you referenced.

    BRYAN

  7. Max - The IT Pro

    Great article! I’m curious…
    Aggregate requires a java server to use. How come no one created a PHP version since it’d be cheaper for non-profits to deploy on their cheap shared hosting accounts? You can do EVERYTHING in PHP these days.

    Or, does any1 know of a similar solution to Aggregate that can be used on a shared hosting account? Looking to use Android phones with KoboCollect & it’d be great to connect with a PHP clone of Aggregate or something similar.

  8. Lorenzo

    Hi! this is a really interesting post. A question for you: is your tomcat+Aggregate behind a Web Server (like Apache)? I tried to use Apache with modjk to use Apache on port 80 and Tomcat on port 8080, but the Aggregate applications does not work as expected. Any suggestion?

  9. Neil Hendrick

    We are ODK users (for large scale population surveys on Human Rights in Africa) and because it’s open source we were able to develop a custom version of the software called KoBo. We also built on the idea of digital data collection with some added software. To handle the complex construction of XML surveys, we developed the KoBo Form Builder (http://formbuilder.koboproject.org/KoBoFormDesigner.html) which helps non-programmers create large surveys with complex skip logic. Because we work in Africa, where connectivity is unreliable, we created the Post Processor for offling synching and aggregation of data. This puts all collected data into a CSV. I encourage you to take a look at KoBoProject.org.

  10. Claudio Drews

    Neil, thanks for the tip about KoboProject!!! Great site! Thanks!

Leave a Reply

  • (will not be published)