Geocoding (5min)

Marc Tobias Metten

Lokku Ltd.

Overview

• free text => longitude, latitude
• 10 Poland Street, W1F 7NJ => 51.512438, -0.133619
• 100.000s transactions/day (10.000/min peak)

Free Trade Wharf, 340 The Highway, Wapping/Limehouse London
Fountain Road (Flat 2), Edgbaston, UK
15, ASTLEY HOUSE, LONDON
Albert Villas, Gilbert Mews, LEIGHTON BUZZARD, Bedfordshire, LU7 1NF
Lake Lock Drive, Wakefield West Yorkshire
Church Lane, London

nestoria.co.uk

• vertical search engine
  • online real estate market
  • clean, fast, relevant
• startup, Apr 2006
• 6 (... and hiring)
• Debian/MacOS,
  Apache,
  MySQL,
  Perl,
  Mason

nestoria.co.uk

www.nestoria.co.uk/wimbledon/

listing data flow

http://api.nestoria.co.uk/api?action=search_listings&place_name=soho&listing_type=rent
...
"request" : {
  "action" : "search_listings",
  "country" : "uk",
  "place_name" : "soho",
  "listing_type" : "rent",
  "number_of_results" : 20,
},
"response" : {
  "created_unix" : 1176295820,
  "link_to_url" : "http://www.nestoria.co.uk/soho/property/rent/results-20",
  "listings" : [
    {
      "title" : "Flat to let, Richmond Mews"
      "summary" : "Set in the heart of vibrant Soho this funky...",
      "keywords" : "Mews, Unfurnished, Flat, Reception",
      "latitude" : 51.513,
      "longitude" : -0.133674,
      "lister_name" : "Ludlow Thompson",
      "price" : 475,
      "price_formatted" : "475 GBP per week",
      "lister_url" : "http://rd.nestoria.co.uk/rd?l...
      "thumb_url" : "http://limg.nestoria.co.uk/6/0/...",
      ...
    

• input
  • data feeds
  • crawling
• database, search index
• output
  • website (incl map)
  • iframe, widgets (incl map)
  • whitelabel
  • RSS, KML, GeoRSS, OpenSearch
  • API (JSON, XML)
    • Perl, PHP, Python

choosing a geocoder

• commercial services
  • often batch processing
  • restrictions on caching
  • cost per (transaction, quota, user)
• Google/Yahoo!
  • Yahoo! - 5,000 requests/day per IP [1]
  • Google - 50,000 requests per day per API key [2]
• build own
  • still license restrictions (per machine, derived data)
  • huge time investment
  • measure/control accuracy level

geocoder - base data

• RoyalMail Postzon© data file [1]
  • 2.1 mio records with coordinates
  • streets, boroughs, cities, counties
  • no housenumber ranges
• public domain
  • FreeThePostcode.co.uk and New Popular Edition Maps
    • together only 1% postcode coverage
  • population, density
• TeleAtlas (experimental)
  • ERSI shapes, housenumber ranges, adminstrative hierarchy

geocoder - added data

 Soho = area(W1F,W1D,WC2H)+20%
 Clapham Park = center(SW4 8)+1km
 Isle of White = Isle of Wight
 Hyde Park Square = Hyde Park
    

• mispellings
• un-official names (1.500)
• tube, train stations
• universities, parks
  
  
• manual ranking (Waterloo London > Waterloo, Pembrokeshire)
  
  
• user feedback
• log files, click patterns

praise regular expressions

(((^[BEGLMNS][1-9]\d?) | (^W[2-9] ) | ( ^( A[BL] | B[ABDHLNRST] |
C[ABFHMORTVW] | D[ADEGHLNTY] | E[HNX] | F[KY] | G[LUY] | H[ADGPRSUX] |
I[GMPV] | JE | K[ATWY] | L[ADELNSU] | M[EKL] | N[EGNPRW] | O[LX] |
P[AEHLOR] | R[GHM] | S[AEGKL-PRSTWY] | T[ADFNQRSW] | UB | W[ADFNRSV] |
YO | ZE ) \d\d?) | (^W1[A-HJKSTUW0-9]) | (( (^WC[1-2]) | (^EC[1-4]) |
(^SW1) ) [ABEHMNPRVWXY] ) ) (\s*)? ([0-9][ABD-HJLNP-UW-Z]{2}))

housenumbers

\b(\d+[\-\/]?\d*[\-]?[A-F]*[\-\/]?\w*)\b

Challenges

• about 800k streets, 350k unique names in UK
  • 1341x High Street
  • 1319x Station Road
  • 1015x Church Street
• pre-calculate center (mean) point of street
• 140 abbreviations
  • Av, Dr, Rd, Sq, Pl, Pk ...
• cleansing, typos, spacing, utf8, house numbers ...

Measure confidence

• assign each method a confidence or accuracy
  • continuous improvement
• used for quality and NestoriaRank©

Google geocoder confidence [1]

0 - Unknown location
1 - Country
2 - Region (state, province, prefecture, etc.)
3 - Sub-region (county, municipality, etc.)
4 - Town (city, village)
5 - Post code (zip code) <= in UK this is 10
6 - Street
7 - Intersection
8 - Address

Questions?

• mtm -at- lokku.com
• nestoria.co.uk/help/feedback
• nestoria.co.uk/help/developers