HubLog

Entities in Scientific News Stories

2009-06-18T13:36:17Z

I ran the text of Guardian articles categorised as 'science' (full text), New York Times articles categorised as 'science and technology' (short sections) and Nature News articles (full text) through OpenCalais to see what entities it identified.

Here are the results.

onabus.com

2009-06-16T11:56:09Z

I've reopened onabus.com, mostly because the excellent London Bus iPhone app now exists and has everything except maps of the routes (presumably just waiting for MapKit).

Also London now has StreetView images, so maximising the info window to see the stop actually works.

The mobile version sort-of works, but it's a bit slow and unfinished.

Annotation of Scientific Articles

2009-06-14T16:40:23Z

I made a web-based interface for curating the results of entity extraction from scientific papers.

It converts XML files to text, passes the text through machine annotators, lets curators add/delete/modify the annotations, then splices the annotations back into the original XML file.

I can't show it publically yet, but here's a screenshot.

Google Wave will be ideal for this, as at the moment only one person can edit at a time.

Now Playing in Songbird

2009-06-14T15:55:59Z

I ported my Now Playing wall to use Songbird's Webpage API, instead of XMPP. It doesn't do as much as the old version did, because it's harder to query/control Songbird than Amarok, but it works pretty well nevertheless.

Live version (load in Songbird while playing a track; you'll need to give it permission to access your library).

Screenshot:

An important part of a recommendation/exploration tool like this is knowing what's already in your local library (though admittedly Spotify is quickly making that irrelevant). I'm integrating it with an Ampache/MySQL database, using Greasemonkey, but obviously that won't work on the web for everyone.

A Private Radio Archive

2009-06-14T15:46:16Z

I made a rolling, three day audio archive of Resonance FM, by recording the MP3 audio stream in 30 minute chunks then using the schedule (which is helpfully in a Google Calendar) to match the files to the programmes, tag the MP3 files and produce an index.

I can't let anyone else use it yet, because of copyrights, but here's a screenshot:

Dealing with election results data

2009-06-11T13:20:53Z

The Guardian produced a set of data that they'd collected for the results of the recent European elections, and published the data as a Google Spreadsheet.

I cloned the spreadsheet and tidied it up (HTML version), then imported it into Google Fusion Tables.

In Fusion Tables I created two separate views of the data - one showing just the number of votes for each party, and one showing the % of votes for each party.

From the Visualize menu, anyone should now be able to visualise that data in different ways: currently the sortable Table, Scatter and Bar visualisations are the most interesting; the Intensity Map would be good, but doesn't yet have enough options to present this data well.

Google's attempt to automatically geocode the location fields is interesting; there needs to be an option to limit the scope of the geocoding, perhaps.

Adding Bing search results to Google

2009-06-04T12:41:28Z

A Greasemonkey script (configurable to select which search engine's results to add) which adds a Google (OpenSocial) Gadget that uses Google's AJAX Feeds API to fetch an RSS (ideally OpenSearch) feed of search results as JSON and inserts them in a floating box on the side of Google search results pages.

Inspired by DeeperWeb, which looks to be built entirely using Google's AJAX Feeds API and Custom Search Engines (plus a little bit of extra code to generate the tag clouds from the search results).

Install the Greasemonkey script then try a Google search (note: you might have to search from Firefox's search bar, as Google have started using AJAX-y searches from the front page that don't use query strings) - the extra results should show up in the top-right-hand corner.

Note that it sends your search query to my server to fetch the results, so better disable the Greasemonkey script once you've tried it out.

Extracting keyphrases from documents using MeSH terms and KEA

2009-06-01T14:30:19Z

KEA extracts keyphrases from a set of documents. The README covers most of this.

Create a folder called 'train' and, for each document in the training set, create a file with extension ".txt" containing the text of the document and a file with extension ".key" containing the known MeSH terms for this document (one per line).
Create a folder called 'test' and, for each document in the test set, create a file with extension ".txt" containing the text of the document.
Download and extract KEA. Fetch meshdata.rdf (a SKOS representation of the MESH hierarchy) and put it in the VOCABULARIES directory.

From within the downloaded KEA folder, set up some environment variables:

export KEAHOME=`pwd`
export CLASSPATH=$CLASSPATH:$KEAHOME:$KEAHOME/lib/commons-logging.jar:$KEAHOME/lib/icu4j_3_4.jar:$KEAHOME/lib/iri.jar\
:$KEAHOME/lib/jena.jar:$KEAHOME/lib/snowball.jar:$KEAHOME/lib/weka.jar:$KEAHOME/lib/xercesImpl.jar:$KEAHOME/lib/kea-5.0.jar

Build the model:
java -Xmx512M kea.main.KEAModelBuilder -l /path/to/training/folder -m articles -v meshdata -f skos -t NoStemmer
Run KEA against the test set, using the model built above:
java -Xmx512M kea.main.KEAKeyphraseExtractor -l /path/to/test/folder -m articles -v meshdata -f skos -t NoStemmer -n 10
There should now be a set of ".key" files in the test folder, containing key phrases corresponding to each of the test documents.

In theory that should be enough, but I'm getting an error when KEA reads in the SKOS vocabulary. It seems to at least work with -v none for now, which doesn't use the vocabulary.

Scraping with YQL Execute

2009-06-01T07:29:20Z

Another attempt at hosted, server-side scraping in Javascript, this time using YQL Execute, which is mostly based around E4X and XPath.

It takes the idea from one of the Execute demo tables to use a CSS2XPath library to convert CSS selectors into XPath (the library handles most selectors well, though not the very newest, like nth-of-type). This allows the selectors to be written using CSS or XPath, which is enough for a lot of cases (but might still have to expanded to allow regular expressions).

Here's an example definition file, and its output.

Clustering documents with CLUTO

2009-05-28T14:47:00Z

After getting a local copy of the metadata for around 360,000 articles, I wanted to use some clustering/topic modelling to divide them into categories for browsing. CLUTO was suggested, and it worked pretty well (though it's only working off the document titles so far, so there isn't much opportunity for semantic analysis - just word matching).

Export all the documents to a file with one document per line. From a MySQL table of document titles, this is as simple as SELECT title FROM articles then export as CSV.
Use doc2mat to convert the list of documents into a term matrix: doc2mat -nostem titles.csv titles.mat
Run vcluster on the matrix to produce 1000 clusters, and ask it to suggest distinctive features and summaries of each cluster:
vcluster -showfeatures -nfeatures 20 -showsummaries cliques titles.mat 1000 > clusters.txt
Parse the clusters file and generate SQL statements for inserting the clusters back into the original database.
Copy the cluster features section of the vcluster output into a new file, and parse it to extract the clusters and their features (there is a libcluto available, and a Perl library, but regular expression parsing was easy enough).
This generates a set of SQL statements for a "clusters" table.
Run the SQL statements to import the clusters data: mysql -u USER -p DATABASE < clusters.sql; mysql -u USER -p DATABASE < cluster-features.sql

Problems with this method:

Each document only gets assigned to one cluster, whereas ideally they could be placed in multiple categories with scores for each category.
There's no immediate way to add new documents to existing clusters, without using a separate tool.
CLUTO isn't open source.