Archive for the ‘jQuery’ Category

Working with the Flickr API

Friday, November 12th, 2010

Flickr provides exemplary tools and documentation for their popular API and is also an excellent case study for social classification in the wild, so it’s worth taking a little time to understand their API.

  1. The App Garden is Flickr’s main API documentation page and the best place to start. From here you can “Create an App” (and get an API key), read articles on general topics (the Overview, REST Request Format and JSON Response Format are particularly useful for us), and see an exhaustive (and exhausting) list of methods that the API provides.
  2. Pick a method from that list that sounds like it might work for your purposes. For this example, we’ll look at flickr.tags.getListUserPopular, but any method in the Tags section is likely to be helpful. Skim through the documentation to make sure this looks like the right thing for you.
  3. Test the method using Flickr’s handy API explore tool (link for this method). If you’re logged in to your Flickr account, Flickr will even provide some sample user and photo IDs to fill in as parameters, which is handy. I like to fill in the White House Flickr account ID (35591378@N03), specify the “Do not sign call?” option and then click “Call Method…” to see the actual results in the box below. Flickr also constructs the full request URL for you.
  4. Inspect the results by copying and pasting the URL into a new browser window. View source in your browser to see the structure of the actual response in XML.

Once you have this URL, you have a couple of options. From Python (in a GAE app, say), you can urlfetch this content, use BeautifulSoup to parse the results and then store or analyze this data. Or you can access this URL from the client-side using jQuery. If you want to take that approach, there are a couple of additional steps you’ll want to keep in mind.

  1. Specify the JSON response format, which you can find documented here. You’ll want to add &format=json to the end of your URL.
  2. Access the API using JSONP. If you’re accessing the Flickr API from a standalone web page (rather than from a Chrome extension, say), you’ll need to use jQuery’s $.getJSON method with the ?callback=? option. But, one particular quirk of the Flickr API, you need to rename this to jsoncallback rather than just callback.

Once you’ve made it through all of these steps, you should be able to pull in data from Flickr to use in your language of choice. If you use getListUserPopular, you can construct a graph in Protovis to see the distribution of tags by White House photographers. Please forgive the rudimentary aesthetics of these graphs, I’m still learning Protovis myself.

Linear scale visualization of flickr tags
(The linear scale shows the clear outliers of DC and USA; switching to a log scale makes the rest of the data visible.)
Log scale visualization of flickr tags

This (short) code sample is available in the repository, so you can see code for accessing Flickr, parsing the response, and visualizing the data in Protovis. (If you make improvements to the visualization, feel free to commit your updates!) The sample uses my Flickr API key, so if you’re going to use this for anything beyond exploratory testing, please create and use your own key.

Working around the same-origin policy

Sunday, October 10th, 2010

As part of the basic security model of the Web, sites can’t usually make requests to pages on other domains — if they could, then just visiting any random site on the Web having recently logged in to your email could reveal the entire contents of your email to an attacker! In class we briefly mentioned three ways to work around the same-origin policy:

  1. Use a server-side proxy.
  2. Make a JSONP request to a server that supports it.
  3. Make a request from a privileged context (like in a Chrome Extension).

This blog post will cover generic uses of the first two methods. If you’re creating a standalone page for your projects, you’ll either need to use APIs that already support JSONP, or use our provided proxy to call an existing API for you and wrap it in JSONP. Ryan’s walkthrough tutorial uses a version of this technique for posting new bookmarks to Delicious from a standalone page, but here we’ll access any generic JSON API.

The internal details aren’t vitally important, but in case you’re curious: JSONP works by loading a new <script> element to the page, where the contents of that <script> element just happen to be (no, I’m kidding, it’s not the least bit coincidental) calling a function with a name you defined with a single parameter which is the response from the API. To take advantage of this in jQuery (which, as usual, does all the hard work for you), just use the $.getJSON() function and include a special callback parameter in the URL '?callback=?'.

To use this with the New York Times API, for example, which doesn’t support JSONP, we instead make a JSONP call to a proxy on our own Berkeley servers and that proxy makes the call to the New York Times and then responds using the JSONP callback standard described above. We just need to pass the URL of the New York Times API and all the query parameters as parameters to the proxy.

var query = $('#search').val();

var apiKey = 'yournytimesapikey';
var proxyUrl = '';

$.getJSON(proxyUrl + '?callback=?', {"url": '', 
                                     "query": query, 
                                     "api-key": apiKey}, 

The full sample page is in the iolab10 repository, including the full path to the PHP proxy that we’re running and even a sample NYTimes API key. We’ve also made the PHP code for the proxy available in case you want to inspect it or modify it for your own purposes. (If you install and run the proxy on the same domain that your page runs on you don’t even need to use JSONP!)

A couple of caveats:

  1. If you’re using JSONP from a content script in a Chrome Extension (or from a Greasemonkey extension, for that matter), you’ll receive an error that a function named json123456789 (or some similar nonsense name) doesn’t exist. This is because jQuery created the callback function in its own sandboxed area, but when the script was inserted into the page it called a function in the original window context. To work around this, cross-domain requests in Chrome Extensions shouldn’t use '?callback=?' and should be made from the background page or a pop-up page with cross-domain permissions declared in the manifest. (For more detail, see the relevant Chrome Extensions documentation.)
  2. When you pass a URL to the proxy as the ?url= parameter, the URL itself shouldn’t include the parameters you’re passing on to the API, those parameters should just be additional parameters to the proxy page. Really, this just means that you should use an ampersand for the first parameter to the API. For example, call proxy.php?url= rather than proxy.php?url=
  3. There are potential security implications here that we haven’t gone into yet. Requests through a PHP proxy on Berkeley servers may be logged by Berkeley, which you might not want (particularly if you’re passing a key, password or other secret data). And if the address for your (or our) proxy becomes widely known, it could be abused by others for denial-of-service or other malicious purposes.

Working around the same-origin policy in Greasemonkey

Wednesday, September 16th, 2009

The same-origin policy prevents scripts on a page from communicating with servers on a different domain from the page. Implemented by all browsers, this prevents some cross-site scripting attacks.

Of course, communicating with different web services is a common goal for mashups like the ones we’re building in this class. So if you need to communicate with a different server from the current page, you have three options:

  1. Use a server-side proxy.
  2. Make a JSONP request to a server that supports it.
  3. Use the Greasemonkey chrome to make the cross-site request for you.

Ryan’s walkthrough of the Delicious Trailmaker has code for both of the first two options, so I’ll focus on the third. Also, if you are writing a Greasemonkey script, the third option is the most flexible and straightforward.

First, let’s start with a stub of a Greasemonkey script. This script loads jQuery and inserts some simple CSS and HTML into the page to add a bar to the bottom of the page.

Now, let’s use AJAX to request something on another page. You can read extensive documentation on the jQuery AJAX commands, but for the most part, you can just call $.get() with the URL you want and a callback function for handling the XML response. For example, if you run this on you should see the full contents of the sublets page outputted in your console.

$.get('/sub/', function(xml){

But if you modify that to request the contents of while still on, you’ll see nothing in the console. (You can try this yourself to convince yourself that the same-origin restriction really works.)

$.get('', function(xml){

Now, to workaround this restriction, we’ll add a Javascript file which tells jQuery to use Greasemonkey to make its XmlHttpRequests. The Greasemonkey-jQuery-XHR Bridge is available on our course website.

To add it to our Greasemonkey script, we’ll add a new @require statement to the metadata block at the top of the script:

// @require
// @require
// ==/UserScript==

In order to get these changes, you must uninstall and re-install your script. To verify that it works, refresh that same page and confirm that the contents of appear in the console.

Now that I can talk to other domains, let’s consider my ultimate goal: I want to fetch all the recent tags for the current page from Delicious. To do this, I’ll use the Delicious feeds API. That API requires that I send the MD5 hash of my URL rather than the raw URL itself, and to compute an MD5 hash, I’ll need to include an additional jQuery plugin.

There are jQuery plugins for almost everything, including MD5 hashing. This requires another @require and another reinstallation of the script.

// @require

Now I can produce the URL that the Delicious feeds API requires, and see on the console what kind of output I’ll receive.

var theUrl = $.md5(window.location.href);

$.get('' + theUrl, function(xml){
<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="" xmlns:content="" xmlns:wfw="" xmlns:rdf="" xmlns:dc="" xmlns:cc="" version="2.0">
		<description>bookmark history for</description>
		<atom:link rel="self" type="application/rss+xml" href=""/>
			<title>[from icresource] craigslist - san francisco, east bay, south bay, marin county, california</title>
			<pubDate>Wed, 16 Sep 2009 17:57:38 +0000</pubDate>
			<guid isPermaLink="false"></guid>
			<source url="">icresource's bookmarks</source>
			<category domain="">San_Francisco-RC</category>
			<category domain="">East_Bay-RC</category>
			<category domain="">South_Bay-RC</category>
			<category domain="">Marin_County-RC</category>
			<category domain="">events-calendar</category>
			<category domain="">jobs</category>
			<category domain="">housing</category>
			<category domain="">rideshare</category>
			<category domain="">personals</category>
			<category domain="">classifieds,</category>
			<category domain="">forums</category>

I can see from that output that it’s the <category> elements that I’m interested in. We can use the same jQuery selection tools on this XML output that we do on the Document Object Model by wrapping the xml variable in $() and running the find method. And for each <category> element that I find, I’ll add a list item to the bar.

	$("#bar ul").append('<li>' + $(this).text() + '</li>');

(In the above code, this refers to the DOM object that we’ve selected from the XML file. By wrapping this in $() I can convert it to a jQuery object and I can use all the jQuery functions that I’m familiar with, like .text().)

At this point, you should be successfully grabbing all the recent tags for the current page from Delicious and inserting them in a bar on the page. And besides the @require lines, your code style should be no different than if you were doing same-origin requests.

Here’s the completed script. If you have any questions, problems, corrections or suggestions, please leave a comment and I’ll respond here.

Add jQuery to any (or every) webpage

Saturday, September 12th, 2009

We’ve already seen how Firebug makes it incredibly easy to inspect the current page loaded in Firefox and run jQuery commands to quickly modify items on the page or test different selectors.

But what if the webpage you’re interested in doesn’t already have jQuery installed? jQuery is becoming more and more widespread (the iSchool website has it, heck, even Craigslist, that paragon of simplicity, has it) but not all websites have it loaded. And if the webpage you visit doesn’t load jQuery then you won’t be able to use the jQuery commands from Firebug on that page.

But it’s easy to write a Greasemonkey script that will insert jQuery into any page. We can’t just use @require — that loads jQuery into the Greasemonkey script but then jQuery won’t be around when the Greasemonkey script ends and we’re trying to run our debug commands in Firebug. Instead, we’ll add a <script> element to the head of the page itself.

// ==UserScript==
// @name           Add jQuery
// @namespace
// @description    Insert the jQuery script so that we can run commands in Firebug
// @include        http://*
// @include        https://*
// ==/UserScript==

var GM_JQ = document.createElement('script');
GM_JQ.src = '';
GM_JQ.type = 'text/javascript';

Easy as pie. (I’ve used plain Javascript so that we don’t have to load the whole jQuery library just to load jQuery.) With this userscript installed, you can test any jQuery command on any page you might want to investigate or modify with your own Greasemonkey script.

More details about this technique from Joan Piedra (from whom I’ve adapted our code).