Working with the Flickr API

Flickr provides exemplary tools and documentation for their popular API and is also an excellent case study for social classification in the wild, so it’s worth taking a little time to understand their API.

  1. The App Garden is Flickr’s main API documentation page and the best place to start. From here you can “Create an App” (and get an API key), read articles on general topics (the Overview, REST Request Format and JSON Response Format are particularly useful for us), and see an exhaustive (and exhausting) list of methods that the API provides.
  2. Pick a method from that list that sounds like it might work for your purposes. For this example, we’ll look at flickr.tags.getListUserPopular, but any method in the Tags section is likely to be helpful. Skim through the documentation to make sure this looks like the right thing for you.
  3. Test the method using Flickr’s handy API explore tool (link for this method). If you’re logged in to your Flickr account, Flickr will even provide some sample user and photo IDs to fill in as parameters, which is handy. I like to fill in the White House Flickr account ID (35591378@N03), specify the “Do not sign call?” option and then click “Call Method…” to see the actual results in the box below. Flickr also constructs the full request URL for you.
  4. Inspect the results by copying and pasting the URL into a new browser window. View source in your browser to see the structure of the actual response in XML.

Once you have this URL, you have a couple of options. From Python (in a GAE app, say), you can urlfetch this content, use BeautifulSoup to parse the results and then store or analyze this data. Or you can access this URL from the client-side using jQuery. If you want to take that approach, there are a couple of additional steps you’ll want to keep in mind.

  1. Specify the JSON response format, which you can find documented here. You’ll want to add &format=json to the end of your URL.
  2. Access the API using JSONP. If you’re accessing the Flickr API from a standalone web page (rather than from a Chrome extension, say), you’ll need to use jQuery’s $.getJSON method with the ?callback=? option. But, one particular quirk of the Flickr API, you need to rename this to jsoncallback rather than just callback.

Once you’ve made it through all of these steps, you should be able to pull in data from Flickr to use in your language of choice. If you use getListUserPopular, you can construct a graph in Protovis to see the distribution of tags by White House photographers. Please forgive the rudimentary aesthetics of these graphs, I’m still learning Protovis myself.

Linear scale visualization of flickr tags
(The linear scale shows the clear outliers of DC and USA; switching to a log scale makes the rest of the data visible.)
Log scale visualization of flickr tags

This (short) code sample is available in the repository, so you can see code for accessing Flickr, parsing the response, and visualizing the data in Protovis. (If you make improvements to the visualization, feel free to commit your updates!) The sample uses my Flickr API key, so if you’re going to use this for anything beyond exploratory testing, please create and use your own key.

Prototyping a todo list in Google App Engine

In this walkthrough we’ll build a simple to-do list on Google App Engine so that users can log in and add new items to the to-do list, which are stored and displayed back to them later. (Much of this walkthrough follows the Getting Started documentation from the Google App Engine docs, and I’ll refer to those sections as they match up.)

The first thing to do is to install Google App Engine (available on Mac, Windows and Linux) and the Google App Engine Launcher (available on Mac and Windows platforms). These instructions for installation are helpful; you’ll need Python 2.5 installed, on a Mac it should already be present.

Once it’s installed (on a Mac be sure to copy the App Engine Launcher to your desktop rather than running it from the disk image), open the Google App Engine Launcher, which will show an empty list. To create a new application, click the + button. Enter the name of your app (use a name that’s all lowercase letters and numbers) and where you want to store the files (putting this in your repository would be great, for example).

This will create a folder with three files in it: app.yaml (for configuration), index.yaml (for advanced use of the database) and the file we most care about main.py where our server-side code exists. You can open this folder in the Finder/Explorer and open main.py in your favorite Python IDE or text editor.

Back to the App Engine Launcher: select your application from the list, click “Run” and (hopefully) the small icon next to your app will turn green. Now just click “Browse” to open the application in your browser. “Hello World!” without writing a line of code. Congratulations.

Also, now your application is running on a mini-web server on your machine. This is a huge advantage because you can quickly and easily make changes to the code on your own machine and immediately see the effects by refreshing your browser. You don’t need to install, maintain or configure a web server, App Engine’s mini-server is fine. To test this quick updating, change “Hello World” in main.py to “Hello IO Lab” and refresh to prove to yourself that updates work.

Now, Google provides all these neat services, so let’s take advantage of one: users. This way we can welcome the user personally rather than welcoming the entire class. (Getting Started explanation | Reference Documentation)

Copying from the documentation, add this new Python library import at the top of your main.py:

	from google.appengine.api import users

And inside the get(self): method, let’s replace the existing single line with the following:

    user = users.get_current_user()

    if user:
      self.response.out.write('Hello, ' + user.nickname())
    else:
      self.redirect(users.create_login_url(self.request.uri))

(Be careful about indentation. Python requires that all indentation be done the same way: either all with spaces or all with tabs. And indentation must line up or your program will fail to execute.)

If you refresh the page now, your local mini-server will come up with a little fake login page, where you can enter any username you want and click “Login”. You should then be welcomed by email address. How would this work in the real world? Let’s upload our app to Google and see. (Getting Started docs)

From the Launcher, click “Dashboard” for this app. Oh noes, it doesn’t exist! Click “Return to Applications screen” to see your list of Applications (you’ll probably have none) and click the button to “Create Application”. Pick an available subdomain name (which is also called your appid) and give your application a title. If this is your first app, Google will make you jump through some hoop that includes SMS verification. This shouldn’t take too long, but you can keep testing on your own machine without uploading, if you want.

You’ll need to specify your selected appid in the app.yaml configuration file in your application’s folder. Enter it after application: .

Now, back in the Launcher, just click “Deploy”, enter your Google username and password and your application files will be uploaded to Google’s servers. Then go to http://<your-appid>.appspot.com and your code is running, except with a real Google login. Congratulations, your application is running in the cloud! Clicking “Dashboard” in the Launcher now will show you a web page with stats and various information about your app’s execution on Google servers.

Most of the time, though, we want to test by using the mini web-server, which picks up changes automatically. So let’s switch back to that by clicking “Run” again. (Deploying your application to the production server will stop it on your mini server.)

Now, we’re combining our presentation and behavior here, which we’ve told you is a bad thing. So let’s move the presentation into an HTML file and keep just the logic in Python.

Create a new file in the same folder called index.html (the HTML skeleton file on the course web site is a good place to start). Here you can fill in the <title> element, make things look nice, and not be mixed up with Python. We’ll leave a little placeholder for the user’s nickname, using a templating language. (Getting Started docs | Django Templating docs)

    <html>
    ...
    <h1>Hello, {{nickname}}</h1>

Now, in main.py, we’ll import two additional libraries:

    import os
    from google.appengine.ext.webapp import template

And then render the template with values we store in a simple Python dictionary:

    nickname = user.nickname()
    template_values = {'nickname': nickname}

    path = os.path.join(os.path.dirname(__file__), 'index.html')
    self.response.out.write(template.render(path, template_values))

Refreshing the page in your browser now should show a nicely formatted page with a large heading welcoming the user by their nickname. Now let’s make this a little more interactive, by adding a form where a user can add a to-do item. In index.html:

	<form action="" method="post">
		<input type="text" name="item" value="To-do item"/>
		<input type="submit"/>
	</form>

I’m using action="" just because it’s the simplest possible thing: it’ll call back to the same page, but use post instead of the usual get, since we’re submitting data. To respond to the post form submission, we define a new function in the same class:

    def post(self):
      self.response.out.write(self.request.get("item"));

Just enough to prove that it works.

Now, we’ll actually do something with the entered data, like storing it. (Getting Started docs | API Reference) To do this, we need to define a Model, a representation of the data we’re going to store. Import the db library, then define the Todo class which will be stored in the database.

    from google.appengine.ext import db

    class Todo(db.Model):
      author = db.UserProperty()
      item = db.StringProperty()
      completed = db.BooleanProperty()
      date = db.DateTimeProperty(auto_now_add=True)

And then use that class to save new Todo’s in the post function:

    todo = Todo()

    todo.author = users.get_current_user()
    todo.item = self.request.get("item")
    todo.completed = False

    todo.put()

    self.response.out.write("Saved: " + self.request.get("item"));

Now, that data has supposedly been stored, let’s see it. Back in the Launcher, click “SDK Console”. In the Datastore Viewer, you can choose a type of entity and then list the entities in that store. (When you’re running live on the server, you’ll have a similar view from the Dashboard.)

Let’s expose that data back in our little web app. Change the get handler to query for all the Todo items in the datastore and store them in the template_values dictionary:

    todos = Todo.all()

    template_values = {'nickname': nickname, 'todos': todos}

And then change the template to loop through them and create list items for each one:

    <ul>
    {% for todo in todos %}
    	<li>{{ todo.item }} by {{ todo.author.nickname }}</li>
    {% endfor %}
    </ul>

Great, now when we add something and go back to the main page and refresh, we see the new content. Let’s make that step automatic though.

    self.redirect('/')

(This goes at the end of the post function; be sure to remove or comment out any self.response.out.write lines.)

What else could we add to this? You could add CSS and Javascript to this HTML page (and with a small workaround, you can add them as static files). Since these are to-do items, you could add checkboxes and make clicking a checkbox into an AJAX post that would mark the item as completed. You could filter to only show recently added todo items, or only your todo items.

A working version of this code is available in the repository.

Working around the same-origin policy

As part of the basic security model of the Web, sites can’t usually make requests to pages on other domains — if they could, then just visiting any random site on the Web having recently logged in to your email could reveal the entire contents of your email to an attacker! In class we briefly mentioned three ways to work around the same-origin policy:

  1. Use a server-side proxy.
  2. Make a JSONP request to a server that supports it.
  3. Make a request from a privileged context (like in a Chrome Extension).

This blog post will cover generic uses of the first two methods. If you’re creating a standalone page for your projects, you’ll either need to use APIs that already support JSONP, or use our provided proxy to call an existing API for you and wrap it in JSONP. Ryan’s walkthrough tutorial uses a version of this technique for posting new bookmarks to Delicious from a standalone page, but here we’ll access any generic JSON API.

The internal details aren’t vitally important, but in case you’re curious: JSONP works by loading a new <script> element to the page, where the contents of that <script> element just happen to be (no, I’m kidding, it’s not the least bit coincidental) calling a function with a name you defined with a single parameter which is the response from the API. To take advantage of this in jQuery (which, as usual, does all the hard work for you), just use the $.getJSON() function and include a special callback parameter in the URL '?callback=?'.

To use this with the New York Times API, for example, which doesn’t support JSONP, we instead make a JSONP call to a proxy on our own Berkeley servers and that proxy makes the call to the New York Times and then responds using the JSONP callback standard described above. We just need to pass the URL of the New York Times API and all the query parameters as parameters to the proxy.

var query = $('#search').val();

var apiKey = 'yournytimesapikey';
var proxyUrl = 'http://courses.ischool.berkeley.edu/path/to/proxy.php';

$.getJSON(proxyUrl + '?callback=?', {"url": 'http://api.nytimes.com/svc/timestags/suggest', 
                                     "query": query, 
                                     "api-key": apiKey}, 
 function(json){
    console.log(json.results);
});

The full sample page is in the iolab10 repository, including the full path to the PHP proxy that we’re running and even a sample NYTimes API key. We’ve also made the PHP code for the proxy available in case you want to inspect it or modify it for your own purposes. (If you install and run the proxy on the same domain that your page runs on you don’t even need to use JSONP!)

A couple of caveats:

  1. If you’re using JSONP from a content script in a Chrome Extension (or from a Greasemonkey extension, for that matter), you’ll receive an error that a function named json123456789 (or some similar nonsense name) doesn’t exist. This is because jQuery created the callback function in its own sandboxed area, but when the script was inserted into the page it called a function in the original window context. To work around this, cross-domain requests in Chrome Extensions shouldn’t use '?callback=?' and should be made from the background page or a pop-up page with cross-domain permissions declared in the manifest. (For more detail, see the relevant Chrome Extensions documentation.)
  2. When you pass a URL to the proxy as the ?url= parameter, the URL itself shouldn’t include the parameters you’re passing on to the API, those parameters should just be additional parameters to the proxy page. Really, this just means that you should use an ampersand for the first parameter to the API. For example, call proxy.php?url=http://example.com&param1=value1&param2=value2 rather than proxy.php?url=http://example.com?param1=value1&param2=value2.
  3. There are potential security implications here that we haven’t gone into yet. Requests through a PHP proxy on Berkeley servers may be logged by Berkeley, which you might not want (particularly if you’re passing a key, password or other secret data). And if the address for your (or our) proxy becomes widely known, it could be abused by others for denial-of-service or other malicious purposes.

Delicious Trailmaker Tutorial

In our first class on August 31, we did an in-class demo showing one possible way to implement trails from Vannevar Bush’s Memex using the Delicious API. This tutorial reviews the steps from class. At the end of this tutorial, you will have constructed a small web application that saves a collection of bookmarks to Delicious as a trail. A trail is identified by a set of special tags: trail:[name_of_trail] and step:[step_number].

Getting Started

0. Before getting started, you may want to create a separate Delicious account for experimenting. The Delicious API has no undo, and you don’t want a mistake to erase any existing bookmarks. Also, unless you use a properly configured proxy server, your password will be sent in the clear. You may want to change your password to something that isn’t used by another of your other accounts.

1. Start with a basic HTML page that includes Google-hosted copies of jQuery and jQuery UI. You can download this template at http://courses.ischool.berkeley.edu/i290-iol/f10/resources/html_skeleton.html

The code samples in this tutorial are broken up by explanations. For the uninterrupted code, you can download the completed Trailmaker at http://courses.ischool.berkeley.edu/i290-iol/f10/resources/delicious_trailmaker.html

2. Create the basic structure that you will need in HTML. This includes a <form> to specify a username, a <div> and <ul> to display loaded bookmarks, and a <div> and a <ul> to create a new trail. The following structure is a good start:

<form id="load-bookmarks" method="get">
<h2 id="enter_a_delicious_username">Enter a Delicious username:</h2>
    <input type="text" name="username" id="username" value="" />
    <input type="submit" value="Get Bookmarks" />
</form>

<div id="bookmarks">
<h2 id="bookmarks">Bookmarks</h2>
    <ul></ul>
</div>

<div id="new-trail">
<h2 id="new_trail">New Trail</h2>
    <ul></ul>
    <form id="save-trail" method="post">
        Username: <input id="save-username" type="text" name="save-username" /><br />
        Password: <input id="save-password" type="password" name="save-password" /><br />
        <input type="button" name="make-trail" value="Make New Trail" id="make-trail" />                
    </form>
</div>

Loading Bookmarks

3. When a user enters an account in the #load-bookmarks form, we want to load the Delicious bookmarks for that account. Delicious provides a feeds API that lets us get the bookmarks for a user at http://feeds.delicious.com/v2/json/{username}.

First, we create a global object called “delicious” to store settings. Then we want to attach the JavaScript for this document once the page is “ready,” so we use $(document).ready() inside a <script> tag in the document <head>. You put an anonymous callback function inside .ready() which executes once the page is in memory:

<script>
// Create an empty global object where we can store settings for connecting to Delicious
var delicious = {};

$(document).ready(function() {
    // What you want to happen when the page loads
});
</script>

In this ready function, we’ll attach a function to the submit event of the #load-bookmarks form. When the form is submitted (by pressing the return key or by clicking on the “Get Bookmarks” button), this function will run. The last thing this function does should be return false;. The browser’s default behavior when submitting a form is to load a new page, and we use return false to prevent this action.

$('#load-bookmarks').submit(function(){
    // What you want to happen when the load bookmarks form is submitted
    return false;
});

CSS selectors are the basis for nearly all of the jQuery Javascript library. Most programming occurs by selecting some group of elements on the current page and manipulating them. Because the form has the id attribute set to ‘load-bookmarks’, we can select this element using #load-bookmarks—the pound sign (#) is used to indicate an id in CSS. Then we attach a function to the submit() event of the form. jQuery can attach functions (which is called “binding”) to many kinds of events, which are described in the jQuery documentation.

The functions that we use on inside .ready() and .submit() do not have a name—they are “anonymous functions,” which are a feature of Javascript and very common. They allow you to define commands that you want to happen inline, without defining a function somewhere else in your code. We’ll discuss anonymous functions more in lecture two; You can also read this guide to anonymous functions in JavaScript.

4. When the form is submitted, we want to get the value of the account name entered in the form, which we do with var username = $('#username').val(); Although Javascript does not require var in variable declarations, you should always use it. Then we use the getJSON() method to get the bookmarks from Delicious. You can provide a function to getJSON that is called when the request is complete. In fact, if you want anything to happen when the request is returned, you have to provide a function to getJSON because of the asynchronous nature of AJAX requests. This function receives the response from the server, which you can use to construct an HTML list item and add it to the list of bookmarks.

var username = $('#username').val();
// This cross-domain request requires that you use '?callback=?' because it is done using JSONP
$.getJSON('http://feeds.delicious.com/v2/json/' + username + '?callback=?',
function(json){
    // json contains the response provided by the server
    // You can use console.log(json) to examine the response
   $(json).each(function(index) {
       // this.u // url
       // this.d // description
       // this.n // extended notes
       // this.t // array of tags

        // Create an HTML string for the bookmark
        // We also use the .data method to store the notes and tags
        // of the bookmark in the DOM with the object
        // Finally, we add it to the list of bookmarks with appendTo()
       $('<li></li>').html('<a href="' + this.u + '">' + this.d + '</a>')
        .data('extended', this.n)
        .data('tags', this.t)
        .appendTo('#bookmarks ul');
   });

   // Later, you'll add dragging functionality here

});
return false;    

Now entering a Delicious user’s account name and clicking “Load Bookmarks” works.

Creating a new trail

5.We want to let users drag loaded bookmarks to the New Trail box and rearrange them before saving the trail. Before we do this, we should make the #bookmarks box and the #new-trail box actually appear side-by-side. Add these lines to between the <style></style> tags in your document head. This makes each box roughly half the width of the screen, and the float: left makes them appear side-by-side:

#bookmarks, #new-trail {
    float: left;
    width: 48%;
    margin-right: 1%;
    min-height: 300px;
    border: 1px solid #666;
}

6. The jQuery UI library makes it easy to script complex actions like dragging and dropping. First, we want to make each of the loaded bookmarks draggable. We can do this with the .draggable() method when we load each bookmark. Add the following line at the end of your getJSON callback function:

$('#bookmarks li').draggable({revert: true});

7. Now we need a place to drop these draggable elements. We’ll make the entire #new-trail box a droppable area, and define a function that copies the bookmark to the list when the user drags a loaded bookmark to the area. Notice that the droppable method only takes one argument, a Javascript object (indicated by the curly braces {}) that provides various parameters by name. This is common practice in jQuery and jQuery UI. Here we are providing the accept and drop parameters.

$('#new-trail').droppable({
    accept: 'li', // This droppable area accepts li elements
    drop: function(event, ui) {
        // Don't confuse ul, the <ul> unordered-list with ui, the user interface element
        // .draggable('disable') says that we want to remove the draggable behavior
        $(ui.draggable).draggable('disable').appendTo('#new-trail ul');
    }
});    

We also want to make the list of bookmarks in the new trail list sortable, to allow a user to re-arrange the order of the points in the trail. To do this we can use the .sortable() function. Insert this line in your document .ready() function. Simple, right?

$('#new-trail ul').sortable();    

Saving the trail

8. Finally, we want to send each bookmark in this newly created trail to a specified user account when it is saved. We start by attaching a function to the submit of the #save-trail form. Remember to return false at the end of the function!

$('#save-trail').submit(function() {
    // Let's ask the user for a name for the trail
    // We are storing the name that the user enters as the text of the
    // h2 in the #new-trail div
    // The || syntax here lets us specify a default value
    $('#new-trail h2').text(prompt('Enter a name for your trail:') || 'My New Trail');

    // Store the username and password to send with each request
    // This isn't the best security practice, but we do it here
    // in the interest of brevity
    delicious.username = $('#save-username').val();
    delicious.password = $('#save-password').val();
    delicious.stepNum = 0;

    saveTrail();
    return false;
});    

In the saveTrail() function we want to gather the information for a single bookmark and post it to Delicious. We have to do this one list item at a time because the Delicious API only provides a call to save a single bookmark. When we have successfully saved one bookmark, we move on to the next one. When they are all done, we let the user know that saving was successful.

In order to update a bookmarks on Delicious, you need to re-save all the information from the old bookmark (including URL, description, notes, and tags) with your changes. Fortunately, we stored all this information using jQuery’s .data() method when we initially loaded the bookmarks. .data() lets you store any information you want in the DOM with an element.

function saveTrail () {
    // We need to keep track of which bookmark number we are saving, so we
    // can use the `step:2` syntax that we have established
    // When the user submitted the form we started with stepNum = 0,
    // so we can increment it each time we call saveTrail
    delicious.stepNum++;

    // Change spaces in the trail name to underscores to follow our trail syntax
    // By default, the .replace() method doesn't replace ALL the occurrances
    // of a string, so we are using the global flag in our regular expression
    // to replace everything. The global flag is set with the "g" after
    // the regular expression (/ /g)
    var newTrailName = 'trail:' + $('#new-trail h2').text().toLowerCase().replace(/ /g, '_');

    // Get the first bookmark to save, which is the first element of the #new-trail list
    var bookmark = $('#new-trail li:first');

    // Assemble the data to send to Delicious
    var postData = {
        url: bookmark.find('a').attr('href'),
        description: bookmark.find('a').html(),
        extended: bookmark.data('extended'),
        tags: bookmark.data('tags').join(' ') + ' ' + newTrailName + ' ' + 'step:' + delicious.stepNum,
        method: 'posts/add',
        username: delicious.username,
        password: delicious.password
    };

Here we have constructed postData, which is a Javascript objet with keys and values for the data we are going to send to the server. The first four values are for Delicious: the url we are saving, its description (which is the title, in Delicious parlance), the extended notes for the bookmark, and the tags. Note the most important part is occurring on the line with the tags assignment: we are getting the existing tags and adding the new trail name, as well as the step number.

We stored the first line of the #new-trail list in bookmark so we could use it repeatedly. The .find() method lets us search for some potion of the document within the current selection. So .find('a').attr('href') finds the link tag within the bookmark and gets the href attribute for it.

We are also sending the method of the Delicious API that we are calling and the user’s Delicious username and password.

Next, we use jQuery’s $.post method to send the data to Delicious. You are welcome to use this code as is without inspecting it closely.

// Send the data to Delicious through a proxy and handle the response
// Use $.post if the script is located on the same server
// Otherwise, use $.get to avoid cross-domain problems
//$.post('delicious_proxy.php',
// If you are running this on your own machine, you may need to swap the $.post line for this $.getJSON line
$.getJSON('http://courses.ischool.berkeley.edu/i290-iol/f10/resources/delicious_proxy.php?callback=?',
  postData,
 function(rsp){
        if (rsp.result_code === "access denied") {
            alert('The provided Delicious username and password are incorrect.');
        } else if (rsp.result_code === "something went wrong") {
            alert('There was an unspecified error communicating with Delicious.');
        } else if (rsp.result_code === "done") {
            // Bookmark was saved properly
            $('#new-trail li:first').remove(); // Remove the bookmark we just saved
            if ($('#new-trail li').length > 0) {
                // If there are any bookmarks left to save
                // Save the next bookmark in the trail in 1000ms (1 second)
                // We have to wait this period of time to comply with the
                // terms of the Delicious API. If we don't we may have access denied.
                setTimeout(saveTrail, 1000);
            } else {
                // We're done saving the trail
                delicious.password = null; // Don't store the user's password any longer than we need to
                alert ("Your trail has been saved!");
            }
        }
    });
}

You’re done with the basic tutorial! If you have any questions, ask ryan@ischool or npdoty@ischool via email, or bring them to lecture two.

Improvements

We can also add some extra niceties, like making sure links open in a new window so a person doesn’t lose his or her work in creating a new trail. Adding this code to the ready function makes all links open in a new window.

$('a').live('click', function() {
    window.open($(this).attr('href'));
    return false;
});    

You might want to provide some feedback while the trail is being saved. Since each bookmark has to be saved with a delay of 1 second, saving a trail could take a long time, and the user should know that something is happening.

You might disable the “Save Trail” button when the form is submitted using jQuery’s .attr() method

Also, our JavaScript does not check if a user actually entered a name for his or her trail, nor if the user entered a username or password. You could implement a check to make sure these values have been entered before saving the trail.

Additional Information

The Delicious API has a variety of methods available. To update an existing bookmark, use the posts/add method and include the replace=no parameter.

If you are going to be using the Delicious API via JavaScript you should change your password to something different from your other online accounts. Because of browsers limitations to prevent cross-site scripting, you have to send your password as a URL GET request. Although this connection happens via https, your password would appear in log files. If you’re interested in why this has to be the case, ask us. Alternately, you can download the PHP proxy script yourself, run it in your own web space (change the .txt extension to .php), and POST securely to it.

Common AppEngine bulk upload issues

Google’s documentation for AppEngine provides a very nice walkthrough for uploading and downloading data. But it doesn’t necessarily help with troubleshooting. Here are a few problems that multiple members of the class have run into and their solutions.

Often you’ll store your database models in a common file (models.py, say) so that your bulk upload and your main code can access the same model. A common error you’ll have in this case is that importing models.py will fail as follows:

  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 3467, in LoadConfig
    ('', 'r', imp.PY_SOURCE))
  File "loader.py", line 10, in 
    import models
ImportError: No module named models

This has to do with the PYTHONPATH environment variable. I’m guessing that because appcfg.py is located in another directory, Python won’t search the current directory for .py files to import modules from. So you want to set (or ideally, append onto) PYTHONPATH to have “.” or the current directory.

How that works on your system depends on your shell. For me, using bash, this worked:

export PYTHONPATH=$PYTHONPATH:.

I expect it’s significantly different on Windows, but a search for how to set PYTHONPATH in Windows should give you some hints.

Next, you may run into authentication errors. Uploading to appspot.com, you might see an error like the following:

[INFO    ] Connecting to test-env.appspot.com/remote_api
[ERROR   ] Exception during authentication
Traceback (most recent call last):
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 3063, in Run
   self.request_manager.Authenticate()
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1148, in Authenticate
   remote_api_stub.MaybeInvokeAuthentication()
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 545, in MaybeInvokeAuthentication
   datastore_stub._server.Send(datastore_stub._path, payload=None)
 File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 344, in Send
   f = self.opener.open(req)
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 389, in open
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 502, in http_response
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 427, in error
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 361, in _call_chain
 File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 510, in http_error_default
HTTPError: HTTP Error 404: Not Found
[INFO    ] Authentication Failed

When you see “Authentication Failed” after a 404 error, that’s likely a sign that you have the wrong handler in place to accept the remote_api calls. In app.yaml, make sure that you have a URL handler for /remote_api in your app.yaml and make sure it’s listed above any catch-all handler for main.py. It should look like the following:

handlers:
- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin
- url: .*
  script: main.py

If you’re running the bulk uploader against your development server running on your own machine, then (as per this question and this walkthrough) you should leave out the login: admin line in order to avoid having to authenticate. If you are asked to authenticate, you should use your Gmail username and password. And if you’ve entered the wrong username and password at some point, you may need to delete your cookies file.

Uploading or downloading data often exposes problems with non-ASCII characters in either your CSV file or your AppEngine datastore. If you’re uploading non-ASCII data, make sure to encode data in UTF-8 rather than using the plain str function (as described in this bug thread).

class MyLoader(Loader):
  def __init__(self):
    Loader.__init__(self, 'MyModel', [('field1', lambda x: unicode(x, 'utf-8'))])

And our own Karen Nomorosa and Hyunwoo Park discovered that for downloading non-ASCII data from AppEngine, you should modify this slightly:

lambda x: x.encode('utf-8')

Other problems (or answers) for AppEngine bulk uploading or downloading? Let us know, in the comments or via email.

Avoiding undefined console

When you’re writing JavaScript it’s handy to use console.log() to output debugging information to the Firebug or Safari console. The only problem is that this code will cause an error on browsers that don’t have a console object, like IE, or like Firefox without Firebug. One solution is to remove or comment out all the console.log statements, but this is tedious and easy to forget. There are two better options.

First, if you’re trying to debug your code on other web browsers, you can include Firebug Lite on your page. This script mimics Firebug’s functionality in other browsers. It also handles any console.log statements, so your script won’t crash on these pages.

If you simply want to avoid errors, you can define console.log to be an empty function. Add this line at the top of your main script:

// Define console to avoid errors when Firebug isn't available
if(!window.console){ window.console = {log:function(){}}; }

This checks to see if Firebug is turned on (by looking for a global console object. If console doesn’t exist, it creates console and adds log as a function that doesn’t do anything. If you have used other console features, like .info() or .time(), you would have to add those as empty functions as well.

Working around the same-origin policy in Greasemonkey

The same-origin policy prevents scripts on a page from communicating with servers on a different domain from the page. Implemented by all browsers, this prevents some cross-site scripting attacks.

Of course, communicating with different web services is a common goal for mashups like the ones we’re building in this class. So if you need to communicate with a different server from the current page, you have three options:

  1. Use a server-side proxy.
  2. Make a JSONP request to a server that supports it.
  3. Use the Greasemonkey chrome to make the cross-site request for you.

Ryan’s walkthrough of the Delicious Trailmaker has code for both of the first two options, so I’ll focus on the third. Also, if you are writing a Greasemonkey script, the third option is the most flexible and straightforward.

First, let’s start with a stub of a Greasemonkey script. This script loads jQuery and inserts some simple CSS and HTML into the page to add a bar to the bottom of the page.

Now, let’s use AJAX to request something on another page. You can read extensive documentation on the jQuery AJAX commands, but for the most part, you can just call $.get() with the URL you want and a callback function for handling the XML response. For example, if you run this on http://sfbay.craigslist.org you should see the full contents of the sublets page outputted in your console.

$.get('/sub/', function(xml){
    console.log(xml);
});

But if you modify that to request the contents of delicious.com while still on craigslist.org, you’ll see nothing in the console. (You can try this yourself to convince yourself that the same-origin restriction really works.)

$.get('http://delicious.com/', function(xml){
    console.log(xml);
});

Now, to workaround this restriction, we’ll add a Javascript file which tells jQuery to use Greasemonkey to make its XmlHttpRequests. The Greasemonkey-jQuery-XHR Bridge is available on our course website.

To add it to our Greasemonkey script, we’ll add a new @require statement to the metadata block at the top of the script:

...
// @require	http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// @require	http://courses.ischool.berkeley.edu/i290-4/f09/resources/gm_jq_xhr.js
// ==/UserScript==

In order to get these changes, you must uninstall and re-install your script. To verify that it works, refresh that same craigslist.org page and confirm that the contents of delicious.com appear in the console.

Now that I can talk to other domains, let’s consider my ultimate goal: I want to fetch all the recent tags for the current page from Delicious. To do this, I’ll use the Delicious feeds API. That API requires that I send the MD5 hash of my URL rather than the raw URL itself, and to compute an MD5 hash, I’ll need to include an additional jQuery plugin.

There are jQuery plugins for almost everything, including MD5 hashing. This requires another @require and another reinstallation of the script.

// @require	http://plugins.jquery.com/files/jquery.md5.js.txt

Now I can produce the URL that the Delicious feeds API requires, and see on the console what kind of output I’ll receive.

var theUrl = $.md5(window.location.href);

$.get('http://feeds.delicious.com/v2/xml/url' + theUrl, function(xml){
    console.log(xml);
});
<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://web.resource.org/cc/" version="2.0">
	<channel>
		<title>Delicious/url/ad1706731c138951229c69f68c6a1c23</title>
		<link>http://delicious.com/url/ad1706731c138951229c69f68c6a1c23</link>
		<description>bookmark history for http://sfbay.craigslist.org/</description>
		<atom:link rel="self" type="application/rss+xml" href="http://feeds.delicious.com/v2/xml/url/ad1706731c138951229c69f68c6a1c23"/>
		<item>
			<title>[from icresource] craigslist - san francisco, east bay, south bay, marin county, california</title>
			<pubDate>Wed, 16 Sep 2009 17:57:38 +0000</pubDate>
			<guid isPermaLink="false">http://delicious.com/url/ad1706731c138951229c69f68c6a1c23#icresource</guid>
			<link>http://sfbay.craigslist.org/</link>
			<dc:creator><![CDATA[icresource]]></dc:creator>
			<comments>http://delicious.com/url/ad1706731c138951229c69f68c6a1c23</comments>
			<wfw:commentRss>http://feeds.delicious.com/v2/rss/url/ad1706731c138951229c69f68c6a1c23</wfw:commentRss>
			<source url="http://feeds.delicious.com/v2/rss/icresource">icresource's bookmarks</source>
			<category domain="http://delicious.com/icresource/">San_Francisco-RC</category>
			<category domain="http://delicious.com/icresource/">East_Bay-RC</category>
			<category domain="http://delicious.com/icresource/">South_Bay-RC</category>
			<category domain="http://delicious.com/icresource/">Marin_County-RC</category>
			<category domain="http://delicious.com/icresource/">events-calendar</category>
			<category domain="http://delicious.com/icresource/">jobs</category>
			<category domain="http://delicious.com/icresource/">housing</category>
			<category domain="http://delicious.com/icresource/">rideshare</category>
			<category domain="http://delicious.com/icresource/">personals</category>
			<category domain="http://delicious.com/icresource/">classifieds,</category>
			<category domain="http://delicious.com/icresource/">forums</category>
		</item>
		<item>
...

I can see from that output that it’s the <category> elements that I’m interested in. We can use the same jQuery selection tools on this XML output that we do on the Document Object Model by wrapping the xml variable in $() and running the find method. And for each <category> element that I find, I’ll add a list item to the bar.

$(xml).find("category").each(function(){
	$("#bar ul").append('<li>' + $(this).text() + '</li>');
});

(In the above code, this refers to the DOM object that we’ve selected from the XML file. By wrapping this in $() I can convert it to a jQuery object and I can use all the jQuery functions that I’m familiar with, like .text().)

At this point, you should be successfully grabbing all the recent tags for the current page from Delicious and inserting them in a bar on the page. And besides the @require lines, your code style should be no different than if you were doing same-origin requests.

Here’s the completed script. If you have any questions, problems, corrections or suggestions, please leave a comment and I’ll respond here.

Add jQuery to any (or every) webpage

We’ve already seen how Firebug makes it incredibly easy to inspect the current page loaded in Firefox and run jQuery commands to quickly modify items on the page or test different selectors.

But what if the webpage you’re interested in doesn’t already have jQuery installed? jQuery is becoming more and more widespread (the iSchool website has it, heck, even Craigslist, that paragon of simplicity, has it) but not all websites have it loaded. And if the webpage you visit doesn’t load jQuery then you won’t be able to use the jQuery commands from Firebug on that page.

But it’s easy to write a Greasemonkey script that will insert jQuery into any page. We can’t just use @require — that loads jQuery into the Greasemonkey script but then jQuery won’t be around when the Greasemonkey script ends and we’re trying to run our debug commands in Firebug. Instead, we’ll add a <script> element to the head of the page itself.

// ==UserScript==
// @name           Add jQuery
// @namespace      http://people.ischool.berkeley.edu/~npdoty
// @description    Insert the jQuery script so that we can run commands in Firebug
// @include        http://*
// @include        https://*
// ==/UserScript==

var GM_JQ = document.createElement('script');
GM_JQ.src = 'http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js';
GM_JQ.type = 'text/javascript';
document.getElementsByTagName('head')[0].appendChild(GM_JQ);

Easy as pie. (I’ve used plain Javascript so that we don’t have to load the whole jQuery library just to load jQuery.) With this userscript installed, you can test any jQuery command on any page you might want to investigate or modify with your own Greasemonkey script.

More details about this technique from Joan Piedra (from whom I’ve adapted our code).

Using separate files for CSS and JS

One of your classmates rightly noted that Nick and I have said that you can (and often should) keep your HTML, CSS, and JavaScript in separate files, but have not provided any instructions about how to do that.

Single File

Using a single file for HTML, CSS, and JS

Using a single file for HTML, CSS, and JS

The single-file approach is good for quick one-off tests or demonstrations. This is where you put all your CSS and JavaScript in the same file as your HTML. Your single .html document will look something like this:

<html>
<head>
    <style type="text/css">
    /* CSS goes here in the style tag. */
    </style>

    <script type="text/javascript">
    // JavaScript goes here in the script tag.
    </script>
</head>
<body>
    HTML content goes here in the body tag.
</body>
</html>

Although it is fast to get started and there are fewer files to keep track of, there are some significant downsides to keeping everything in one HTML file. For example, if you put your CSS and JavaScript in the same file with your HTML, you can’t reuse the same styles and code on other pages without needless duplication.

Multiple Files

Multiple Files for HTML, CSS, and JS

Multiple Files for HTML, CSS, and JS

The alternative is to store your HTML, CSS, and JavaScript in separate files. In the main .html file you use a <link> tag to include the external stylesheet and a <script> tag to include the external JavaScript. Now your .html document will look like this:

<html>
<head>
    <link rel="stylesheet" href="file.css" type="text/css" media="screen" />
    <script type="text/javascript" src="file.js"></script>
</head>
<body>
    HTML content goes here in the body tag.
</body>
</html>

Note that the CSS file is specified by href="..." while the JavaScript file is specified by src="..."

Delicious Trailmaker Tutorial

In our first class on September 1, we did an in-class demo showing one possible way to implement trails from Vannevar Bush’s Memex using the Delicious API. This tutorial reviews the steps from class. At the end of this tutorial, you will have constructed a small web application that saves a collection of bookmarks to Delicious as a trail. A trail is identified by a set of special tags: trail:[name_of_trail] and step:[step_number].

Getting Started

0. Before getting started, you may want to create a separate Delicious account for experimenting. The Delicious API has no undo, and you don’t want a mistake to erase any existing bookmarks. Also, unless you use a properly configured proxy server, your password will be sent in the clear. You may want to change your password to something that isn’t used by another of your other accounts.

1. Start with a basic XHTML page that includes Google-hosted copies of jQuery and jQuery UI. You can download this template at http://courses.ischool.berkeley.edu/i290-4/f09/resources/jquery_template.html

The code samples in this tutorial are broken up by explanations. For the uninterrupted code, you can download the Trailmaker at http://courses.ischool.berkeley.edu/i290-4/f09/resources/delicious_trailmaker.html

2. Create the basic structure that you will need in HTML. This includes a <form> to specify a username, a <div> and <ul> to display loaded bookmarks, and a <div> and a <ul> to create a new trail. The following structure is a good start:

<form id="load-bookmarks" method="get">
<h2 id="enter_a_delicious_username">Enter a Delicious username:</h2>
    <input type="text" name="username" id="username" value="" />
    <input type="submit" value="Get Bookmarks" />
</form>

<div id="bookmarks">
<h2 id="bookmarks">Bookmarks</h2>
    <ul></ul>
</div>

<div id="new-trail">
<h2 id="new_trail">New Trail</h2>
    <ul></ul>
    <form id="save-trail" method="post">
        Username: <input id="save-username" type="text" name="save-username" /><br />
        Password: <input id="save-password" type="password" name="save-password" /><br />
        <input type="button" name="make-trail" value="Make New Trail" id="make-trail" />                
    </form>
</div>

Loading Bookmarks

3. When a user enters an account in the #load-bookmarks form, we want to load the Delicious bookmarks for that account. Delicious provides a feed API that lets us get the bookmarks for a user at http://feeds.delicious.com/v2/json/{username}.

We want to attach all of the Javascript for this document once the page is “ready,” so we use $(document).ready() inside a <script> tag in the document <head>. You put an anonymous function inside .ready() which executes once the page is in memory:

<script>
$(document).ready(function() {
    // What you want to happen when the page loads
});
</script>

In this ready function, we’ll attach a function to the submit event of the #load-bookmarks form. When the form is submitted (by pressing the return key or by clicking on the “Get Bookmarks” button), this function will run. The last thing this function does should be return false;. The browser’s default behavior when submitting a form is to load a new page, and we use return false to prevent this action.

$('#load-bookmarks').submit(function(){
    // What you want to happen when the load bookmarks form is submitted
    return false;
});

CSS selectors are the basis for nearly all of the jQuery Javascript library. Most programming occurs by selecting some group of elements on the current page and manipulating them. Because the form has the id attribute set to ‘load-bookmarks’, we can select this element using #load-bookmarks—the pound sign (#) is used to indicate an id in CSS. Then we attach a function to the submit() event of the form. jQuery can attach functions (which is called “binding”) to many kinds of events, which are described in the jQuery documentation.

The functions that we use on inside .ready() and .submit() do not have a name—they are “anonymous functions,” which are a feature of Javascript and very common. They allow you to define commands that you want to happen inline, without defining a function somewhere else in your code. We’ll discuss anonymous functions more in lecture two; You can also read this guide to anonymous functions in Javascript.

4. When the form is submitted, we want to get the value of the account name entered in the form, which we do with var username = $('#username').val(); Although Javascript does not require var in variable declarations, you should always use it. Then we use the getJSON() method to get the bookmarks from Delicious. You can provide a function to getJSON that is called when the request is complete. In fact, if you want anything to happen when the request is returned, you have to provide a function to getJSON because of the asynchronous nature of AJAX requests. This function receives the response from the server, which you can use to construct an HTML list item and add it to the list of bookmarks.

var username = $('#username').val();
// This cross-domain request requires that you use '?callback=?' because it is done using JSONP
$.getJSON('http://feeds.delicious.com/v2/json/' + username + '?callback=?',
function(json){
    // json contains the response provided by the server
    // You can use console.log(json) to examine the response
   $(json).each(function(index) {
       // this.u // url
       // this.d // description
       // this.n // extended notes
       // this.t // array of tags

        // Create an HTML string for the bookmark
        // We also use the .data method to store the notes and tags
        // of the bookmark in the DOM with the object
        // Finally, we add it to the list of bookmarks with appendTo()
       $('<li></li>').html('<a href="' + this.u + '">' + this.d + '</a>')
        .data('extended', this.n)
        .data('tags', this.t)
        .appendTo('#bookmarks ul');
   });

   // Later, you'll add dragging functionality here

});
return false;    

Now entering a Delicious user’s account name and clicking “Load Bookmarks” works.

Creating a new trail

5.We want to let users drag loaded bookmarks to the New Trail box and rearrange them before saving the trail. Before we do this, we should make the #bookmarks box and the #new-trail box actually appear side-by-side. Add these lines to between the <style></style> tags in your document head. This makes each box roughly half the width of the screen, and the float: left makes them appear side-by-side:

#bookmarks, #new-trail {
    float: left;
    width: 48%;
    margin-right: 1%;
    min-height: 300px;
    border: 1px solid #666;
}

6. The jQuery UI library makes it easy to script complex actions like dragging and dropping. First, we want to make each of the loaded bookmarks draggable. We can do this with the .draggable() method when we load each bookmark. Add the following line at the end of your getJSON callback function:

$('#bookmarks li').draggable({revert: true});

7. Now we need a place to drop these draggable elements. We’ll make the entire #new-trail box a droppable area, and define a function that copies the bookmark to the list when the user drags a loaded bookmark to the area. Notice that the droppable method only takes one argument, a Javascript object (indicated by the curly braces {}) that provides various parameters by name. This is common practice in jQuery and jQuery UI. Here we are providing the accept and drop parameters.

$('#new-trail').droppable({
    accept: 'li', // This droppable area accepts li elements
    drop: function(event, ui) {
        // Don't confuse ul, the <ul> unordered-list with ui, the user interface element
        // .draggable('disable') says that we want to remove the draggable behavior
        $(ui.draggable).draggable('disable').appendTo('#new-trail ul');
    }
});    

We also want to make the list of bookmarks in the new trail list sortable, to allow a user to re-arrange the order of the points in the trail. To do this we can use the .sortable() function. Insert this line in your document .ready() function. Simple, right?

$('#new-trail ul').sortable();    

Saving the trail

8. Finally, we want to send each bookmark in this newly created trail to a specified user account when it is saved. We start by attaching a function to the submit of the #save-trail form. Remember to return false at the end of the function!

$('#save-trail').submit(function() {
    // Let's ask the user for a name for the trail
    // We are storing the name that the user enters as the text of the
    // h2 in the #new-trail div
    // The || syntax here lets us specify a default value
    $('#new-trail h2').text(prompt('Enter a name for your trail:') || 'My New Trail');

    // Get the username and password
    // Normally it isn't a good idea to set global variables like this
    // explicitly, but we'll do it here in the interest of brevity
    window.delicious_username = $('#save-username').val();
    window.delicious_password = $('#save-password').val();

    saveTrail();
    return false;
});    

In the saveTrail() function we want to gather the information for a single bookmark and post it to Delicious. We have to do this one list item at a time because the Delicious API only provides a call to save a single bookmark. When we have successfully saved one bookmark, we move on to the next one. When they are all done, we let the user know that saving was successful.

In order to update a bookmarks on Delicious, you need to re-save all the information from the old bookmark (including URL, description, notes, and tags) with your changes. Fortunately, we stored all this information using jQuery’s .data() method when we initially loaded the bookmarks. .data() lets you store any information you want in the DOM with an element.

function saveTrail () {
    // We need to keep track of which bookmark number we are saving, so we
    // can use the `step:2` syntax that we have established
    // Global variables aren't the best way to do this, but they're quick here
    // This line says "if window.stepNum exists, increment it by one.
    // Otherwise, set it to one"
    window.stepNum ? window.stepNum++ : window.stepNum = 1;

    // Change spaces in the trail name to underscores to follow our trail syntax
    // By default, the .replace() method doesn't replace ALL the occurrances
    // of a string, so we are using the global flag in our regular expression
    // to replace everything. The global flag is set with the "g" after
    // the regular expression (/ /g)
    var newTrailName = 'trail:' + $('#new-trail h2').text().toLowerCase().replace(/ /g, '_');

    // Get the first bookmark to save, which is the first element of the #new-trail list
    var bookmark = $('#new-trail li:first');

    // Assemble the data to send to Delicious
    var postData = {
        url: bookmark.find('a').attr('href'),
        description: bookmark.find('a').html(),
        extended: bookmark.data('extended'),
        tags: bookmark.data('tags').join(' ') + ' ' + newTrailName + ' ' + 'step:' + window.stepNum,
        method: 'posts/add',
        username: window.delicious_username,
        password: window.delicious_password
    };

Here we have constructed postData, which is a Javascript objet with keys and values for the data we are going to send to the server. The first four values are for Delicious: the url we are saving, its description (which is the title, in Delicious parlance), the extended notes for the bookmark, and the tags. Note the most important part is occurring on the line with the tags assignment: we are getting the existing tags and adding the new trail name, as well as the step number.

We stored the first line of the #new-trail list in bookmark so we could use it repeatedly. The .find() method lets us search for some potion of the document within the current selection. So .find('a').attr('href') finds the link tag within the bookmark and gets the href attribute for it.

We are also sending the method of the Delicious API that we are calling and the user’s Delicious username and password.

Next, we use jQuery’s $.post method to send the data to Delicious. You are welcome to use this code as is without inspecting it closely.

// Send the data to Delicious through a proxy and handle the response
// Use $.post if the script is located on the same server
// Otherwise, use $.get to avoid cross-domain problems
//$.post('delicious_proxy.php',
// If you are running this on your own machine, you may need to swap the $.post line for this $.getJSON line
$.getJSON('http://courses.ischool.berkeley.edu/i290-4/f09/resources/delicious_proxy.php?callback=?',
  postData,
 function(rsp){
     // POST typeof rsp == string
     // GETJSON typeof rsp == object
    if (typeof rsp == 'string') {
        // Response via POST must be eval'ed
        var rsp = eval( '(' + rsp + ')'); // The double parentheses are crucial. Omitting them will cause a parse error.
    }
        if (rsp.result_code === "access denied") {
            alert('The provided Delicious username and password are incorrect.');
        } else if (rsp.result_code === "something went wrong") {
            alert('There was an unspecified error communicating with Delicious.');
        } else if (rsp.result_code === "done") {
            // Bookmark was saved properly
            $('#new-trail li:first').remove(); // Remove the bookmark we just saved
            if ($('#new-trail li').length > 0) {
                // If there are any bookmarks left to save
                // Save the next bookmark in the trail in 1000ms (1 second)
                // We have to wait this period of time to comply with the
                // terms of the Delicious API. If we don't we may have access denied.
                setTimeout(saveTrail, 1000);
            } else {
                // We're done saving the trail
                window.delicious_password = null; // Don't store the user's password any longer than we need to
                alert ("Your trail has been saved!");
            }
        }
    });
}

You’re done with the basic tutorial! If you have any questions, ask ryan@ischool or npdoty@ischool via email, or bring them to lecture two.

Improvements

We can also add some extra niceties, like making sure links open in a new window so a person doesn’t lose his or her work in creating a new trail. Adding this code to the ready function makes all links open in a new window. (The preferred method of doing this—the target attribute—has been deprecated in XHTML.)

$('a').live('click', function() {
    window.open($(this).attr('href'));
    return false;
});    

You might want to provide some feedback while the trail is being saved. Since each bookmark has to be saved with a delay of 1 second, saving a trail could take a long time, and the user should know that something is happening.

You might disable the “Save Trail” button when the form is submitted using jQuery’s .attr() method

Also, our JavaScript does not check if a user actually entered a name for his or her trail, nor if the user entered a username or password. You could implement a check to make sure these values have been entered before saving the trail.

Additional Information

The Delicious API has a variety of methods available. To update an existing bookmark, use the posts/add method and include the replace=no parameter.

If you are going to be using the Delicious API via Javascript you should change your password to something different from your other online accounts. Because of browsers limitations to prevent cross-site scripting, you have to send your password as a URL GET request. Although this connection happens via https, your password would appear in log files. If you’re interested in why this has to be the case, ask us. Alternately, you can download the PHP proxy script yourself, run it in your own web space (change the .txt extension to .php), and POST securely to it.