Archive for September, 2009

Working around the same-origin policy in Greasemonkey

Wednesday, September 16th, 2009

The same-origin policy prevents scripts on a page from communicating with servers on a different domain from the page. Implemented by all browsers, this prevents some cross-site scripting attacks.

Of course, communicating with different web services is a common goal for mashups like the ones we’re building in this class. So if you need to communicate with a different server from the current page, you have three options:

  1. Use a server-side proxy.
  2. Make a JSONP request to a server that supports it.
  3. Use the Greasemonkey chrome to make the cross-site request for you.

Ryan’s walkthrough of the Delicious Trailmaker has code for both of the first two options, so I’ll focus on the third. Also, if you are writing a Greasemonkey script, the third option is the most flexible and straightforward.

First, let’s start with a stub of a Greasemonkey script. This script loads jQuery and inserts some simple CSS and HTML into the page to add a bar to the bottom of the page.

Now, let’s use AJAX to request something on another page. You can read extensive documentation on the jQuery AJAX commands, but for the most part, you can just call $.get() with the URL you want and a callback function for handling the XML response. For example, if you run this on http://sfbay.craigslist.org you should see the full contents of the sublets page outputted in your console.

$.get('/sub/', function(xml){
    console.log(xml);
});

But if you modify that to request the contents of delicious.com while still on craigslist.org, you’ll see nothing in the console. (You can try this yourself to convince yourself that the same-origin restriction really works.)

$.get('http://delicious.com/', function(xml){
    console.log(xml);
});

Now, to workaround this restriction, we’ll add a Javascript file which tells jQuery to use Greasemonkey to make its XmlHttpRequests. The Greasemonkey-jQuery-XHR Bridge is available on our course website.

To add it to our Greasemonkey script, we’ll add a new @require statement to the metadata block at the top of the script:

...
// @require	http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js
// @require	http://courses.ischool.berkeley.edu/i290-4/f09/resources/gm_jq_xhr.js
// ==/UserScript==

In order to get these changes, you must uninstall and re-install your script. To verify that it works, refresh that same craigslist.org page and confirm that the contents of delicious.com appear in the console.

Now that I can talk to other domains, let’s consider my ultimate goal: I want to fetch all the recent tags for the current page from Delicious. To do this, I’ll use the Delicious feeds API. That API requires that I send the MD5 hash of my URL rather than the raw URL itself, and to compute an MD5 hash, I’ll need to include an additional jQuery plugin.

There are jQuery plugins for almost everything, including MD5 hashing. This requires another @require and another reinstallation of the script.

// @require	http://plugins.jquery.com/files/jquery.md5.js.txt

Now I can produce the URL that the Delicious feeds API requires, and see on the console what kind of output I’ll receive.

var theUrl = $.md5(window.location.href);

$.get('http://feeds.delicious.com/v2/xml/url' + theUrl, function(xml){
    console.log(xml);
});
<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://web.resource.org/cc/" version="2.0">
	<channel>
		<title>Delicious/url/ad1706731c138951229c69f68c6a1c23</title>
		<link>http://delicious.com/url/ad1706731c138951229c69f68c6a1c23</link>
		<description>bookmark history for http://sfbay.craigslist.org/</description>
		<atom:link rel="self" type="application/rss+xml" href="http://feeds.delicious.com/v2/xml/url/ad1706731c138951229c69f68c6a1c23"/>
		<item>
			<title>[from icresource] craigslist - san francisco, east bay, south bay, marin county, california</title>
			<pubDate>Wed, 16 Sep 2009 17:57:38 +0000</pubDate>
			<guid isPermaLink="false">http://delicious.com/url/ad1706731c138951229c69f68c6a1c23#icresource</guid>
			<link>http://sfbay.craigslist.org/</link>
			<dc:creator><![CDATA[icresource]]></dc:creator>
			<comments>http://delicious.com/url/ad1706731c138951229c69f68c6a1c23</comments>
			<wfw:commentRss>http://feeds.delicious.com/v2/rss/url/ad1706731c138951229c69f68c6a1c23</wfw:commentRss>
			<source url="http://feeds.delicious.com/v2/rss/icresource">icresource's bookmarks</source>
			<category domain="http://delicious.com/icresource/">San_Francisco-RC</category>
			<category domain="http://delicious.com/icresource/">East_Bay-RC</category>
			<category domain="http://delicious.com/icresource/">South_Bay-RC</category>
			<category domain="http://delicious.com/icresource/">Marin_County-RC</category>
			<category domain="http://delicious.com/icresource/">events-calendar</category>
			<category domain="http://delicious.com/icresource/">jobs</category>
			<category domain="http://delicious.com/icresource/">housing</category>
			<category domain="http://delicious.com/icresource/">rideshare</category>
			<category domain="http://delicious.com/icresource/">personals</category>
			<category domain="http://delicious.com/icresource/">classifieds,</category>
			<category domain="http://delicious.com/icresource/">forums</category>
		</item>
		<item>
...

I can see from that output that it’s the <category> elements that I’m interested in. We can use the same jQuery selection tools on this XML output that we do on the Document Object Model by wrapping the xml variable in $() and running the find method. And for each <category> element that I find, I’ll add a list item to the bar.

$(xml).find("category").each(function(){
	$("#bar ul").append('<li>' + $(this).text() + '</li>');
});

(In the above code, this refers to the DOM object that we’ve selected from the XML file. By wrapping this in $() I can convert it to a jQuery object and I can use all the jQuery functions that I’m familiar with, like .text().)

At this point, you should be successfully grabbing all the recent tags for the current page from Delicious and inserting them in a bar on the page. And besides the @require lines, your code style should be no different than if you were doing same-origin requests.

Here’s the completed script. If you have any questions, problems, corrections or suggestions, please leave a comment and I’ll respond here.

Add jQuery to any (or every) webpage

Saturday, September 12th, 2009

We’ve already seen how Firebug makes it incredibly easy to inspect the current page loaded in Firefox and run jQuery commands to quickly modify items on the page or test different selectors.

But what if the webpage you’re interested in doesn’t already have jQuery installed? jQuery is becoming more and more widespread (the iSchool website has it, heck, even Craigslist, that paragon of simplicity, has it) but not all websites have it loaded. And if the webpage you visit doesn’t load jQuery then you won’t be able to use the jQuery commands from Firebug on that page.

But it’s easy to write a Greasemonkey script that will insert jQuery into any page. We can’t just use @require — that loads jQuery into the Greasemonkey script but then jQuery won’t be around when the Greasemonkey script ends and we’re trying to run our debug commands in Firebug. Instead, we’ll add a <script> element to the head of the page itself.

// ==UserScript==
// @name           Add jQuery
// @namespace      http://people.ischool.berkeley.edu/~npdoty
// @description    Insert the jQuery script so that we can run commands in Firebug
// @include        http://*
// @include        https://*
// ==/UserScript==

var GM_JQ = document.createElement('script');
GM_JQ.src = 'http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js';
GM_JQ.type = 'text/javascript';
document.getElementsByTagName('head')[0].appendChild(GM_JQ);

Easy as pie. (I’ve used plain Javascript so that we don’t have to load the whole jQuery library just to load jQuery.) With this userscript installed, you can test any jQuery command on any page you might want to investigate or modify with your own Greasemonkey script.

More details about this technique from Joan Piedra (from whom I’ve adapted our code).

Using separate files for CSS and JS

Wednesday, September 9th, 2009

One of your classmates rightly noted that Nick and I have said that you can (and often should) keep your HTML, CSS, and JavaScript in separate files, but have not provided any instructions about how to do that.

Single File

Using a single file for HTML, CSS, and JS

Using a single file for HTML, CSS, and JS

The single-file approach is good for quick one-off tests or demonstrations. This is where you put all your CSS and JavaScript in the same file as your HTML. Your single .html document will look something like this:

<html>
<head>
    <style type="text/css">
    /* CSS goes here in the style tag. */
    </style>

    <script type="text/javascript">
    // JavaScript goes here in the script tag.
    </script>
</head>
<body>
    HTML content goes here in the body tag.
</body>
</html>

Although it is fast to get started and there are fewer files to keep track of, there are some significant downsides to keeping everything in one HTML file. For example, if you put your CSS and JavaScript in the same file with your HTML, you can’t reuse the same styles and code on other pages without needless duplication.

Multiple Files

Multiple Files for HTML, CSS, and JS

Multiple Files for HTML, CSS, and JS

The alternative is to store your HTML, CSS, and JavaScript in separate files. In the main .html file you use a <link> tag to include the external stylesheet and a <script> tag to include the external JavaScript. Now your .html document will look like this:

<html>
<head>
    <link rel="stylesheet" href="file.css" type="text/css" media="screen" />
    <script type="text/javascript" src="file.js"></script>
</head>
<body>
    HTML content goes here in the body tag.
</body>
</html>

Note that the CSS file is specified by href="..." while the JavaScript file is specified by src="..."

Delicious Trailmaker Tutorial

Tuesday, September 1st, 2009

In our first class on September 1, we did an in-class demo showing one possible way to implement trails from Vannevar Bush’s Memex using the Delicious API. This tutorial reviews the steps from class. At the end of this tutorial, you will have constructed a small web application that saves a collection of bookmarks to Delicious as a trail. A trail is identified by a set of special tags: trail:[name_of_trail] and step:[step_number].

Getting Started

0. Before getting started, you may want to create a separate Delicious account for experimenting. The Delicious API has no undo, and you don’t want a mistake to erase any existing bookmarks. Also, unless you use a properly configured proxy server, your password will be sent in the clear. You may want to change your password to something that isn’t used by another of your other accounts.

1. Start with a basic XHTML page that includes Google-hosted copies of jQuery and jQuery UI. You can download this template at http://courses.ischool.berkeley.edu/i290-4/f09/resources/jquery_template.html

The code samples in this tutorial are broken up by explanations. For the uninterrupted code, you can download the Trailmaker at http://courses.ischool.berkeley.edu/i290-4/f09/resources/delicious_trailmaker.html

2. Create the basic structure that you will need in HTML. This includes a <form> to specify a username, a <div> and <ul> to display loaded bookmarks, and a <div> and a <ul> to create a new trail. The following structure is a good start:

<form id="load-bookmarks" method="get">
<h2 id="enter_a_delicious_username">Enter a Delicious username:</h2>
    <input type="text" name="username" id="username" value="" />
    <input type="submit" value="Get Bookmarks" />
</form>

<div id="bookmarks">
<h2 id="bookmarks">Bookmarks</h2>
    <ul></ul>
</div>

<div id="new-trail">
<h2 id="new_trail">New Trail</h2>
    <ul></ul>
    <form id="save-trail" method="post">
        Username: <input id="save-username" type="text" name="save-username" /><br />
        Password: <input id="save-password" type="password" name="save-password" /><br />
        <input type="button" name="make-trail" value="Make New Trail" id="make-trail" />                
    </form>
</div>

Loading Bookmarks

3. When a user enters an account in the #load-bookmarks form, we want to load the Delicious bookmarks for that account. Delicious provides a feed API that lets us get the bookmarks for a user at http://feeds.delicious.com/v2/json/{username}.

We want to attach all of the Javascript for this document once the page is “ready,” so we use $(document).ready() inside a <script> tag in the document <head>. You put an anonymous function inside .ready() which executes once the page is in memory:

<script>
$(document).ready(function() {
    // What you want to happen when the page loads
});
</script>

In this ready function, we’ll attach a function to the submit event of the #load-bookmarks form. When the form is submitted (by pressing the return key or by clicking on the “Get Bookmarks” button), this function will run. The last thing this function does should be return false;. The browser’s default behavior when submitting a form is to load a new page, and we use return false to prevent this action.

$('#load-bookmarks').submit(function(){
    // What you want to happen when the load bookmarks form is submitted
    return false;
});

CSS selectors are the basis for nearly all of the jQuery Javascript library. Most programming occurs by selecting some group of elements on the current page and manipulating them. Because the form has the id attribute set to ‘load-bookmarks’, we can select this element using #load-bookmarks—the pound sign (#) is used to indicate an id in CSS. Then we attach a function to the submit() event of the form. jQuery can attach functions (which is called “binding”) to many kinds of events, which are described in the jQuery documentation.

The functions that we use on inside .ready() and .submit() do not have a name—they are “anonymous functions,” which are a feature of Javascript and very common. They allow you to define commands that you want to happen inline, without defining a function somewhere else in your code. We’ll discuss anonymous functions more in lecture two; You can also read this guide to anonymous functions in Javascript.

4. When the form is submitted, we want to get the value of the account name entered in the form, which we do with var username = $('#username').val(); Although Javascript does not require var in variable declarations, you should always use it. Then we use the getJSON() method to get the bookmarks from Delicious. You can provide a function to getJSON that is called when the request is complete. In fact, if you want anything to happen when the request is returned, you have to provide a function to getJSON because of the asynchronous nature of AJAX requests. This function receives the response from the server, which you can use to construct an HTML list item and add it to the list of bookmarks.

var username = $('#username').val();
// This cross-domain request requires that you use '?callback=?' because it is done using JSONP
$.getJSON('http://feeds.delicious.com/v2/json/' + username + '?callback=?',
function(json){
    // json contains the response provided by the server
    // You can use console.log(json) to examine the response
   $(json).each(function(index) {
       // this.u // url
       // this.d // description
       // this.n // extended notes
       // this.t // array of tags

        // Create an HTML string for the bookmark
        // We also use the .data method to store the notes and tags
        // of the bookmark in the DOM with the object
        // Finally, we add it to the list of bookmarks with appendTo()
       $('<li></li>').html('<a href="' + this.u + '">' + this.d + '</a>')
        .data('extended', this.n)
        .data('tags', this.t)
        .appendTo('#bookmarks ul');
   });

   // Later, you'll add dragging functionality here

});
return false;    

Now entering a Delicious user’s account name and clicking “Load Bookmarks” works.

Creating a new trail

5.We want to let users drag loaded bookmarks to the New Trail box and rearrange them before saving the trail. Before we do this, we should make the #bookmarks box and the #new-trail box actually appear side-by-side. Add these lines to between the <style></style> tags in your document head. This makes each box roughly half the width of the screen, and the float: left makes them appear side-by-side:

#bookmarks, #new-trail {
    float: left;
    width: 48%;
    margin-right: 1%;
    min-height: 300px;
    border: 1px solid #666;
}

6. The jQuery UI library makes it easy to script complex actions like dragging and dropping. First, we want to make each of the loaded bookmarks draggable. We can do this with the .draggable() method when we load each bookmark. Add the following line at the end of your getJSON callback function:

$('#bookmarks li').draggable({revert: true});

7. Now we need a place to drop these draggable elements. We’ll make the entire #new-trail box a droppable area, and define a function that copies the bookmark to the list when the user drags a loaded bookmark to the area. Notice that the droppable method only takes one argument, a Javascript object (indicated by the curly braces {}) that provides various parameters by name. This is common practice in jQuery and jQuery UI. Here we are providing the accept and drop parameters.

$('#new-trail').droppable({
    accept: 'li', // This droppable area accepts li elements
    drop: function(event, ui) {
        // Don't confuse ul, the <ul> unordered-list with ui, the user interface element
        // .draggable('disable') says that we want to remove the draggable behavior
        $(ui.draggable).draggable('disable').appendTo('#new-trail ul');
    }
});    

We also want to make the list of bookmarks in the new trail list sortable, to allow a user to re-arrange the order of the points in the trail. To do this we can use the .sortable() function. Insert this line in your document .ready() function. Simple, right?

$('#new-trail ul').sortable();    

Saving the trail

8. Finally, we want to send each bookmark in this newly created trail to a specified user account when it is saved. We start by attaching a function to the submit of the #save-trail form. Remember to return false at the end of the function!

$('#save-trail').submit(function() {
    // Let's ask the user for a name for the trail
    // We are storing the name that the user enters as the text of the
    // h2 in the #new-trail div
    // The || syntax here lets us specify a default value
    $('#new-trail h2').text(prompt('Enter a name for your trail:') || 'My New Trail');

    // Get the username and password
    // Normally it isn't a good idea to set global variables like this
    // explicitly, but we'll do it here in the interest of brevity
    window.delicious_username = $('#save-username').val();
    window.delicious_password = $('#save-password').val();

    saveTrail();
    return false;
});    

In the saveTrail() function we want to gather the information for a single bookmark and post it to Delicious. We have to do this one list item at a time because the Delicious API only provides a call to save a single bookmark. When we have successfully saved one bookmark, we move on to the next one. When they are all done, we let the user know that saving was successful.

In order to update a bookmarks on Delicious, you need to re-save all the information from the old bookmark (including URL, description, notes, and tags) with your changes. Fortunately, we stored all this information using jQuery’s .data() method when we initially loaded the bookmarks. .data() lets you store any information you want in the DOM with an element.

function saveTrail () {
    // We need to keep track of which bookmark number we are saving, so we
    // can use the `step:2` syntax that we have established
    // Global variables aren't the best way to do this, but they're quick here
    // This line says "if window.stepNum exists, increment it by one.
    // Otherwise, set it to one"
    window.stepNum ? window.stepNum++ : window.stepNum = 1;

    // Change spaces in the trail name to underscores to follow our trail syntax
    // By default, the .replace() method doesn't replace ALL the occurrances
    // of a string, so we are using the global flag in our regular expression
    // to replace everything. The global flag is set with the "g" after
    // the regular expression (/ /g)
    var newTrailName = 'trail:' + $('#new-trail h2').text().toLowerCase().replace(/ /g, '_');

    // Get the first bookmark to save, which is the first element of the #new-trail list
    var bookmark = $('#new-trail li:first');

    // Assemble the data to send to Delicious
    var postData = {
        url: bookmark.find('a').attr('href'),
        description: bookmark.find('a').html(),
        extended: bookmark.data('extended'),
        tags: bookmark.data('tags').join(' ') + ' ' + newTrailName + ' ' + 'step:' + window.stepNum,
        method: 'posts/add',
        username: window.delicious_username,
        password: window.delicious_password
    };

Here we have constructed postData, which is a Javascript objet with keys and values for the data we are going to send to the server. The first four values are for Delicious: the url we are saving, its description (which is the title, in Delicious parlance), the extended notes for the bookmark, and the tags. Note the most important part is occurring on the line with the tags assignment: we are getting the existing tags and adding the new trail name, as well as the step number.

We stored the first line of the #new-trail list in bookmark so we could use it repeatedly. The .find() method lets us search for some potion of the document within the current selection. So .find('a').attr('href') finds the link tag within the bookmark and gets the href attribute for it.

We are also sending the method of the Delicious API that we are calling and the user’s Delicious username and password.

Next, we use jQuery’s $.post method to send the data to Delicious. You are welcome to use this code as is without inspecting it closely.

// Send the data to Delicious through a proxy and handle the response
// Use $.post if the script is located on the same server
// Otherwise, use $.get to avoid cross-domain problems
//$.post('delicious_proxy.php',
// If you are running this on your own machine, you may need to swap the $.post line for this $.getJSON line
$.getJSON('http://courses.ischool.berkeley.edu/i290-4/f09/resources/delicious_proxy.php?callback=?',
  postData,
 function(rsp){
     // POST typeof rsp == string
     // GETJSON typeof rsp == object
    if (typeof rsp == 'string') {
        // Response via POST must be eval'ed
        var rsp = eval( '(' + rsp + ')'); // The double parentheses are crucial. Omitting them will cause a parse error.
    }
        if (rsp.result_code === "access denied") {
            alert('The provided Delicious username and password are incorrect.');
        } else if (rsp.result_code === "something went wrong") {
            alert('There was an unspecified error communicating with Delicious.');
        } else if (rsp.result_code === "done") {
            // Bookmark was saved properly
            $('#new-trail li:first').remove(); // Remove the bookmark we just saved
            if ($('#new-trail li').length > 0) {
                // If there are any bookmarks left to save
                // Save the next bookmark in the trail in 1000ms (1 second)
                // We have to wait this period of time to comply with the
                // terms of the Delicious API. If we don't we may have access denied.
                setTimeout(saveTrail, 1000);
            } else {
                // We're done saving the trail
                window.delicious_password = null; // Don't store the user's password any longer than we need to
                alert ("Your trail has been saved!");
            }
        }
    });
}

You’re done with the basic tutorial! If you have any questions, ask ryan@ischool or npdoty@ischool via email, or bring them to lecture two.

Improvements

We can also add some extra niceties, like making sure links open in a new window so a person doesn’t lose his or her work in creating a new trail. Adding this code to the ready function makes all links open in a new window. (The preferred method of doing this—the target attribute—has been deprecated in XHTML.)

$('a').live('click', function() {
    window.open($(this).attr('href'));
    return false;
});    

You might want to provide some feedback while the trail is being saved. Since each bookmark has to be saved with a delay of 1 second, saving a trail could take a long time, and the user should know that something is happening.

You might disable the “Save Trail” button when the form is submitted using jQuery’s .attr() method

Also, our JavaScript does not check if a user actually entered a name for his or her trail, nor if the user entered a username or password. You could implement a check to make sure these values have been entered before saving the trail.

Additional Information

The Delicious API has a variety of methods available. To update an existing bookmark, use the posts/add method and include the replace=no parameter.

If you are going to be using the Delicious API via Javascript you should change your password to something different from your other online accounts. Because of browsers limitations to prevent cross-site scripting, you have to send your password as a URL GET request. Although this connection happens via https, your password would appear in log files. If you’re interested in why this has to be the case, ask us. Alternately, you can download the PHP proxy script yourself, run it in your own web space (change the .txt extension to .php), and POST securely to it.

Hello world!

Tuesday, September 1st, 2009

Welcome to Information Organization Lab for the fall semester 2009. This is the blog for this course where students can exchange advise and experiences with other students in the class.