Preloading Data with Ajax and JSON

By Ryan Campbell · Jan 24, 2006

Introduction

Web applications have made huge leaps and bounds in improving user experience thanks to a lot of recently developed fancy-pants technology. In an effort to help you take it up a notch, we’d like to share a method for helping your site anticipate a user’s next move.

On most sites, there are usually only a handful of options that can be done at any given moment and often some of these options are more likely than others. By determining what is important on each page, we can preload the data of the user’s next action and store it on the client with JSON. When the user decides to perform their next action, they’ll see their results almost instantly because the info was loaded in the background.



Advertise on Particletree

See It In Action

As always, we’ve created a little demonstration for you to observe, illustrate, and dissect. It’s a paging demo that shows how we can preload in the background the next and previous queries from a database. When the user pages through, they can do so quickly without having to wait for an Ajax call to finish.





Download the code 

Also note that this demo and the tutorial that follows use the Prototype.js framework along with our own custom database class. You’ll find both in the zip file above.

JSON

Before we get started on the paging, let’s take a quick look at JavaScript Object Notation, or JSON. JSON is a data interchange format, similar to XML, which is built using JavaScript. The easiest way to get your head around it is to imagine the format as a variable containing multiple arrays that can be nested endlessly. Implementing JSON in JavaScript is simple and the official documentation actually does a great job illustrating lots of examples. For this tutorial, we will be returning a string of JSON from the server, and that string will contain our preloaded data. The only tricky part is that once the data is fetched, we’ll have to convert it into a JavaScript object after the client receives it. To do that, we’ll use

eval()

.

currentPage = eval('(' + response.responseText + ')');

The JavaScript line above makes use of the eval function, which converts the properly formatted string from the server into a JavaScript object. Once that is ready to go, the rest is easy. Now, if you’re not comfortable using JSON, you can always use XML instead. To compare the two, check out PPK’s review. That man never stops doing his homework.

Determining What to Preload

Once we know how to store our data, we’ll want to determine what kind of data we’d like to store. As far as I am aware, any amount of data can be stored on the client side. It is all dependent on the client computer (however, do correct me if I am wrong). Chances are, we don’t wish to crash the client, so the choice of and amount of data becomes important. For example, an online store with hundreds of thousands of products may not want to load every single product into client memory. Likewise, loading millions of records of data that need to be paged is also not ideal, so for this example we just set up a simple next and previous page to always be preloaded.

Getting Started

To set up a paging system, we need to know a few things: the total amount of records, the current page, the next page, the previous page, and the paging size. Starting from the initial page load, here is what needs to happen:

Grab the total amount of records so we know how many total pages there might be.
Grab the first page of data and display it to the user.
Grab the next page of data and store it as JSON on the client.
No previous page of data is needed, since record 0 is always loaded first. Let’s take a look at each step individually.

Total Records

To get the total amount of records possible, we need to create a basic query that returns the value to the client.

function getRecordCount() {
$db     = new database();
$sql    = "SELECT COUNT(*) AS recordCount FROM Accounts";
$rs     = $db->query($sql);
$row    = mysql_fetch_array($rs, MYSQL_ASSOC);
echo    $row['recordCount'];
}

This function simply gets the record count and prints the value. The following Ajax call retrieves the value:

function getRecordCount() {
var myAjax = new Ajax.Request(
'flow.php?action=count',
{
method: 'get',
parameters: '',
onComplete: function(response) {
recordCount = response.responseText;
}
});
}

The global variable,

recordCount

, is now initialized. Also, note that the parameter,

action

. in the querystring, is used to tell the server what action to perform. In this case, it told the server to return the record count.

Get Initial Data

Once the user first hits the page, we need to show them the first page of data. To do that, we have to do a few things. First, let’s look at the function that will draw out the HTML that the user sees.

function drawTable(page, contain) {
table =     '<table>';
alt = '';
for(i = 0; i < page['players'].length; i++) {
table +=         '<tr class="'+alt+'">' +
'<td>' + page['players'].lastName   + ',</td>' +
'<td>' + page['players'][i].firstName  + '</td>' +
'<td>' + page['players'][i].position   + '</td>' +
'</tr>';
(alt == '')
?   alt = 'alt'
:   alt = '';
}

table += '</table>';
contain.innerHTML = table;
}

This function creates a table, and inserts it as the innerHTML of an element with the ID of

container

. You’ll notice when the table is drawing, it is looping through an object named page. The next step is to take a look at the page object. The page object is a string of data that the server returned to the client, and the client then converted into a JSON object. On the server side, the data is retrieved and displayed like this:

function getTableData() {
$ret = '{"players" :[';
$db     = new database();
$sql    = "SELECT * FROM Accounts LIMIT ".$_GET['current'].", ".$_GET['size'];
$rs     = $db->query($sql);

while($row  = mysql_fetch_array($rs, MYSQL_ASSOC)) {
$ret .= '{ "firstName" : "'.$row['FirstName'].'", "lastName" : "'.$row['LastName'].'", "position"  : "'.$row['Position'].'" }, ';
}

$ret = rtrim($ret, ', ').']}';
echo $ret;
}

Then, using the

eval()

function mentioned earlier:

currentPage = eval('(' + response.responseText + ')');

We convert the server response into a JavaScript object that looks like this:

{ "players" : [
{ "firstName" : "Ryan", "lastName" : "Campbell", "position" : "S" },
{ "firstName" : "Chris", "lastName" : "Campbell", "position" : "QB" },
{ "firstName" : "Kevin", "lastName" : "Hale", "position" : "DT" }
]}

Once the object is created, the

drawTable()

function just loops through all players and displays their information. So now the user is viewing the initial data, and we wish to preload the next set of data. To do this, we only have to recreate what we just did, and not call the

drawTable()

function. Also, we need to store the response in a variable called

nextPage

instead of

currentPage

.

Paging

At this point, we have the user viewing the current page of data, and that data is stored in a JSON object named

currentPage

. We also have the next page of data stored in a JSON object named

nextPage

. So, when the user clicks the “next page” button, we want to do a few things:

function getNextPage() {
currentRecord += pagingSize;

Increase the current record by the paging size. If they were viewing records 0 - 20, and they hit “next page,” the current record will be increased by 20, making the next set 20 - 40.

showNavigation();

Explained below, this function hides and shows the previous and next buttons as necessary.

previousPage = currentPage;

We know the previous page will become the current page, since the current page is advancing one. No need to hit the server to get previous data—just change the variables.

currentPage = nextPage;

nextPage

was preloaded into a JSON object, so we can now set the current page to the next page.

drawTable(currentPage, $('view'));

Draw the new

currentPage

to the screen.

getNextData();
}

Preload the new

nextPage

. This will work until the last page is hit. When a user clicks “previous page,” the same thing happens in reverse order.

Controlling the Navigation

The last thing to do is hide the “next page” button if the user is on the last page, and hide the “previous page” button if they are on the first. The following function should do the trick:

function showNavigation() {
(currentRecord == 0)
?   $('previousLink').style.visibility = 'hidden'
:   $('previousLink').style.visibility = 'visible';
((currentRecord + pagingSize) >= recordCount)
?   $('nextLink').style.visibility = 'hidden'
:   $('nextLink').style.visibility = 'visible';
}

Basically, this function just changes the style based on the

currentRecord

. If we’re at 0, then we’re on the first page. If the currentRecord equals the pagingSize (or total number of pages), then we’re on the last. Hide and show as deemed fit.

Improvements and Possibilities

Lately, I have found JavaScript to be fairly stable except when a user spams an action. The same problem applies here. If a user spams the “next page” button, the code may get thrown off. In order to prevent this problem, I sometimes find it necessary to create global variables, such as

isActive

, that prevents anything from happening until the variable is

false

. Other than that, everything should work out fine when preloading data.

I also mentioned earlier on that a store could use this technique. Right now, Ajax is fast, but it’s not instant since it still has to hit the server. By storing the current page of cart items on the client, you could add to the cart instantly and the user would have no wait while browsing through products. The store example and the example I used here both emphasize heavy sets of data, but it’s important to also realize that this can be used for trivial things like preparing the next set of HTML the user will see, or allowing for mass saving at the end of multiple actions rather than saving each individual action as it happens.

As always, we’d love to hear about some more possible uses of this technique, and suggestions for improving it are always welcome.


Preloading Data with Ajax and JSON
Author : Ryan Campbell
Last Modified : January 24, 2006 02:29 PM
Filed Under : AJAX · Features · Ideas · JSON · Javascript
Navigate : « Previous Entry · Archives · Next Entry »

Around The Web

See what other sites are saying about Preloading Data with Ajax and JSON (14)

Particletree

Home
About
Archives
Links

Subscribe

Particletree


Features


Notebook


Comments

Recent Features

Replacing Trackback with Blog Search
Lightbox Gone Wild!
Preloading Data with Ajax and JSON
Using Flash as an Animation Underlayer
A Guide to Starting Your Business

Recent Notebook

Prototype Bind Revisited
How to Make Firefox Forms Suck Less
Treehouse - March 2006
Introducing Wufoo
Event Handlers on Disabled Inputs
More Love For JSON
Treehouse - February 2006

Recent Comments

Marko · Introducing Wufoo
Lars Olesen · Lightbox Gone Wild!
Robert · An Argument for Small Business Blogging
Jasper Jasper · Prototype Bind Revisited
Francesco~ · How to Make Firefox Forms Suck Less

Comments

Very nice application Ryan. It ist pretty fast and ideal to compare datalists. Greetings from Berlin, Germany!

Posted by Heiko · January 26, 2006 03:52 AM Perhaps rather than isActive the variable, each request could return the ‘page’ it was sent for. That way the buttons can keep functioning, users can jump back pages, but the JavaScript discards returned objects for a page that the site isn’t currently on? Otherwise it’s more like SJAX than AJAX. Well, SJAJO in this instance.

Posted by Relapse · January 26, 2006 04:05 AM Great stuff, it could easily be adapted so that it could be unobtrusive and only added to the page if the user’s browser is Ajax Comptable.

Only thing I can see missing is a buffer - why not save some of the data gained from the database into an object? Then if the user browses back and forth there will be fewer database queries.

Posted by Ross · January 26, 2006 04:57 AM Great tutorial!

We tested JSON when we implemented new AJAX parts in our web application but went back to pure XML due to portability reasons.

Also, it can be dangerous to second guess the user. If there are more than 1 possible actions a user can execute, you could spend time preloading data that will never be viewed.

With this would come obvious scalability problems :-)

Posted by David Kaspar · January 26, 2006 07:02 AM David: Good advice.

In this case, however, the cost is not very high, incurring one extra database hit for each time the user picks a direction by which to navigate the database (up or down).

Relapse: SJAJO? You’ve just pushed me over the edge to being a proponent of naming the concept rather than the technology! ;)

Posted by Steve · January 26, 2006 11:59 AM Sweet but i prefer XML :)

Posted by theCreator · January 26, 2006 12:06 PM Ross - right now the buffer only holds 1 additional page of data. Depending on how much data each buffer contains, there could be more. The only thing to wathc out for is holding too much in buffers and potential crashing the browser.

David - I think each case will be different, and the programmer just has to realize the bandwidth problems. A site that is heavy on Ajax should have significant savings as is, and a technique like this may not cause too much of a problem compared to the bandwidth used by a traditional, non ajax site.

Posted by Ryan Campbell · January 26, 2006 03:56 PM Rather than using “active” to cause the getNext/Previous to do nothing, you could implement fallback to non-AJAX behaviours. Have the next/previous controls make a boring old HTTP request to the server to get a page of data, starting with record X, along with Javascript overrides. If the next/previous page is in local cache, replace the data as usual and cancel the event. If not, let the event continue, and the server round-trip will slow the user’s spamming.

Also it’s probably worth noting that asking the server for twice as much data as you’re going to need is probably still a bandwidth saving over asking the server for exactly the data you need, but also all the HTML to present it, as you would get in a non-AJAX environment, depending on just how much data you’re displaying on a single page. And in my view, AJAX is about enhancing the user experience, not about saving on server costs. :)

Posted by Byron Ellacott · January 26, 2006 08:37 PM [i]Web applications have made huge leaps and bounds in improving user experience thanks to a lot of recently developed fancy-pants technology.

application is cool but I doubt about user experience and abaut that what you call “improving”. Everything is cool with those ajax apps untill user starts to use next and back buttons on its default UI (like brauser, mouse buttons aso.)

so the next huge leap could be the ajax integration with “next” and “previous” buttons of users default UI not within the UI in default UI

Posted by ei kommentaari · January 29, 2006 07:19 AM Ryan, very interesting article. JSON could really make things faster.

But you don’t want the count query at the beginning. This may be quite slow, for large tables. Or not?

You can use other techniqes to know wether to show/hide next/prev links.

For example: You can ask for one extra record in the queries. If you are quering next x records, on the server you will query for (x + 1) records. If you really receive x+ 1 records, then the next link should be visable. If not, you now you reached the end. You can then pass it as a parameter along with the other data.

Posted by Stoicho · January 30, 2006 05:53 AM Stoicho, it should be a relatively fast query, but if you were to run into problems with it then the method you suggested would work out better. The one thing I like more about your suggestion is that the user doesn’t have to wait for an additional query on page load. On the other hand, the server will be hit more looking for that last query. Good suggestion that is definitely worth looking into.

Posted by Ryan Campbell · January 30, 2006 02:14 PM What about predicting where a user will click and preloading that page or link/data? do you know if there is something like that out there? maybe we can detect where the mouse is on the screen and preload any links in that area of the screen?

Posted by Kim · January 31, 2006 03:12 PM Thanks Ryan, this is great information, you saved me hours of work trying to solve this very issue! Although I love JSON and it’s certainly here to stay with the wide adoption by Yahoo and other companies, does anyone else feel it’s odd that XML was created specifically for this kind of purpose and yet data structures such as JSON are being used? Don’t get me wrong, I use JSON myself and understand it’s necessity regarding speed and CDN JavaScript, it just seems like a shame that XML isn’t always the best fit under certain conditions.

Posted by Shaun Shull · January 31, 2006 04:46 PM Great article, and idea, and nicely executed. Exactly what the database engines do (oracle,sql server) to optimize paging, pre-fetching. not to mention its been built into hard disk drivers as read ahead buffers for decades.

How come all this stuff seems so simple once someone executes it.

Posted by Hubris Sonic · January 31, 2006 08:28 PM Great article, great information and very well written.

Thank you for the information.

A happy subscriber of Treehouse.

Posted by Jorge Alvarez · February 1, 2006 07:34 AM Um, how does a user “spam” a button? :-)

Also, this code can be simplified:

function showNavigation() {
(currentRecord == 0)
?   $('previousLink').style.visibility = 'hidden'
:   $('previousLink').style.visibility = 'visible';
((currentRecord + pagingSize) >= recordCount)
?   $('nextLink').style.visibility = 'hidden'
:   $('nextLink').style.visibility = 'visible';
}

to:

function showNavigation() {
$('previousLink').style.visibility =
currentRecord == 0 ? 'hidden' : 'visible';
$('nextLink').style.visibility =
currentRecord + pagingSize) >= recordCount ? 'hidden' : 'visible';
}

Which in turn suggests a further improvement:

function showNavigation() {
showIf( 'previousLink', currentRecord == 0 );
showIf( 'nextLink', currentRecord + pagingSize >= recordCount );
}

function showIf( id, show ) {
$(id).style.visibility = ( show ? 'visible' : 'hidden' );
}

Posted by Michael Geary · February 1, 2006 12:10 PM Ack, where is the Edit button?! :-)

That last version of the code should read:

function showNavigation() { showIf( ‘previousLink’, currentRecord > 0 ); showIf( ‘nextLink’, currentRecord + pagingSize < recordCount ); }

function showIf( id, show ) { $(id).style.visibility = ( show ? ‘visible’ : ‘hidden’ ); }

Posted by Michael Geary · February 1, 2006 12:12 PM Try that again… No preview… No edit… Embarrassed commenter…

function showNavigation() {
showIf( 'previousLink', currentRecord == 0 );
showIf( 'nextLink', currentRecord + pagingSize >= recordCount );
}

function showIf( id, show ) {
$(id).style.visibility = ( show ? 'visible' : 'hidden' );
}

Posted by Michael Geary · February 1, 2006 12:14 PM I give up. Did I mention I was looking for a Preview or Edit button? ;-)

This is the correct code:

function showNavigation() {
showIf( 'previousLink', currentRecord > 0 );
showIf( 'nextLink', currentRecord + pagingSize

Posted by Michael Geary · February 1, 2006 12:17 PM Right. No less-than sign. Let’s try the good old ampersand-lt-semicolon and see if that helps:

function showNavigation() {
showIf( 'previousLink', currentRecord > 0 );
showIf( 'nextLink', currentRecord + pagingSize < recordCount );
}

function showIf( id, show ) {
$(id).style.visibility = ( show ? 'visible' : 'hidden' );
}

Posted by Michael Geary · February 1, 2006 12:19 PM Thomas Edison was right: It’s 10% inspiration and 90% perspiration!

Posted by Michael Geary · February 1, 2006 12:23 PM That was entertaining for me to watch. The dialogue really added to it :) On a positive note, I like your improvement to the code — thanks for taking the time to get the post working.

Posted by Ryan Campbell · February 1, 2006 12:43 PM The demo will only page once, after that clicking the links doesn’t do anything. I am not hitting the button again until the next page information has been loaded, so I don’t think that this is the ‘spam action’ problem.

I am OS: XP SP2 Browser: FF 1.5

Posted by Wesley Walser · February 1, 2006 12:48 PM Nice tutorial. Been working w/ Prototype, AJAX and JSON for a whole day today.

Don’t know what version of prototype you are using, but there’s built-in JSON support in mine (1.4).

First: If your server-side script responds with X-JSON: header, JSON object is created automatically and is passed as a second argument to onComplete().

I used the following code:

require('Services/JSON.php'); $json = new Services_JSON(); $jsonString = $json->encode($someVarOrArray); // doesn't work if you don't specify brackets header('X-JSON:('.$jsonString.')');

(Services_JSON is a PEAR package, available here - http://pear.php.net/pepr/pepr-proposal-show.php?id=198, simplifies JSON object creation)

Same thing happens if your server returns Content-type: text/javascript, but in this case response body will be eval()-ed.

First method is much more interesting, as it allows you to return content AND JSON object.

Posted by German Rumm · February 1, 2006 12:51 PM Having the ability to return content along with an object is interesting. I’ll have to play around with it. Thanks for pointing that out.

Posted by Ryan Campbell · February 1, 2006 01:30 PM Oh, that’s good, Ryan, I’m glad it was entertaining even if I did feel like an idiot. :-)

It was funny timing, because I’d just been working on some JSONP preloading code for my own scrollable list this morning when I ran across your article. So it was interesting and enlightening to compare approaches. Thanks!

Posted by Michael Geary · February 1, 2006 02:07 PM Thanks, if it is possible, could someone share a link about paging in Ajax basicly without the libraries.

Posted by Bruker · February 7, 2006 10:09 AM This would also be great as a calendar switching application.

It would be easy to “hide” the next and previous panes to also provide the perception of speed. Very nice application.

Posted by Deco Rior · February 8, 2006 01:23 AM Good work!

Preloaded next and previous page is really great idea. Nice url, using PHP heh :)

Syntatic sugar - html code nicer when use http://bennolan.com/behaviour/

Posted by helmi03 · February 8, 2006 07:42 PM Ryan,

Nice tutorial and really good work. I helps me so much understanding ajax. I refined the web page and the paging script of your demo to allow adding record. Here is what I did: After the record is added, the callback function will refresh the view page to reflect the latest snap shot (another ajax request to complish this). However, the later ajax request for the table data returns old recordset rather than the new recordset. It is the same even though I page next and previous in an attempt to retrieve the new data from the database. The new record is not shown until I close and open a new browser session. It seems there is some caching going on but I don’t exactly sure about that.

Posted by Anthony Siu · February 10, 2006 03:37 PM Hi, Very good example. Here’s one thing i would like to do with Ajax, it is the preloading of one yes one jpeg (not multiple jpeg) with a progress bar, a true progress bar that count the data bytes per bytes. Please let me know.

There are a lot a progress bar for multiple preloading jpeg or gif files mostly small but a lot of them, it is not the same thing here.

Thank you very much!

Posted by x-mob · February 12, 2006 04:27 AM A sneaky bug waiting to pounce.

Here’s perhaps a reason you’ve found some of your script unpredictable when a user “spams” it.

I just learned this recently myself, so no high-horsey antics here. Just good old-fashioned info:

You know that cute little “var” keyword you’ve seen sometimes in JavaScript? If you’re like me you might’ve thought it was some nasty alias for “variant”. Which would be bad right? We don’t need no stinkin’ variants!

Except that’s not what it means. “var” is actually the variable declaration keyword in JavaScript. So up there in those functions where you’re declaring variables? I bet you think those are scoped to the functions huh? Think again! They’re actually global variables!

In JavaScript you must declare a variable with the “var” keyword and within the scope of a function or block (like if, for, etc) to have it scoped to that block.

An example is worth a thousand words:

function sayItTenTimes(message) { for(i = 0; i < 10; i++) { document.write('' + message + ''); } }

You’d probably expect this to repeat the numbers 0 through 9 ten times each on a page. You might be surprised what actually happens if you paste it into a test page however…

This version has the output you probably expected of the first however:

function sayItTenTimes(message) { for(var i = 0; i < 10; i++) { document.write('' + message + ''); } }

So… yeah.

And Wufoo looks very nice. :-)

Posted by Sam · February 25, 2006 01:25 AM

for(var i = 0; i < 10; i++) {
sayItTenTimes(i);
}

for(i = 0; i < 10; i++) {
sayItTenTimes(i);
}

Hi Ryan,

Great tutorial… I’m slowing learning this type of development and tutorials like this really help.

A few questions… 1. I’m trying to get this working using php4 and a slightly older mysql db. (I can’t change the version used, hosted solution.) I’ve added some logging and I can see the flow.php->getTableData function printing results. However, the tables are always empty!

Is there a way to debug the AJAX calls ? i.e. I can not seem to figure out if the drawTable function ever gets called.

In the getTableData js function, where is the syntax $(‘view’) coming from ? Is it js syntax or perhaps prototype ?
Thanks…

Posted by Ben · March 6, 2006 08:20 AM Ben,

I suggest downloading Firebug. It is a free Firefox plugin that can track Ajax requests and responses among other things. That should help with the debugging problems you’re having.

And yes, the $() function comes from prototype, and replaces document.getElementById.

Posted by Ryan Campbell · March 6, 2006 01:53 PM