Stanford University Wiki
Don't have an account?
Register
Advertisement
Register

Project 1: Gmail

Edit

Seng Keat Teh, Julien Wetterwald

Introduction[]

At the beginning of each session, a page containing an empty document as well as several JavaScript files embedding Gmail's behavior is loaded into the browser. Gmail's UI is constructed in the initially empty document by the browser. All subsequent interaction between the browser and the Gmail service is just an exchange of data. When you make a request to the Gmail service, whether to refresh your inbox or to read an e-mail, the response is a set of JavaScript function calls and associated data objects that the JavaScript engine uses to update the document by modifying the DOM.

In this project, we analyze specifically how the browser communicates with the Gmail service when paginating through the inbox. Pagination in Gmail's dynamic AJAX web application happens much faster than pagination in traditional, non-AJAX, non-dynamic web applications such as old-style Hotmail. Traditional pagination is usually implemented as links between documents generated by the server. Each time the user navigates between pages, a complete HTML document is generated and sent to the browser. This approach has mainly two disadvantages:

  1. Each time a new page is loaded, the whole UI is sent to the browser. Because most of the UI is identical to the previous page, superfluous information is transfered between the server and the browser.
  2. Unless the application relies on cookies, there is no way to maintain state and cache data objects in the browser each time the user leaves a page. It is thus difficult to do any kind of smart prefetching, such as prefetching the content of the e-mails on the current page.
    Browser progressbar active

    Browser progress bar depicts progress in fetching next set of email messages

However, pagination in traditional non-dynamic web apps has the following advantages over dynamic AJAX-based web applications:

  1. Allows browser's in-built progress bar to depict genuine information about the progress of loading the HTML page showing the next set of results. This gives a user a more accurate expectation of how long more it would take to load and render the next set of results as the user paginates. A non-informative placeholder spinner is instead used in most AJAX web applications while waiting for the server response containing the requested data.
  2. Allows browser history while paginating as the next set of results are loaded in a completely new HTML page. A user can get back to the previous set of results by merely clicking the browser's Back button. The same cannot be done with AJAX-based web apps as they usually modify the DOM structure of the current webpage to render additional new information. Thus, the browser records no history.

Gmail's dynamic pagination addresses both weaknesses with traditional non-AJAX-based web apps. As a dynamic AJAX-based web app, Gmail modifies the same HTML page when displaying the next set of data items as the user paginates, and caches and prefetch email messages to speed up pagination. At the same time, Gmail's pagination also has the advantages of traditional non-dynamic web apps. It allows the browser's in-built progress bar to depict genuine progress information as Gmail fetches the next set of emails, and also supports browser history when a user paginates.

What is interesting about Gmail's pagination feature?[]

  • Gmail takes advantage of the browser's built-in progress bar to depict genuine progress information while fetching the next set of emails instead of the usual placeholder spinner used in most AJAX web applications
  • Support of browser history in pagination of emails
  • How a user's e-mail data is loaded and stored on the client-side as the user paginates through his/her e-mail messages
  • Prefetching and caching decisions made for fetching the next set of email messages when paginating in the user's inbox
  • The use of "chunked transfer encoding" to allow Comet-style updates from server to client about new e-mail messages received (in the /bind URL)

Overall structure of Gmail's Pagination interaction[]

Iframe1

Role of JS 1/2

Iframe2

Role of JS 2/2

  1. User clicks on one of the pagination links "« Newest", "‹ Newer", "Older ›" and "Oldest »" to navigate among his/her collection of emails
  2. Gmail initiates a GET HTTP request to fetch the email titles, email message snippet previews and email IDs of the next requested set of emails
  3. The browser's in-built progress bar depicts the progress to completion of this GET HTTP request
  4. Gmail modifies the same HTML page to remove the previous set of email results displayed, and loads the email titles and email message snippet previews (the result of the successful GET HTTP request) of the next requested set of emails
  5. While user views the rendered email titles and snippet previews, in the background, Gmail initiates a series of XHR calls using POST HTTP requests to fetch the full contents of the user's e-mail messages for the currently viewed set of emails

Role of JavaScript in Gmail's Pagination[]

  • Processes the event handling when clicking on the pagination links "« Newest", "‹ Newer", "Older ›" and "Oldest »".
  • Used to initiate various asynchronous GET and POST HTTP requests to fetch a requested set of e-mail messages. (Note: Loading a new page in a normal browser window or in an IFrame is initiating a GET request. )
  • Supports the 2-step AJAX e-mail data fetching in Gmail as the user paginates through e-mails:
    • 1st step is a GET HTTP request to get only email titles, email message snippet previews and email IDs triggered by changing the target of a hidden IFrame.
    • 2nd step is a series of XMLHttpRequest (XHR) calls that results in a series of POST HTTP requests fetching full contents of user's e-mail messages. Full e-mail messages are requested in batches of 10 emails instead of the whole set of results all at once.
  • Helps interpret the results returned by the series of GET and POST HTTP requests involved in supporting the 2-step AJAX email data fetching in Gmail:
    • The response of the GET HTTP request from the 1st step is basically a HTML page containing only JavaScript code that passes e-mail titles, e-mail message snippet previews and e-mail IDs to other Gmail JS functions that process and render these data to the user.
    • The response of each POST HTTP request initiated by the series of XHR calls from the 2nd step is a JavaScript Object Notation (JSON) array holding the full contents of 10 e-mail messages at a time. JavaScript is used to interpret and process the JSON data.
  • Helps facilitate caching of e-mail messages previously fetched so that paginating and viewing such emails do not require any further interaction with the Gmail service.

Client and Server-side Interaction in Gmail's Pagination[]

We will explain how the interesting features of Gmail's pagination interaction work together such as Gmail's use of the browser's in-built progress bar, browser history support for pagination, and its 2-step AJAX e-mail data fetching, as the user paginates through e-mails. We will point out which parts of the interaction are on the client-side/browser and which requires interaction with the server/cloud.

1. Gmail's web app layout is made up of four IFrames (HTML elements that allow the embedding of other HTML pages in a HTML page), labelled respectively as:
  • js_frame: A hidden IFrame that contains most of the JavaScript code for the Gmail web app
  • hist_frame: A hidden IFrame that is probably utilized to give browser history support to Gmail
  • sound_frame: A hidden IFrame where a flash object is loaded to provide sounds for Gmail's chat feature
  • canvas_frame: The main IFrame that is seen by the user and where all the graphical rendering of the Gmail web app occurs
2. When the user clicks on one of the pagination links "« Newest", "‹ Newer", "Older ›" and "Oldest »" to navigate among his/her collection of emails, Gmail initiates a GET HTTP request that fetches the email titles, email message snippet previews and email IDs of the requested next set of emails from the Gmail server (Server Interaction). This is also the 1st step in the 2-step AJAX email data fetching that Gmail does.
3. The event handling when a user clicks on a pagination link and the initiation of the GET HTTP request (Client Interaction) is handled by Gmail's JS code residing in the browser.
4. This GET HTTP request is initiated by Gmail's JS code by changing the target of a hidden IFrame, which we suspect is probably Gmail's hist_frame. This hidden IFrame then attempts to load and render the HTML page that is the response of the GET HTTP request. (Client Interaction)
5. This HTML page contains solely of JavaScript code, which upon loading in the hidden IFrame, passes e-mail titles, snippet previews and e-mail IDs to Gmail's other JS functions for rendering/display in the canvas_frame. (Client Interaction)
How this enables Gmail Pagination's support of Browser History
  • The browser treats any IFrame element, hidden or not, like a browser window. Whenever the target of an IFrame element (the address of the website currently displayed in the IFrame) changes, the browser recognizes this as a change in browser history. The browser effectively records a snapshot of the DOM state of the webpage before its embedded IFrame element had its target changed.
  • By loading the result of the pagination's GET request in a hidden IFrame, this effectively gets the browser to record the DOM state of the Gmail app before paginating
How this enables Gmail Pagination's use of the browser's Progress Bar
  • Most AJAX applications uses a placeholder "spinning wheel" to inform the user that some asynchronous operation is in progress. It works well for small and quick operations but is not adequate when performing large operations or computations whose responses might be delayed. It does not provide the user with an expectation or estimate when such a computation would be done.
  • Fortunately, the web app programming technique termed as the IFrame transport mechanism provides exactly what's needed. Whenever the browser is loading a page inside an IFrame (hidden or not), the browser's in-built progress bar also depicts the loading progress just as it would when loading a new webpage in a normal browser window.
  • The idea of the IFrame transport is to initiate the GET request by changing the target of a hidden IFrame to load the response of the request. This response contains email titles and IDs that will later be rendered to the user.
  • As the hidden IFrame loads the response of the GET request, users can then notice the progress bar depicting the progress of Gmail in retrieving the email titles and IDs of the next requested set of emails.
    TwoStep AJAX Email Fetch withBindConnection

    FireBug tracking of GET and 5 POST HTTP requests involved in the 2-step email data fetching by Gmail.

6. As the 1st step in the 2-step AJAX email data fetching completes (the GET request successfully obtains the email titles, preview snippets and email IDs from the Gmail server, and these information is displayed to the user), Gmail initiates a series of XHR calls (in the form of POST HTTP requests) to the Gmail server to fetch the full content of the email messages whose titles are currently displayed to the user. (Server Interaction)
7. Each successful response of the XHR POST HTTP request contains a JSON formatted data structure that holds the full email contents for 10 email messages.
8. Thus, if a user configures his/her Gmail to display 50 emails at a time, then 5 XHR POST HTTP requests are made in the background (while the user is viewing the titles of the emails) to fetch the full content of these 50 emails.
9. The Gmail JS code inserts the email content obtained from successful POST HTTP requests into the DOM structure of the Gmail webpage. (Client Interaction)
10. When the user clicks on the title of an email to view its full content, the result is often immediate with no visible delay as the email content was previously fetched in the background.
11. This series of XHR POST HTTP requests by Gmail is the 2nd step in the 2-step AJAX email data fetching by Gmail.

Data fetching and updates in Gmail's Pagination[]

All the data required while paginating is fetched with the two following calls:

Gmail service's response to asynchronous RPCs contains the minimal set of data objects required to update the UI. Consequently, the granularity of updates is finer than in traditional web applications, where each action requires a full reload of the document.

Despite not being used by the pagination per see, updates such as a new e-mail arriving in the inbox are pushed by the server on an open Comet-style connection at this URL:


Techniques for hiding latency in Gmail's Pagination[]

  • The initial HTML document sent to the browser contains dynamically generated JavaScript objects such as the titles and IDs of first page of e-mails in your inbox as well as your address book. This avoids an additional round trip at the beginning of each session.
  • When paginating, e-mail titles and IDs are fetched first and thus displayed as quickly as possible to the user.
  • The full content of e-mail messages is then prefetched in batches of 10s. Consequently, clicking on an e-mail subject to read its content doesn't require to talk to the server and can thus be done without any latency. Some RPCs are still triggered but only to return secondary data such as advertisements.
  • Once the content of an e-mail is fetched, it is cached in the browser's memory. When a new list of e-mail titles and IDs is fetched, the JavaScript code first checks if some of the IDs map to e-mail messages already cached in memory. If this is the case, there is no need to prefetched them again.

Questions[]

Are there things about the facilities of the Web that made Gmail's Pagination either easy or hard?

  • To be able to run a web application on a single document, the ability to dynamically modify the DOM in the browser is required. The JavaScript DOM API makes this kind of modifications possible.
  • The ability to use CSS to make span elements look like hyperlinks is useful to provide a usual "look and feel" to the controls in the UI.


Did Gmail's Pagination have to contort itself to get around problems with the Web?

To get around the stateless nature of the Web, Gmail pushes the UI logic to the browser. Because a single document is used, keeping data objects in memory is easier. But using a single document introduces other problems, some of them being solved by Gmail's use of a hidden IFrame. Manipulating an IFrame can:
  • trick the browser in providing some progress feedback on pending RPCs to the user.
  • add "fake" events in the browser's history.


Are there improvements to the Web that would have made Gmail's Pagination easier to implement, or that would have allowed a more powerful interaction?

  1. JavaScript may not be the best language for writing such a large UI engine (the obstructed code base is larger than 1 MB!).
  2. Browsers haven't been designed to handle such JavaScript intensive applications.
  3. The data store of JavaScript objects can only be used as a temporary cache. JavaScript won't garbage collect the objects, of course, but the data is lost if the user leaves the document or if the browser crashes.
Interestingly, Google is working on providing a solution to these three problems with:
  1. GWT (the Google Web Toolkit): a framework compiling Java code to JavaScript code.
  2. Chrome: the Google's web browser built to handle rich applications using large JavaScript files.
  3. Gears: let web applications interact naturally with user's desktop by providing a local, off-line data store.


Are the techniques used in Gmail's Pagination relevant for a broader class of interactions?

  • Yes, Gmail's pagination techniques can be very useful for any rich web application that needs to display a long list of data items that requires some method of pagination/navigation such as for a search catalog or an online database
  • It is interesting to remember that Gmail doesn't use AJAX for eye candy effects or unpractical animations. They use it solely to smartly prefetch data to improve responsiveness and to reduce as much as possible the size of data being transfered from the client to the server.
Community content is available under CC-BY-SA unless otherwise noted.
Advertisement

Fan Feed