The conventional way of building web-based applications involves writing the user-facing components of the application in html-based languages such as Sun's JSP (Java Server Pages) or Microsoft's ASP.
This paper describes a web application architecture that exploits Java's type-checking ability to validate field parameters and to detect invalid links between pages. Supporting tools are described that allow the application to be built entirely as Java servlets with no html-based languages at all. The html can be read from files or coded inline as multi-line strings as described in the supporting paper, Multi-line Strings with Executable Inclusions.
This paper is the outcome of a decade of experience building large interactive web-based applications:
This diagram shows
the sample application we'll be using. It is drawn as a state machine
with the states as ovals and the legal transitions between states as arrows.
The Home state is the entry page and the only part of the application
that can be accessed without logging in. From there the user can leave
via the arrow at the top right or click a link for that user's account.
If the user is not logged in the account redirects the request to the
login page. If this visitor doesn't have an account yet, the login page
provides a link to the Register page. In a real system the account would
provide links to the services of that user's account. Our example only
provides a single service, an address page via which the visitor can edit
the account's address and phone number. The demo is simple, but it
demonstrates everything that is involved in real applications: logging
in, redirecting requests between pages, maintaining a database,
supporting user interaction and validating user requests.
Each state abides by the model, view, controller framework as shown at
the left. This framework was originated by Trygve Reenskaug for Smalltalk
during a summer visit to Xerox and since adapted to other languages and
environments. The model is the object to be made accessible to the
user. The lens in the center supports the interaction by providing two
parts, the view and the controller. The controller
controls the interaction by validating information arriving from the
user along the upper arrow and updates the model accordingly. The view
receives updated information from the model and notifies the user by
sending html text to the browser.
In the architecture to be described here, each state is a Java class
(a subclass of State) and its Controller is a public method named
controller()
. How views are implemented is not specified
because each view is private to each state. The style I'll demonstrate
below implements views as private methods within each state class,
but other approaches such as reading html text from files are also
common.
In this architecture, an application consists of a foundation package (com.sdi.wap.demo.bean) that contains the applications models as Java Beans, and a superstructure (com.sdi.wap.demo.site) that contains the application's State classes. The superstructure package also provides a subclass of Site that defines the site as a whole. The site class provides a servlet that manages the servlet protocol and calls the States' controller() methods as required. But that's it. Everything is Java. Html-based languages like JSP are not used.
This
figure shows the architecture in more detail. An interaction begins with
the application sending html text to the user's browser along the
response
arrow. Typically the text includes html forms
commands to invite the user to revise information that the view retrieved
from the model via the Retrieve arrows and embedded into the
value="..."
fields of the html form. The user clicks
a submit button to returns revised information along the
request
arrow as html request parameters, including a
special parameter that tells the PageServlet the state class that
should handle this request. The state's a controller
method validates the received information and responds accordingly,
either returning incorrect entries to the user for correction or
sending correct entries to the model. The model applying highest-level
validation (business rules) and updates the DBMS as appropriate.
In practice, each state is a subclass of the abstract class, State. This class provides instance variables for maintaining execution-time variables (in particular, the HttpServletRequest and HttpServlet response objects provided by the servlet engine). The abstract class provides a number of methods that subclasses use to access session attributes, obtain request parameters, and so forth.
The servlet engine's request.getParameter()
methods
returns Java Strings
and the model will generally wind up
storing these in the DBMS via the jdbc setString()
method.
So we should declare the Street, City, USState,
and
Zipcode
fields of our example as Strings, right?
Wrong!. Declaring everything as String because the endpoints use this type ignores the applicaion that will connect them. Strings should be used only as a lowest-level building blocks and should never appear except at the very lowest levels of an application.
Here's why. If everything a String, Java's type checking will be unable
to prevent errors like passing a zipcode to a method that expects a street
name. Less obviously, validation is tedious because the code must be
duplicated in each state that handles that kind of field. If you
provid no validation architecture in lowest-level fields like Street
and Zipcode, there is no infrastructure for validating higher-level
types like AddressBean or AccountBean. And finally, the servlet
engine's methods for retrieving Strings from request parameters
will return null under some conditions. If you let these leak
into the application, you multiply the number of cases your code
will have to handle. Having to write if (field == null || ...)
all the time is not fun, wastes RAM, and makes your code hard to read.
Rather, each form in your application should be examined with the question, "What is really being handled here?" in mind. The answer will never be String but an application-specific datatype such as Street, City, USState, Zipcode, or Phone. Make this explicit by providing a Java class for each one, and encapsulate the syntactic validation rules within that class. At the very least, every class should prevent inputs that are too long to fit without truncation into the database field size. Often there will be several ways for inputs to fail, provide a way to return explanations of exactly why the validation failed in terms the user can understand. For example, a phone number might not pass validation because the field is empty or null, because it is too long for the database field, or because it is not parseable as a phone number, and so forth.
A validation architecture is a set of interfaces and abstract classes that provide a consistent foundation for building the application-specific classes. The validation architecture we'll be using here is defined by the following API as defined in the Validatable interface:
The com.sdi.wap.demo.field package in the software distribution for this paper defines several low-level field types, each implemented as a subclass of Field: Op, Identifier, Email, Street, City, USState, Zipcode and Phone. The Op field, by convention, holds the value of the submit button (which is, by convention, named "op"). Identifier is used for values that will serve as public keys in the database.
The abstract Field class provides supporting methods that help to make these subclasses remarkably simple. For example, here is the full text of the Zipcode class.
package com.sdi.field; import gnu.regexp.*; public class Zipcode extends Field implements Validatable { private final static RE re = re("\\d{5}([ -_]?\\d{4})?"); private final static String msg = "A 5 or 9 digit zipcode is required"; public final static Zipcode Null = new Zipcode(null); public Zipcode(String value) { super(value); } public boolean setValue(String v) { setValue(v, true, ""); return requireNonNull() & requireMatch(re, msg); } }
This example relies on the GNU regular expression package by
passing a regular expression and an error message to display to the user
if the match fails, to the requireMatch()
method, which
Zipcode inherits from Field.
The architecture also provides a simple technique that helps to
reduce the dreaded NullPointerException from your application.
Notice that the Zipcode class defines a public final static
instance named Null
that is initialized to an empty (and
therefore invalid) instance of that class. Aggregate types, such as AddressBean,
use these to substitute for Java's null
when initializing
variables of each type. For example:
public class AddressBean extends BeanImpl { private Identifier accountID = Identifier.Null; private Street street = Street.Null; private City city = City.Null; private USState state = USState.Null; private Zipcode zipcode = Zipcode.Null; private Phone phone = Phone.Null; public final static AddressBean Null = new AddressBean( Identifier.Null, Street.Null, City.Null, USState.Null, Zipcode.Null, Phone.Null );
Now let's turn to how the validation architecture helps to simplify higher level code. Here is the controller method for the LoginView state, which supports the login procedure.
public final void controller throws Exception { try { AccountBean thisAccount = (AccountBean)site.getAccount(this); redirect(DemoSite.Account); return; // should never happen } catch (SessionExpiredException e) { /* empty */ } Op op = (Op)getField("op", Op.Null); Identifier identifier = (Identifier)getField("identifier", Identifier.Null); if ( op.isValid() && identifier.isValid()) { try { Connection connection = ConnectionPool.getConnection(); AccountBean account = AccountBean.load(connection, identifier); site.setAccount(account, this); redirect(DemoSite.Account); } catch (Exception e) { identifier.setValid(false, "This identifier was not found"); sendPage(viewLogin(identifier)); } } else sendPage(viewLogin(identifier)); }
The try block at the beginning of this method bypasses the login procedure if the user is already logged in. The first statement inside the try block asks the site object (described later) to retrieve the AccountBean of the currently logged in user. If this succeeds, LoginView simply forwards the request to AccountView page. Otherwise the catch block is taken which falls through to the body of the method.
The first step of all controllers is to retrieve informatio from the request parameters into the application-specific field types discussed in the previous section. The getField method provides a simple way of doing this. The method's first argument is is the name of the request parameter and the second is a default value to be used if the request parameter is empty ("") or null. When this controller is called the first time, no request parameters will exist and all fields will take the default values in the second argument, so the else clause at the very bottom will be taken to display the login page.
If the visitor types a valid email address into the account identifier field and clicks the submit button, the form will return control to LoginView.controller(). This time, the op field will contain the value that the form assigned to the submit button ("op") and the identifier will contain the value of the input field named "identifier". If both of these are valid, the second if statement will succeed and the login procedure in the main clause will be attempted.
The login procedure first obtains a connection from the ConnectionPool
and tells AccountBean to attempt to load the instance from the database
record that has identifier
as its primary key.
This attempt will fail and an exception will be thrown if the identifier
is semantically invalid, e.g. an otherwise valid email address was
provided, but it is not the identifier of a registered account.
If so, the code marks the identifier as semantically invalid
by calling setValid with a description of the problem that
the view will display to explain the problem.
As recommended in most JDBC texts, connection pooling is used to avoid the overhead of creating a new connection for each DBMS access. The LoginView retrieves a connection from the pool. The AccountBean.load method saves this connection in the loaded instance and provides a finalize method that returns the connection to the pool when the AccountBean is no longer referenced, which will happen when the user logs our or the session expires.
If the load succeeds, the connection is saved in a transient AccountBean instance variable so that this connection will be available for all database accesses during this session.. The site object (Site) stores the AccountBean in the servlet session object. It will persist there as the account of the currently logged in user until the user logs out or the session expires. The redirect call reroutes the reqeust to the AccountView page via which gives the user access to the account's services.
The pattern shown for the LoginBean controller is followed by all controllers. They all begin by calling getField to construct field variables from the request parameters, providing suitable default parameters in the second argument for when the request parameters do not exist. The LoginView is unique only in that all of its default parameters are Null. The others are able to provide default values by loading their previous values from the database. The AddressView controller, shown below, shows how this is done. This page is called by clicking a hot link that AccountView provides as the sole account service in this small demonstration.
public void controller() throws Exception { AccountBean thisAccount = (AccountBean)site.getAccount(this); AddressBean address = thisAccount.getAddress(); Op op = (Op)getField("op", Op.Null); Street street = (Street)getField("street", address.getStreet()); City city = (City)getField("city", address.getCity()); USState state = (USState)getField("state", address.getState()); Zipcode zipcode = (Zipcode)getField("zipcode", address.getZipcode()); Phone phone = (Phone)getField("phone", address.getPhone()); if ( op.isValid() && street.isValid() && city.isValid() && state.isValid() && zipcode.isValid() && phone.isValid()) { address.setAddress( street, city, state, zipcode, phone); address.save(thisAccount.getConnection()); forward(DemoSite.Account); } else sendPage(viewAddress( op, street, city, state, zipcode, phone)); }
The first line retrieves the AccountBean for the currently logged in user by requesting the site object to retrieve it. If the session has expired, this call will throw a SessionExpiredException. This will be caught by the site's servlet which will redirect the request to LoginView so that the user can log in again.
The AccountBean loads its AddressBean from the database and uses it to obtain default values for each field variable. The Op field's default value is always Op.Null to ensure that the else branch will be taken when the controller is first called.
The simplicity of the controller methods is due to two things.
First, the logic for presenting the session state to the
user is encapsulated within view methods (to be
shown later) and doesn't clutter the controller's code.
Second, the logic for syntactically validating user input is
encapsulated the Validatable
interface. Low-level Java types like String
or null
never appear in higher levels of the application.
In conventional html, the pages of a web site reference one another via urls and these are hard-coded into <a href="..."> or <form action="..."> commands. If a target page is moved or renamed, the error will not be noticed until an the invalid link is clicked. The Site architecture originated in the desire to use Java type checking to report invalid links at compile time.
The solution was to define a container object to represent the
site as a whole, and to have this object define protected final
static
variables for each web page within that site. The
com.sdi.wap package supports this via the
Site interface.
The package also provides an abstract implementation of this
interface, SiteImpl,
which each site subclasses to define that site. For example,
DemoSite
defines the demonstration application described in this paper.
DemoSite defines a protected final static
variables for each page as follows:
public class DemoSite extends SiteImpl implements Site { public static final Page Home = newStaticPage( "Home", null, Role.Null, "/html/index.htm", "Demo home page" ); public final static DynamicPage Account = newDynamicPage( "Account", AccountView.class, "Account", Registered, "To your account" ); public final static DynamicPage AccountRegistration = newDynamicPage( "AccountRegistration", AccountRegistrationView.class, "Registration", Role.Null, "Account Registration" ); public final static DynamicPage EditAddress= newDynamicPage( "EditAddress", EditAddressView.class, "Address", Registered, "Modify your address and/or phone number" ); public final static DynamicPage Login = newDynamicPage( "Login", LoginView.class, "Login", Role.Null, "Login procedure" ); public final static DynamicPage Logout = newDynamicPage( "Logout", LogoutView.class, "Logout", Role.Null, "Logout" ); public final static DynamicPage Refuse = newDynamicPage( "Refuse", RefuseView.class, "Permission Denied", Role.Null, "Permission denied" ); }
Notice that the initializers call the static methods,
newwStaticPage
and newDynamicPage
according to whether the page is plain html text (which will be
served by the web server) or dynamic servlet-based code which
will be served by the servlet engine.
These two methods pass their arguments to the appropriate StaticPage or
DynamicPage constructors,
storing the resulting instances in private static final Hashtables,
staticPages
and dynamicPages
. These
classes extend the abstract Page
class, which defines storage for the information provided in the
constructors and provides default implementations for methods
inherited by the two Page subclasses.
When the servlet's init method is called (when the servlet engine is starting up), the instances in the two hashtables validate the information provided to their constructor. In particular, StaticPages ensure that the file specified in their url constructor parameter actually exists the appropriate filesystem location relative to the web server's document root, and prints a warning in the log file as appropriate. When a view needs to emit a <a href="..."> or <form> command to the browser, it calls the desired page's emitLink or emitForm method to generate the url. Therefore, if the code compiles correctly and the log file contains no warnings, all hotlinks within the DynamicPages of that website are guaranteed to be valid. (StaticPages, of course, a different matter since they contain hand-coded urls.) For example, AccountView generates a link to the EditAddress page as follows:
DemoSite.EditAddress.emitLink(this, "Address")
A second benefit will only be obvious to those who've used servlet sessions extensively. The session machinery uses cookies to manage sessions if the browser supports cookies. But since some browsers do not support cookies, and since users often turn cookies off, the servlet engine provides a low-level mechanism to encode the needed session information into each url that the programmer remembers passes through this mechanism. The problem, of course, is that urls that are hard coded into html text do not pass through this mechanism, so the user's session information will to be lost if they click on such a link. The site infrastructure automatically ensures that this problem cannot happen since all DynamicPage references automatically pass through the url-encoding mechanism. Links between StaticPages are, of course, still prone to this problem because the urls are hard-coded into the html text.
The Site object is responsible for providing the HttpServlet protocol
for that site, so SiteImpl is a subclass of HttpServlet. It defines
HttpServlet's doGet
and doPost
methods
to transfer requests to the doRequest method. This method handles both
kinds of request by obtaining the identifier of the requested State
from a request parameter reserved for this purpose (do
),
looks up the DynamicPage with this identifier, and launches the Page's
controller by using the reflection API to call the no-args constructor
and calling the instance's initialize method to populate its instance
variables with a reference to the site and the request and response
servlet arguments. Those familiar with JSP will notice the strong
similarity between State and JSP's PageContext class.
Each site's Site subclass is also responsible for providing concrete implementations of the abstract htmlPageOpen and htmlPageClose methods. Each controller inherits sendPageOpen and sendPageClose methods, which relies on the site's htmlPageOpen and htmlPageClose to generate standard strings of html text to begin and end each page. Most simple pages use the simpler sendPage(aString) method, which simply calls these before and after emitting aString as the content of the page. Every site override these methods and they can be as complex as desired. For example, the htmlPageOpen method for the superdistributed.com web application generates a rather elaborate navigational menu at the top of each page that shows not only where the user is now, and where they can get to, with respect to the current page, and also the site resources that are accessible to that user's Role in the system. This menu simply presents additional links for users authorized to play employee versus administrative roles versus customer roles.
Notice that sites are defined by defining a Java subclass, not by editing resource files, configuration files, XML files, or the like. I noticed that I'm far more comfortable working with Java code than I am with the arcane and continually changing configuration conventions of systems like Apache and Tomcat. Every Tomcat release seems to involve a new configuration format based on brand new terms understood only by the Tomcat developers. Common configuration errors are rarely checked and reported with the precision we take for granted of Java compilers. Why force users to learn a specialized configuration languag for each tool when the job could be done in a widely understood language that reports errors properly?
In the model, view, controller framework, views do not access the database directly. Rather they interact with models, which provide an object-oriented API and hide messy SQL details inside their load and save methods. In our example, when a user registers a new account, the RegisterView object calls the AccountBean load method by providing it with a database connection object and the Identifier (primary key) of the instance to be loaded. Notice how getString values are immediately converted to type-checkable Fields as soon as they emerge from the database.
public static AddressBean load(Connection connection, Identifier accountID) throws Exception { String sql = "select * from DemoAddress " + "where accountID=\"" + accountID + "\""; try { Statement stmt = connection.createStatement(); ResultSet s = stmt.executeQuery(sql); if (!s.next()) throw new Exception("couldn't do " + sql); return new AddressBean( accountID, new Street(s.getString("street")), new City(s.getString("city")), new USState(s.getString("state")), new Zipcode(s.getString("zipcode")), new Phone(s.getString("phone")) ); } catch (SQLException e) { throw new Exception("SQLException in " + sql + "\n\t" + e); } }
When all fields pass their syntactic validation checks, RegisterView calls the AccountBean's save() method to add the new account to the database. The save methods typically call isValid() internally before issuing the SQL commands to add this instance to the DBMS. The following example shows how JDBC is used to add information to the database:
public Identifier save(Connection connection) throws Exception { String sql = "replace into DemoAddress set " + "accountID=?, " + "street=?, " + "city=?, " + "state=?, " + "zipcode=?, " + "phone=?"; try { PreparedStatement s = connection.prepareStatement(sql); s.setString(1, accountID.toString()); s.setString(2, street.toString()); s.setString(3, city.toString()); s.setString(4, state.toString()); s.setString(5, zipcode.toString()); s.setString(6, phone.toString()); int n = s.executeUpdate(); s.close(); return accountID; } catch (Throwable e) { throw new Exception("Throwable in " + sql + "\n\t" + e); } }
I've saved the views for last because how they are implemented is not specified by the architecture described in this paper. That is, the State interface only requires that each State must provide a public controller() method. How the views are implemented is up to the controller, not the architecture.
The reason the architecture doesn't specify a single "right" way is that there is simply no "right" way for all environments. Most shops involve cooperation between programmers and user interface designers and way the two groups (and their tools) view a web application are opposed. Html designers view html as static text in a file. But web applications are dynamic, with the html generated by the application. Between static and dynamic polar opposites are an infinity of intermediate points, all of which could be criticized for veering too far to one side or another. For example, JSP, JSP Tablibs, WebMacro, Enhydra, Turbine, WebObjects, Cocoon and others extend html with language features such as executable inclusions (variable substitution), conditionals (if statements), loops, and so forth. All of these languages are better than no language at all. But all of them are limited compared to a general purpose language like Java For example, Java can support extensible validation architectures as described in this paper, whereas html extension languages like JSP support request parameters as Strings.
In my envinonment, I play both roles and only use WYSIWG page layout tools for designing the look and feel for sites as a whole. When the site design is completed, I import the site's html into the site's htmlPageOpen() and htmlPageClose() methods, and then extend these with code to provide dynamic features such as automatically generated navigation menus and the like. The html within each individual page is simple, routine, and usually written by hand. Rather than inventing a restricted language to embed code within html, I use Java as the template language. Instead of using languages restrictions to enforce discipline, I use a language that is capable of supporting a discplined architecture to separate presentation (html) from implementation (code). I simply code the views as private methods of each state class which the controller calls like any other.
The main obstacle to this way of working is that Java's string syntax
makes long sequences of html text tedious to write by hand. The solution
is described in the paper,
Multi Line Strings with Executable Inclusions.
The executable inclusion feature described there amounts to a way
of integrating code with html, and could therefore be considered a
template language like those listed above. The difference is that
the tool is used during the compile process instead of when files
are read at runtime. For example, here is the viewLogin method
of the LoginView class:
private final String viewLogin( Identifier identifier)
throws Exception
{
String message = "Click here to proceed";
return {{
<h2 align=center>Login Page</h2>
<p>This demonstration uses the email address you gave when you
registered as the account identifier and does not support passwords.
{{DemoSite.Login.emitForm(this)}}
<input
name=identifier
size=14
type=text
value="{{identifier.getValue()}}"
> Account Identifier
{{htmlFontRed(identifier.getMessage())}}
<br><input
type=submit
name="op"
value="Login"
> {{message}}
</FORM>
{{ DemoSite.AccountRegistration.emitForm(this) }}
<input
name="op"
type=submit
value="Register"
>If you haven't registered a name and email address yet, begin here.
</form>
}};
}
This example has been artificially colored to emphasize the two
modes that the preprocessor operates in. The parts of this text that
the preprocessor treats as an executable inclusion are colored
blue and the parts that
are treated as part of a Java string are colored red.
Notice that the view rely heavily on the validation architecture to
emit field values
({{identifier.getValue()}}
)
and diagnostic messages
({{identifier.getMessage()}}
)
into the form. Notice that the site architecture only emits
the opening <form> command
({{DemoSite.Login.emitForm(this)}}
),
but the closing </form> is done by hand.
The model, view controller paradigm basically divides a web application into two parts. The models (beans) provide the low-level foundation upon which the high-level superstructure rests. This consists of the controller/view classes that interface the models to the user.
For the foundation-level classes, I use Visual Age for Java. My only real dissatisfaction is that I mainly use the Linux edition which only supports Java 1.1.8. I'm evaluating OTI's Visual Age Micro Edition as a possible replacement.
For superstructure-level work, I use vi, mls, and IBM's jikes, driven by a Unix Makefile to emit binary into the application's WEB-INF/classes directory. I often import the superstructure's source code into Visual Age for browsing and to double check that all is well.
Visual Age is a very capable debugging environment, reminiscent of Smalltalk and Interlisp. By loading Tomcat into the Visual Age environment, the entire web applications can operate under control of the debugger. However Tomcat must be specially configured for this to work, and the complexity of the Tomcat configuration process and the frequency of new Tomcat releases which seem to always involve a painful reconfiguration, made me seek a more sustainable approach.
The boundary between foundation and superstructure is a natural place to do unit testing as distinct from debugging. I've adopted Kent Beck's junit unit test infrastructure to build a large library of unit tests for most of the beans in the system. I run these inside Visual Age, whose debugger is quite useful for tracking down any problems. Ensuring that the superstructure interacts with a stable, tested foundation goes a long way towards reducing the need to do line-by-line debugging of the superstructure level code, so I usually just add print statements to debug problems at this level.
The primary conclusions from this experiment is that Java is a congenial environment for building web applications once its main limitation, uni-line strings, is relaxed with tools such as MLS. The Validation Architecture makes controller logic simpler than in JSP by encapsulating syntactic field validation logic in application-specific Field classes and by providing a simple mechanism whereby views can obtain error messages to return to the user. A second reason is that the Site architecture provides a high-level technique for ensuring that all lines are valid at compile time while ensuring that all urls are properly encoded for browsers that don't support cookies. Although similar features could be added to JSP, I doubt that I'll go back because the current approach has a significantly smaller footprint and is more supportive of my style of work.
The architecture described here is neutral to whether html text should be held in files or in RAM as Java strings. I've described a personal coding style that uses a version of the MLS preprocessor that converts text to Java strings, which is also JSP's answer to this question. The difference is that JSP will compile JSP files and load the resulting Java code into memory when the page is referenced and unload it when the last reference disappears.
By contrast, the site's SiteImpl subclass holds a reference to all DynamicPages that will keep them in memory for the lifetime of the servlet. This consequence follows directly from the need to need to emit all <a href="..."> and <form action"..."> commands via java code intead of hard-coding them into html files so that all urls will be session-encoded and to support invalid link detection at compile time. This table suggests that the space overhead is modest even for large sites like superdistributed.com.
Bytes | Name | Contents |
---|---|---|
133895 | bean.jar | Low-level site-specific classes. Java beans and supporting classes such as com.sdi.wap. |
148247 | site.jar | High-level site-specific classes; e.g. Site and State classes |
619081 | tomcat.jar | Apache's Tomcat 3.2 distribution |
637419 | xml.jar | IBM's XML distribution |
The high-level classes for this site (the views and controllers) adds only 150kb to the image size. This is negligible compared to the image size as a whole (64mb) and is only 1/6 the size of the Tomcat and XML distributions. This supports the conclusion that space optimization efforts needn't start from the assumption that the html text within the view methods of dynamic pages is the dominant contributor to execution image size.
The End