The app was pulling large amounts of data from an Oracle database using stored procedures, and building presentation html right there in the servlet layer. We couldn’t afford software to properly analyze the app, so we wrote a quick timer and started doing simple start/stop events throughout the loading of particularly slow pages, tracking the time it took to perform specific actions.

We slowly figured out that no one part of the app was being egregiously slow; The whole thing was just uniformly slow. Obviously loops through resultsets were taking up the bulk of the time, but inside the loops each statement performed at the same equal snail’s pace.

Searching for help on the web didn’t help. I’m not sure if my google-fu was just in its infancy back then, or if there really was no information out about this problem. Eventually I started writing simple test servlets and tried to do the same task in different ways, hoping to flush out the piece of code that was causing the slowdown. As it turned out, I stumbled upon the solution in Sun’s Javadocs.

The Problem

Apparently in Java, Strings are immutable (they cannot be altered after they are created). The idea is like so: Lets say you’re going to display an invoice to a customer. You have their name in a string, pulled from the customer table. You read in the invoice, and you have their name again, pulled twice from the billing and shipping information. Now you have three string variables all storing the same static bit of information in memory. Rather than waste 3x the space, Java simply points all three to the same chunk of memory.

Now you might think this would be a bad thing; What if the customer changes their billing address name? Because strings cannot be changed after they are created, every time you think you’re changing a string, you’re actually creating a brand new string and pointing your variable to the new string. In the event that the old string has no more variables pointing to it, it will be garbage collected.

Understanding how this whole mess affected this application has to do with how we were building our html. You see, building a page went something like this:

...
String body = new String();
body += "<table border='1'></table>";
body += "<tr>";
body += "    <th>Column 1</th>";
body += "    <th>Column 2</th>";
body += "</tr>";

while( rs.next() )
{
    body += "<tr>";
    body += "    <td>"+rs.getString(1)+"</td>";
    body += "    <td>"+rs.getFloat(2)+"</td>";
    body += "</tr>";
}
body += "</table>";
...
return header + body + footer;

That’s right, every line of html that was being created for every page was done by string concatenation. With strings, when body += "html here" is used, it creates a new string containing the concatenated contents of both strings, then points body to the new string. This means that the old string body pointed to, along with the html that was concatenated with it, are now floating around waiting for garbage collection. Each and every line of html created using this method has the side effect of creating 1 or more new string variables. Object creation is expensive in Java, and the garbage collection for these excess strings was nearly as expensive.

The Solution

The solution was actually pretty trivial, but time consuming to implement. Java has a StringBuffer class which is specifically designed for this type of problem. It stores the same information as a String, but it a mutable fashion. This allows us to alter its contents without the hassle of object creation and garbage collection. The fixed code looked like so:

...
StringBuffer body = new StringBuffer();
body.append( "<table border='1'></table>" );
body.append( "<tr>" );
body.append( "    <th>Column 1</th>" );
body.append( "    <th>Column 2</th>" );
body.append( "</tr>" );

while( rs.next() )
{
    body.append( "<tr>" );
    body.append( "    <td>").append( rs.getString(1) ).append( "</td>" );
    body.append( "    <td>").append( rs.getFloat(2)  ).append( "</td>" );
    body.append( "</tr>" );
}
body.append( "</table>" );
...
return (new StringBuffer( header.toString() )
                 .append( body )
                 .append( footer )
       ).toString();

For anybody doing Java web apps now, you just won’t run into this problem as often. JSPs allow for much more readable presentation code, and you rarely run around appending to your html like this. That said, the misuse of strings in Java is in no way limited to this example here. The problem crops up more than you’d imagine, with experienced and inexperienced developers alike.

New Job, Same Problem

At my second big computer job, I was surrounded by some very smart people. They had been using Java for years, and were currently maintaining a huge, high performance web app using JSPs, Servlets, and EJBs. They were handling exclusively backend work, and passing off finished code webmonkeys to design UIs around. I remember sitting in my little cubicle, pouring over code rapidly, trying to get a grasp of how the application worked. I overheard three of the senior developers discussing a problem a few cubicles over. Much to my surprise, they were having performance problems with a new piece of code that sounded suspiciously like it was an immutable string issue.

I listened for a minute or two, then swaggered over to solve their problem. What took days to figure out at my first job was related and then fixed in mere minutes here, and the performance issue disappeared. I’m always reminded of this moment, and how even experienced developers don’t know everything. There’s always something more to learn.

<?php
if(is_single() || is_page()) {
    $description_meta = get_post_meta($post->ID, "description", true);
    $keywords_meta    = get_post_meta($post->ID, "keywords", true);
}
else if( is_category() ) $description_meta = category_description();

if("" == $description_meta) $description_meta = get_bloginfo('desciption');
if("" == $keywords_meta)    $keywords_meta    = "curtis,tasker,curtis tasker";
?>

<meta name="description" content="<?php echo($description_meta); ?>"/>
<meta name="keywords" content="<?php echo($keywords_meta); ?>"/>

Every post or page in WordPress has a nice little GUI for entering Custom Fields of data. By using these, I would be able to customize each blog post’s metadata, while maintaining the default for the rest of the pages.

While this performed admirably, I recently did a little improvement. While working on something totally unrelated, I realized that the post tags and keywords meta for each blog post were almost exactly the same. Apparently the overlap never occurred to me when I first built the site.

A solution was actually pretty simple:

$keywords_meta = implode( ",", wp_get_post_tags( $post->ID,
                                                 array('fields'=>'names') ));

Now it pulls the list of tags for the blog post, then converts it into a comma-delimited string, and uses that for the keywords meta. All in all, a much cleaner solution that speeds up the process of publishing a blog post. Every little bit helps.

Common Reasons For Removal

• The content of the page becomes outdated, and is removed entirely.
• The website is redesigned, and old information is moved to an entirely new location
• The type of information changes (a pdf becomes a web page)
• Static content becomes dynamic

Coping With The Problem

There are a few basic ways to cope with out of date URLs. You can display a custom 404 error, perhaps with a search box to allow the user to more easily find the page he was looking for. You can redirect the user back to the root of your website, and hope they can find their way from there. Ideally, you can configure your web-server to redirect all requests from the old URL to the new URL. Unfortunately, the latter process can become quite unwieldy, and is often just ignored, especially on larger websites and over long time periods.

The best solution is to plan ahead when first launching your website. With proper planning, you can configure your website’s URL structure to cope with changes in underlying design and technology without changing the basic URL structure.

If a piece of content is going to be advertised or linked to, its URL should be a folder. example.com/resume is far preferable to example.com/resume.htm or example.com/resume/resume.pdf.

If your website is highly dynamic, generating content on the fly, then your URL structure is even more critical. example.com/home.php?page=intro&lang=en&spash=yes is confusing, and painful to type by hand. example.com/intro is simple and easy to type.

How I Handle It

Everybody seems to have their own opinions on how to set up your blog with permanent links. The main concerns seem to be SEO, which isn’t terribly important to me, but I took it into account. I’ve never had a lot of publicly accessible links on my website, so there weren’t many links to preserve. I did choose to maintain most of the hosted files and private content, as I’d planned for that ages ago.

I settled on this structure: http://curtistasker.com/blog/%category%/%post_id%/%posttitle%. I wanted the word ‘blog’ in the url somewhere, to indicate at a glance (and to search engines) that this was a blog post. I plan on using a small number of categories with little overlap, so having the category in the URL is reasonable. It provides an additional keyword for search engines, and offers users a bit more information to go along with the post title. Post ID is there mostly to make search engines happy, as quite a few people seemed to think a three or more digit numeric in your URL was beneficial. And finally it closes with the post title.

Tweaking WordPress to handle this structure wasn’t terribly bad, though its since gotten easier. I’ve been able to get rid of a custom plugin I wrote to handle category links, as newer versions of WordPress can handle what I want by default. I also use a lot URL Rewriting to help prevent duplicate links, by forcing www.example.com/file1 and example.com/file1 and example.com/files/file1 to all redirect to the same URL. I also discourage duplicate indexing of content (category and tag pages with summaries).

Finding Perfection

I started out with a standard IBM PC keyboard, one of those giant clackers, and a dreary old grey mouse with my first computer. Since then I’ve gone through literally dozens of upgrades. Ball mice gave way to optical mice, which in turn gave way to laser mice. In the past few years I’ve gone through half a dozen gaming mice, before finally settling on a Logitech G9. Most of my relatives have a castoff mouse that once graced my desk, but my right hand is finally happy.

My keyboard churn has followed the same breakneck pace. I’ve gone through keyboards with built in trackpoints, purportedly ergonomic split keyboards, a dozen near-disposable $10 keyboards, a Das Keyboard (100% blank, of course), and two backlit keyboards. I started with loud clicky keys, moved to silent soft keys, then back to clicky, then back to soft. I never did find one I was happy with.

That changed when the aluminum Apple Keyboard caught my eye. After a trek to an Apple Store and half an hour coding away on it (to the mild amusement of the staff), I brought one home. I’ve since moved to the wireless version, and aside from my occasional lapses when I try to use the nonexistent Numpad or Home/End PgUp/PgDn keys, I’m happy. In fact, the quality of the typing experience was so good that it factored in heavily to my MacBook Air purchase.

Windows Support

Despite both the wired and wireless keyboards being fairly fantastic as far as the hardware goes, the software support for Windows is severely lacking. The default mapping isn’t ideal. The software offered by Apple for bootcamp is difficult to find for a Windows user, and once found is just not up to the task. When I first bought the wired keyboard, I spent hours putting together a melange of hacks to turn the keyboard into a usable device on Windows.

A few of the problems:

• Media & Volume keys are nonfunctional
• Command key where Alt key should be, but acts like Windows key
• Option key where Windows key should be, but acts like Alt key
• Right Control key acts like Right Alt key
• Fn key where Insert key should be (wired keyboard only)
• Clear key where NumLock key should be (wired keyboard only)

That covers most of the big issues, but the list of minor issues can go on for some time.  I used SharpKeys to remap the F7-F12 scancodes to the windows media keyboard equivalents, turn F13-F15 into the PrintScreen, Scroll Lock, and Pause keys, and make the Ctrl and Alt keys work properly. This proved to be an adequate solution at the time.

After I got my Macbook Air, I wanted more parity between how the keyboard on my laptop and desktop worked. Since I had upgraded to the wireless Apple Keyboard around the same time, I particularly needed shortcuts to replicate the lost buttons. Again using scancodes, I turned the Command key into Ctrl, to make cut/copy/paste use the same gesture as the Mac. I used a scripting language called AutoHotkey to add some additional functionality, namely Ctrl+ArrowKeys shortcuts for Home/End/PgUp/PgDn, and Fn+Backspace for Delete.

More Problems Arise

All was well until I started beta testing new program, and my mapping fell apart. You see, this program bypassed the standard Windows keyboard device driver, and used their own instead. This made things problematic as my scancode mappings were thus ignored. The program used Ctrl + # and Alt + # heavily, and the Ctrl and Alt keys were just not where my fingers expected them to be. In addition, Command started acting like the Windows key again, launching the start menu when pressed. Words cannot adequately express how highly annoying this all was.

I tried to code my way out of this problem with more scripting, and ended up realizing that scancodes just weren’t doing the job anymore. I decided the only viable solution was to replicate all the key mapping functionality in my script instead.

An Easy Solution

Before I got too involved in this project, I did a little digging on the net, and a program called UAWKS caught my eye. It had been released just weeks prior, and it basically does everything my scancode remapping and scripting did, with a slick bit of user interface thrown on top. I swapped over entirely to this program, with minimal tweaks to the underlying code to suit my needs.

There are a few lingering issues, but I wholeheartedly recommend UAWKS to anybody running an Apple keyboard under Windows. It is a requirement to get the proper Windows behavior out of the keyboard.

Of course, all this is going to be a moot point once I replace my desktop with a Mac. At least I’ll be able to use the same mouse and keyboard, without all the hassle. At this point, my lust for new input devices is sated. Of course, who can say what the next year’s product line will bring?

Update (03/25/2009): Just spent an hour on the Wacom Intuos4. *sigh* With a little willpower, perhaps I can keep the tablet consumption rate down to something reasonable.

You can use a .htaccess file in any folder of your website, and it will apply to any subfolders. A single .htaccess file placed in the root of your domain can apply to the entire website. While this is advantageous for blocking access to files, you’re going to need a seperate .htaccess in each subfolder that you want to password protect.

Blocking access to resources

# block access to all .ht* files
<files ~ "^\.ht">
    Order allow,deny
    Deny from all
</files>

# block access to wp-config
<files wp-config.php>
	Order allow,deny
	Deny from all
</files>

This blocks access to all files that begin with .ht, as well as the wp-config.php file. You can use this type of definition to block access to any file or folder of your choosing.

If you’re running WordPress, ideally you should move your wp-config.php file to a location on your server that is above the root of your domain; It contains security information that you really don’t want somebody getting access to.

Password protecting files and directories

This is not a terribly good method of protecting your content, as none of the data is encrypted as you access it. In addition, unless you use digest mode, your user name and password are sent in the clear every time you type them in.

Creating users and passwords

First, you need to create some user names and passwords and store them in a .htaccess file.

% cd /home/username/webapps/www.example.com/
  htpasswd -c .htpasswd user1
  Adding password for user1.
  New password:  pass
  Re-type new password:  pass

% chmod a+r .htpasswd

Switch to the root folder of your domain, and run the htpasswd command with -c to create a new password file, with user1 as a user. Then type in a password for that user. Finally you chmod the file to ensure it has the proper permissions.

You can add additional users to the file by simply using the above command without the -c option. You can also create groups of users, by creating a file called .htgroup and that looks like so:

my-users: user1 user2 user3 user4

You should ideally put your .htpasswd file in a location that isn’t accessible from your website, such as your home directory.

Setting up the authentication (Basic)

AuthType Basic
AuthName "My Protected Folder"
AuthUserFile /home/username/webapps/www.example.com/.htpasswd
AuthGroupFile /dev/null
Require valid-user

AuthType we’re using is Basic (password sent in the clear).

AuthName is an arbitrary name you assign to your protected content. If you protect multiple directories, and give each directory the same AuthName, then the user will only be required to entire their information once; They will then be granted access to all the directories.

AuthUserFile is the location of the .htpasswd file, which grants users listed in this file access.

AuthGroupFile is the location of the .htgroup file, if you wish to grant a group of users access. In this case I’m defining a null group.

Require can list either specific user names group names, or any valid user or group from the Auth files.

Setting up authentication (Digest)

This is slightly more secure than Basic mode, as your password is sent as an md5 hash rather than in the clear. To use Digest mode, use the following code:

AuthType Digest
AuthDigestDomain /
AuthDigestProvider file
AuthUserFile /home/username/webapps/www.example.com/.htdigest
AuthName "My Protected Folder"
Require valid-user

You will need a new password file, which can be created using htdigest. Follow the same steps used when creating the .htpasswd file, the syntax for the command is the same.

Well, such a procedure is relatively easy. Apache .htaccess files can be used in any folder of your website, and will apply to any subfolders. Thus, a single .htaccess file in the root of your domain can apply to the entire website.

The aforementioned example is fairly simple, but URL rewriting of extreme complexity is possible once you know the rules. Generally, instead of simply changing file1.html to folder/file1.html using a single rule, you will instead change every file fitting the form fileX.html to folder/fileX.html. That is to say, any request for a file that matches a specific pattern will be rewritten to a new URL.

Regular Expressions

It’s a bit beyond the scope of this post to teach regex in depth, but here’s a few pointers to help decipher the code in the next section.

As a mildly complex example, consider ^www\.([^\.]+\.[^\.]+)$. This starts at the beginning of the line, matches the character string www, followed by a period, followed by 1 to N characters that are not period, followed by a period, followed by 1 to N characters that are not a period, followed by the end of the line.

It matches any URL that fits the form: www.example.com. It then stores a backreference for the example.com portion of the match. Think of a backreference as a saved variable containing the text that was matched inside a pair of parenthesis. Multiple backreferences can be made per block of regex, and they can be used later on in your code.

URL Rewriting

Apache defines a few simple commands that allow you to use regex to dynamically alter a URL. The official documentation is a great help, but the examples on the web are of much greater instructional value. Generally you begin URL rewriting like so:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /

and ends like so:

</IfModule>

It uses an if..then block to enclose all URL rewriting commands, and will only evaluate the block if the correct Apache module is installed. It begins by turning on the rewrite engine, and setting the base of all rewriting to the root directory.

RewriteCond and RewriteRule control how a URL is rewritten. Generally you write 0 to N RewriteCond statements followed by a single RewriteRule statement. The RewriteRule will only be executed if every RewriteCond statement preceding it matches something.

RewriteCond takes two arguments: the string to match against, followed by the pattern to match. RewriteRule takes two arguments: the pattern to match, followed by the rewritten URL. RewriteRule always uses the REQUEST_URI variable as its string to match against; If the URL was http://www.example.com/folder1/file1.html, it would match against the string folder1/file1.html.

Strip off the www subdomain

RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

This starts with the HTTP_HOST variable, which contains just the www.example.com portion of the incoming URL. It then matches www., and stores a backreference to all characters that come after that.

Assuming that the URL did indeed contain a www., then the RewriteRule comes into play. The pattern ^(.*)$ will match everything in the REQUEST_URI, and store a backreference to the string. The URL is then rewritten using the two backreferences.

Map subdomains to subfolders

RewriteCond %{HTTP_HOST} ^([^.]+)\.([^.]+\.[^.])$
RewriteRule ^(.*)$ http://%2/%1/$1 [R=301,L]

This starts with the HTTP_HOST variable, which contains subdomain.example.com. It finds the subdomain using ([^.]+)\. to match 1..N characters that is not periods until it reaches a period. A backreference is stored as %1.

Next, it finds the domain using ([^.]+\.[^.])$ to match 1..N of characters that are not periods (example) followed by a period followed by 1..N characters that are not periods (com). It stores the domain in the backreference %2.

Finally, the RewriteRule uses the pattern ^(.*)$ tomatch everything in the REQUEST_URI, and stores a backreference as $1. The URL is then rewritten using the three backreferences. So, subdomain.example.com now becomes example.com/subdomain.

Prevent image hotlinking

If you host images on your website, you may want to prevent other websites from stealing your bandwidth by hotlinking to your images.

#RewriteCond %{HTTP_REFERER} !^$
#RewriteCond %{HTTP_REFERER} !^http://(www\.)?example.com/.*$ [NC]
#RewriteRule \.(gif|jpg|png)$ - [F]

The first RewriteCond will match any request where the referring site is not empty. The second will match any request where the referring site is any site except your own site (example.com or www.example.com).

If either of those conditions are met, the RewriteRule kicks in, and matches any file that ends in gif, jpg, or png. So, if any outside website links to any file on your website that ends in those 3 file extensions, it will return a forbidden response.

WordPress URL Rewriting

If you use WordPress, when you customize your permalinks through the admin interface, WordPress will attempt to alter your .htaccess file to add the following lines:

RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

What this does is check to see if a requested file or directory exists (example.com/directory). If the file or directory exists, nothing happens, and you are able to access the resource as usual. If it does not exist (example.com/postname), the RewriteRule activates, sending the request to the WordPress index.php. From here, the WordPress permalink php code takes over, translating your request into the WordPress resource you requested.