Anyway, if you take a quick look, you’ll notice that our Server header is “nginx/0.5.26″. (Note: Don’t look on this blog; it’s not hosted on our production server. Look on YaleDailyNews.com.) Anyway, you might notice that’s not one of the two servers that run the vast majority of Web sites out there on the Internet - Apache and Internet Information Services (IIS). We used to send back Apache headers, but what is nginx?

To quote the English version of its documentation (it’s a Russian product), “Nginx (pronounced ‘engine x’) is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server.” Essentially, nginx is a FAST Web server. It’s great at handling requests for static files - CSS, JS, images, etc. Its downside is that it’s not very good at serving requests for dynamic pages: anything that involves parsing server-side code, which is PHP in our case.

So we decided to use nginx in what’s called a reverse-proxy. When you make a request to our Web server, nginx handles that request. It checks to see if the request is for a static file. If it is, it serves the file. That’s much faster than Apache! Under the old system, Apache was serving both dynamic requests, which it’s good at, and static requests, which in a LAMP configuration it is not very good at. In order to serve simple CSS files, it had to use its heavy PHP-interpreter-laden process to serve that request, which is quite memory-intensive and relatively slow. Nginx is much faster.

But what about dynamic requests? After checking to see if the request is static and serving it if so, nginx passes on all other requests to Apache, which is better at handling PHP requests. Apache is running on our server on another port, so nginx passes the request to localhost on that port. Apache sees that request, generates the page using PHP and passes the content back to nginx, which sends it back to you.

I wish we had better numbers, but we didn’t really do as much benchmarking as we should have. One of our main bottlenecks is memory (especially now that we’re doing view caching, which will be the subject of my next post), and nginx uses much less memory than Apache, so we’re able to serve more requests without overloading our server. I’d estimate a 20 percent increase in requests per second.

The other alternative we considered was moving our static assets to Amazon S3. However, in order to do so, we would have to do some reworking in our code to point to the right images location, we’d have to develop a deployment process to synchronize to S3 and we’d have to pay for Amazon’s bandwidth costs. We may revisit the idea in the future though, as well as considering moving to Amazon EC2 for hosting as well. However, for the time being, nginx was an easy and helpful solution.

Sorry about the delay. Posts should come more regularly now. I’m aiming for one every week, but don’t hold me to it!

There are several places around our site where we use MySQL’s DATEDIFF function, which calculates the number of days between two given times. One example is the “Most Popular” box, which shows the most popular viewed, e-mailed and commented stories on our site. That works by retrieving stories posted within the last week sorted by the relevant count (hitcount, emailcount, or commentcount). If there aren’t 10 stories within the past seven days, it will go to 14 days to try and fill the box, then 21 and so on.

The query we were running had the following WHERE clause: DATEDIFF(NOW(), articles.dateposted) < 7. That got back only articles posted in the last week. That worked like we wanted it to, but it was a bit slow. Instead of taking just a couple milliseconds, it would take about half a second. Remember that it has to do at least three queries to get all three types, and if it doesn’t find enough articles within the past week, it has to run the queries again with 14 days, then 21 etc. That really only happens over breaks though, when we’re not publishing daily. Regardless, even three queries taking half a second is a lot.

The problem was mitigated by the fact that we cache the results of our database calls. So we actually only generate that “Most Popular” box every couple hours, rather than for every request. So did I actually need to optimize it? Well why not! I enjoy do this kind of stuff, and it looked like a quick fix. I was reading some slides called Join-Fu by Jay Pipes, who works for MySQL. Slide 17 was particularly relevant. He was talking about date calculations and optimizing them. I thought of our “Most Popular” box and set about optimizing it.

The problem with our query was that it could not take advantage of two important parts of what makes MySQL fast: indexes and query caching. First, apparently queries with functions operating on a column means it can’t use an index. (Indexes are kind of a large topic to get into in this post, so I’ll just direct you to some reading on them.) Not using indexes means the query runs much more slowly. To solve this problem, I followed his advice and rewrote the WHERE condition to be “articles.dateposted > NOW() - INTERVAL 7 DAY,” using MySQL’s INTERVAL syntax. The time taken to execute the query dropped to around 100 milliseconds - 1/5 of what it was!

But because the query still had a function in it, MySQL has to evaluate the result when the query is executed and therefore couldn’t cache the results of the query. Again, query caching is a bit outside the scope of this post (more reading on it), but at the basic level, MySQL caches the results of queries it executes, so the next time they’re run it serves the cached result instead of running the query again. I wanted to make our query use the query cache instead of having to be run every time. To do that, I just replaced the NOW() in the WHERE condition with today’s date as generated by PHP’s date function. So now the WHERE condition looks like “articles.dateposted > ” . date(’Y-m-d’) . ” - INTERVAL 7 DAY”. That query takes about 100 milliseconds to run the first time, but one millisecond or less to run each subsequent time. Remember: MySQL’s query cache persists beyond requests, so it’s cached for as long as MySQL is running (or until MySQL expires it), not just until the end of PHP’s life at the end of the request.

Was the optimization worth it? Probably not, but it’s always nice to improve the speed of frequently run queries by orders of magnitude.

First things first: We changed hosts. For the past year or so, we’ve been hosted on a dedicated box at Softlayer. I have nothing but good things to say about Softlayer - fair prices (a little higher than Layeredtech, but competitive), good features, responsive support, etc. etc. But the server we purchased was the first dedicated server any of us had ever managed, and crud had started to accumulate and make things very difficult.

To begin with, we had Softlayer set up our box with CPanel, which is sort f the industry standard when it comes to server management-control panels. The problem was that as we started to become more competent, we began changing things manually, and CPanel couldn’t stay in sync and thus was rendered effectively useless. I also had a sneaking suspicion that some of its “helpful” services that run to keep your server maintained were actually slowing us down. I definitely thought that it was time to move to a new
server, one where we could start fresh.

After reading around (the Web Hosting Talk forums are invaluable), I decided Slicehost would be a good place for us. (We also considered Linode.) As I alluded to in my previous post, getting a VPS offered several benefits over a dedicated server. First, Slicehost takes daily and weekly snapshot backups of our server, meaning we could restore to a working server easily if something every happened. Second, we could upgrade the size of our “slice” if we ever experienced a surge in traffic. I suggested it would take “a matter of minutes,” but as neodude pointed out, it actually takes about 10-15 minutes. Regardless, being able to upgrade our server that easily is a very helpful feature.

Additionally, Slicehost’s tutorials for setting up a LAMP server are excellent. We decided to install Ubuntu Gutsy on our slice, and using the magic of apt-get, we had everything we needed set up in just a couple hours (much of that learning how to do some fairly basic things). Most importantly, the server was fresh and entirely configured by us: We could make changes easily and keep everything organized how we wanted to. It is great.

If you have some Linux competency, you should consider setting up your own server. With package managers these days, you don’t need to worry about compiling, just configuring, and even the defaults are generally suitable, so it’s very easy.

There are other fantastic features of Slicehost. Although not actually run by the company, the members of the Slicehost IRC channel have been extremely helpful. Also, Slicehost provides a SSH console in their Web manager that you can use to log in to the server if SSH stops working for some reason. (Like, for example, forgetting to leave your SSH port open in the iptables.)

We’ve made some other tweaks to make our Web site run faster. (Hint: look at the “Server” header we’re sending back.) The goal is to survive a Drudging or something similar. I will have more information in future posts about some more of what we’re doing.

The first goal was to get some page online, instead of just appearing down. Not loading anything was the worst thing that could have happened. Fortunately, we have a check built-in very early in our application to see if it can connect to MySQL. All we had to do was change the MySQL password to be incorrect, and the application starting redirecting to our error page. That’s fairly simple for the server to handle — all it has to do is try and connect to MySQL, fail, and then redirect to a static HTML page. (We didn’t shut down MySQL entirely because it also serves some other sites besides yaledailynews.com, and we wanted those to continue working as much as possible.) But an error page definitely isn’t ideal, and if we didn’t get the site back up quickly, Drudge would drop its link. We would lose the traffic, and it would be less likely to link to us again.

The next step was to get something showing up for Drudge users. We copied the text of the article and pasted it into a static HTML file. Then we had our application redirect all users coming from Drudge to that static HTML file with the popular story. Our server was able to handle that. But it was erroring for everybody else, which was no good. So we put a link on the error page to the static version of the story, which seemed like a good idea at the time. However, as our editors informed us, we didn’t want to seem like we were blowing the story up unnecessarily, so we removed the link and went back to the drawing board.

The story was continuing to explode. Our static page was holding up, but we really wanted to get all of our stories back online. We had implemented view caching (I will blog about this in the future), and the pages were being cached in memory in XCache. However, I noticed that we were hitting our memory limit as Apache processes were spawned. Our caching system can fall back to file-based caching if we tell it to, so I figured we could try that. I allowed only my IP address to be able to access the main site, and clicked on a couple pages to prime the caches. This is important — if we started our site up with an empty cache, the server would overload trying to fill the cache as people started visiting. I primed the cache, then opened things up.

Wonder of wonders, miracle of miracles, the site stayed alive. It was a little slow, but it was going. We kept users with Drudge in their referrer going to the static site. Drudge’s traffic is overwhelming — it dwarfed all of the other referrers, which are not small. With this setup, we managed to stay alive for most of the rest of the time.

On Friday, we hit the front page of Digg, Fox News and Drudge at the same time for our follow-up stories. I made static pages for each of those stories, and routed all traffic just to those specific URLs to the static pages. With those optimizations, we made it through the week, and over the weekend traffic subsided to manageable levels.

Going forward, we are implementing some changes very soon. First of all, we are going to move from our dedicated host to a virtual private server. We’ll be able to rid ourselves of CPanel, which is a waste of resources if you can manage a LAMP server adequately by yourselves. Also, we’ll be able to resize and get more resources in a matter of minutes, rather than hours, in case we get another spike. Additionally, we’re considering moving some of our static files (CSS and background images, JavaScript files) to Amazon S3. That will result in faster downloads for our visitors, and Apache won’t have to serve as many requests. Even though it’s fast with static files, it can only help.

So that’s what happened. As we implement some further changes, I will blog more about them. The goal is to be able to survive a Drudging or Digging or any major linkage. We are close to getting there, but we have some work to do.

Some of you may have erroneously received many duplicate emails from the Yale Daily News in the past few minutes. It was a result of server testing we were conducting to improve our headlines email service, and it went awry. I want to apologize for the error, and assure you that we are taking steps to prevent it from happening again. If you have any questions, please feel free to contact me.

Thank you for visiting the Yale Daily News online.

Robert Baskin
Online Director, Yale Daily News
webmaster@yaledailynews.com

I’m pleased to announce the introduction this week of two interactive additions to YaleDailyNews.com. Even if each serves a “niche” interest, I hope one or the other sparks in you a related idea for how the Yale Daily News can continue to enhance its online presence.

For decades, Yale College Council elections have dominated Yale College life for at least one week each spring. But this year, the regular inundation of commentary and campaigning is set to amplify at least tenfold with the introduction of “Yale Decides 2008,” a live blog established to guide the student population through next week’s race with punditry, predictions, interviews, video and analysis.

Second, in attempt to promote a more transparent newsroom, we have launched “Inside 202 York,” a forum in which readers can prod editors — the News editors in this week’s case — with questions, however friendly or hostile, about recent coverage or general practices.

Thank you, as always, for your ideas and support.

Sincerely,
Andrew Mangino
Editor in Chief
Yale Daily News

While I was perplexed that he bothered to drag himself out of bed for a 9 a.m. lecture that he wasn’t even going to listen to, I was ecstatic that he chose to spend his time looking at our Web site.

Competition for viewers online is fierce so once we get a reader on our site we try to do everything in our means to keep them from browsing to some other entertaining Web site.  An extensive tagging system is one of the prime ways we’ve come up with to encourage people to explore the site instead of leaving after reading just one popular article.  By providing links to other content that users might be interested in, the site tries to guess what interests each viewer.

For example, when a viewer is reading an article about Richard Levin, they will be given links to related content pages about the Yale Corporation, the campus expansion, and Richard Levin himself.  Once on those topic pages, a viewer can click through to any number of other articles, tags, photos, or videos related to that topic.

Cool, huh?

Conceptually, the tagging system is very simple.  When editors upload articles to the site, they add content tags to each stories by hand.  Once the story has been put online, the content management system does the rest of the work.

One of the problems we ran into with this feature was that the editors refused/forgot/didn’t want to add tags to stories and photos as they upload them to the site.  We quickly realized that if wanted the system to get used, the content management system would have to tag stories automatically.  While still allow editors to tag stories manually, we now tag stories automatically when they’re uploaded by searching through old articles and guessing where a tag might apply.

While we were initially skeptical about having an auto-tag feature, it turned out to work pretty well.  For example, when looking for articles to tag with Admissions, the system finds:

• Elis graduate with no skills — and few prospects
• Ivy admissions prompt frenzy
• Acceptance rates decline across the Ancient Eight

Pretty good.

However, when the auto-tag feature looks for articles to tag with the word Court (as in the New Haven County Superior Court), it doesn’t do as well:

• Appointment of Tony Blair sullies University reputation
• Before Yale, Gentry a ‘finesse player’ on court
• Shin found guilty of faking doctorate

We generally ended up getting more sports articles with the word court (e.g. basketball court, tennis court, etc.) than articles about the legal system.  Refining the tag name a little, by changing Court to Crime, makes the search work much better.

To incorporate the tag system into the user experience, we decided to put lots and lots of links to tags pages all over the place.  Every time the editors post an article, the system automatically links every instance of a tagphrase in the article body to its corresponding tag topic page.

Look familiar?  We got some inspiration from the New York Times’ application of the same concept.

One of the first steps we took was to combine CSS and JavaScript files into one file for production. In development, we split up those files to make them easier to manage. But, in production, you want browsers to have to make as few connections as possible. Each HTTP request for a new file (HTML, CSS, JavaScript, image, Flash object, etc.) takes time to look up the domain name, connect successfully and download the file. Therefore, we combine our code into one CSS and one JS file.

You may notice two things about these files. First of all, we remove all white space and unnecessary characters — for example, the last semicolon in a CSS declaration isn’t actually needed. We run the Minify library on our files to format them to minimize their size. We also have Apache’s GZIP module enabled to zip the files as they go over the wire to minimize transfer time and bandwidth costs. GZIPing of CSS and JS files isn’t enabled by default, because older browsers could not handle it properly, but all modern Grade-A browsers accept it with no problem.

The second thing you may notice is the vXXXXX in the filenames (CSS and JS). We have Expires headers turned on for our CSS and JS files. That means that, when your browser requests them for the first time, we let it know that those files will not change for the next 10 years — forever, as far as the Web is concerned. The next time you load the page, your browser knows they haven’t changed, so it doesn’t request them again. But, of course, we do change those files whenever we update our CSS or JS, and the only way to make your browser request them is to change the filename. To avoid having to go through our code and change the filenames every place they’re included, we pass a “Last Updated” time stamp in the filename. Apache Rewrite Rules remove that from the request, so the Web server always serves ydn-min.css or ydn-min.js. All we have to do is update the time stamp whenever we change the CSS or JS.

Those are just some of the performance measures we’ve enacted. We’re hoping to do more, both on the server and client side, in the near future.

We decided to retrieve the crimes data from the Yale Police Department daily crime logs. The next task was how to present the data. We knew we wanted a map, and for a while we toyed with the idea of creating our own Flash map. Google Maps was always an option, but we can’t customize the look and feel as much, and it doesn’t show the names of campus buildings. Eventually, we decided that an established mapping solution with an easy API was worth using, so we chose Google Maps. The only major problem we had with it was that their GMarkerManager doesn’t have a removeAll() or clear() function (which we needed when we cleared the map to show new crimes), so we turned to the basically second-party MarkerManager script. It’s a bit slow with a
lot of markers, so we limit the results to 100 crimes so the browser doesn’t hang for 5-10 seconds.

Then we turned to the table of crimes, which we had to regenerate on every Ajax request. We went with Scriptaculous’ Builder functionality. As we can say about many things, it worked great. in everything except Internet Explorer. First of all, IE demands that all nodes are appended to a <tbody> instead of directly to the table. And it requires that you build the entire <tbody> before appending it to the <table>. But we finally got it working. And after some basic testing, we actually found that it was faster than creating the table by setting innerHTML, but we may revisit that. (We tested this with Firebug’s JavaScript profilers in Firefox 3 Beta 3, so the results are definitely not comprehensive.)

We ran into some fun issues with the tab box structure for the form on the left - look for an upcoming blog post here that will talk a bit more about the tab boxes.

Overall, we’re very excited about what we came up with. We think the data is relevant and well-presented, and it’s very cool to take a feature from conception to launch in less than two weeks. We hope that you find it useful.

Robert Baskin
Online Director
Yale Daily News

It may only have “Yale” in its title, but the Oldest College Daily has long included comprehensive coverage of New Haven as well. In the spirit of that dual mission, today we launch an interactive crime map for Yale’s campus and the surrounding neighborhoods, with crimes — sortable by type, date or location — dating back more than a year. You may, for example, select your residential college (if applicable) and see every crime committed nearby over the past week, month or year. (You may also select the two proposed residential colleges on Grove Street!)

You can access the map at http://www.yaledailynews.com/crimes. As always, please send your comments and questions to editor@yaledailynews.com.

Sincerely,
Andrew Mangino
Editor in Chief
Yale Daily News