Michi Knows

I finally get why Apple chose Safari for the iPhone. It’s because Safari does content rendering amazingly fast: especially AJAX. This should have been more evident when the iPhone was first revealed and a customized Google Maps application showed the little red pin dropping on the map a smooth animated way. I’ve included the video below.

Watch the animations carefully and remind yourself that Google Maps is running inside a browser on a phone:

At work, I am writing an application that does some AJAX event handling. The scripts I use fire up as soon as the page finishes loading, and takes linearly more time to initialize as the page size increases.

• In IE, the page will freeze up for about 1 second while the JavaScript fires up.
• In Firefox, this same freeze time is about 1/2 a second. 
• In Safari, I can’t notice the load up time. And I’m really paying attention.

On the iPhone, this will be a Godsend. I say that because sometimes I load up a website on my Blackberry, and occasionally things freeze up for 10 seconds while it tells me “running JavaScript…” (stupid CNN.com)

The very first time I launched Safari, it was slow as pudding — probably doing updates or something behind the scenes. But after the third or fourth start up, it was amazingly fast. I mean really fast. Firefox and IE both take a few sounds to launch. Safari is probably twice as fast (as advertised!). I don’t know how they do it, especially since the entire application has a visual skin, but they did it.

Another interesting aspect of the Safari browser is memory usage. Many people have probably noticed it is a beast when it is being actively used. And it is. But it is actually very smart in how it manages your memory. Observe:

1. I have a blank window of Safari open and it is using 22 megs of memory.
2. I visit Google.com and it goes up to 29 megs.
3. I open a tab and visit Digg.com. It is now up to 43 megs.
4. I minimize it. It is now using 1.2 megs.

Now that was unexpected. Clearly Safari only uses memory it knows is freely available, and then releases it back as soon as it recognizes it is no longer needed. In a limited environment such as the iPhone, I can see why this sort of streamlined memory management is ideal.

So despite there being a few hiccups in the (still Beta) browser, I am, overall, very impressed with its quality.

Blizzard has officially launched starcraft2.com, and with it, the official announcement that Starcraft 2 is well under development. There is already game play footage all over YouTube. I was a big fan of Starcraft when it came out a decade ago. The game looks amazing. No release date has been revealed. The game is introducing tons of new units and is definitely not just an expansion set to the original Starcraft. See footage below for proof.

Update: more videos added:

The following is part 3, but I couldn’t find one with the audio commentary. Make sure to watch this one to the end.

Part four:

I just realized a funny irony about Microsoft competing with Google. See, Google has some really advanced JavaScript when it comes to its Adsense scripts (look to the left of or below this article). If you’ve never thought about this process, here’s your chance. The JavaScript that makes these ad unit boxes must:

• Scan the page and send the contents to Google for analysis, get the response, and serve an ad accordingly (AJAX)
• Build an entire block of HTML from scratch (DOM manipulation - very annoying)
• Fire after the page loads, to ensure all of the content is in place (event handling)
• Override any CSS or other page altering scripts to ensure people can’t be tricked into clicking (CSS hacking!)
• Look and function exactly the same in all browsers

Well, the stuff I listed up there involves some of the most annoying aspects of programming in JavaScript (believe me). AJAX, Event handling, DOM manipulation, and CSS pretty much sum up the four pillars of “the most inconsistent things in JavaScript.” Thanks to IE, these tasks are a pain in the ass, when they really shouldn’t be.

Microsoft has to ensure their ads appear in all browsers, especially Firefox (#2 browser). You can’t sell your services to advertisers if it’s known that your ads break, and possibly misfire, when the wrong browser hits it. That means they’re forced to use a standards compliant implementation to do their ads.

This, of course, means they’re taking their own medicine and finding out what a horrible pain it is to support Firefox 1.5, Firefox 2.0, IE5, IE6, IE7, Opera, Safari, etc. It’s mostly a pain because of IE and its inconsistent and buggy support for the established JavaScript and CSS standards.

Maybe this is why they appear to be supporting some standards in IE7. Or, maybe they’re just building in the ones they use. Either way, this explains a lot.

WARNING: Extreme computer geek post!!

In the near future, someone will come to this page because they are trying to setup rsync using SSH authentication. The documentation out there isn’t that great, and the process is rather complex. I just spent my entire day setting this up, so I am going to share how you can do this too. Thank me.

Step By Step - Setting up rsync!

Some assumptions:

• Assume [S] is the SOURCE that you are syncing FROM (where your files are)
  • [S-IP] is the IP of [S]
• Assume [D] is the DESTINATION that the code is being pulled to (where your files will be)
  • [D-IP] is the IP of [D]
• It is assumed the rsync script is run FROM [D].
• Assume the user [USER] will be used to do the syncing. (DO NOT USE ROOT)

Now blindly follow these instructions:

1. Install rsync. It is highly advised you install as new a version as you can find that is past 2.6.4.
2. Create a [USER] (or whatever you want to call it) user on both servers.
3. If it doesn’t exist already, make sure the home directory (i.e., /home/[USER]/) has a .ssh directory. This directory should have chmod 700
4. On [D], go to login as [USER] and go to your home directory. Now make the SSH public keys:
   1. Type in ssh-keygen -t dsa -b 2048 -f rsynckey. Do not use a passphrase.
   2. Edit rsynckey.pub
      1. At the very beginning of the file, add in (no line breaks and directly before the stuff that’s already there) from=”[D-IP]“,command=”/home/utility/valid_rsync_commands.sh” (with a space between this and the old stuff)
   3. Copy rsynckey.pub to [S] in the /home/[USER]/ folder (type: scp rsynckey.pub [USER]@[S-IP]:/home/[USER]/).
   4. Type in mv rsynckey ./.ssh/rsynckey
   5. Type in mv rsynckey.pub ./.ssh/rsynckey.pub
   6. If they aren’t already there on [S], add the two following lines to /etc/services
      1. rsync 873/tcp
      2. rsync 873/udp
5. Login as [USER] on [S]. Make sure you are in the home directory (/home/[USER]/).
6. Run chmod 700 ./valid_rsync_commands.sh
7. Run cat ./rsynckey.pub >> ./.ssh/authorized_keys
8. Run chmod 600 ./.ssh/authorized_keys
9. Run rm rsynckey.pub
10. create 3 files:
    1. rsycd.conf (chmod 600)
       1. Example contents:

          pid file = /home/[USER]/rsyncd.pid
          log file = /home/[USER]/rsync.log
          use chroot = no
          read only = no
          list = false
          uid = nobody
          gid = nobody
          hosts deny = *
          secrets file = /home/[USER]/rsyncd.secrets
          [sometitle]
          path = /var/www/mycode
          comment = Sync my folder
          auth users = syncuser
          hosts allow = [D-IP]

    2. rsyncd.secrets (chmod 600) Add in a single line that says (no spaces): syncuser:mypassword

    3. valid_rsync_commands.sh (chmod 700)

       1. Put in the following code:

          #!/bin/sh case “$SSH_ORIGINAL_COMMAND” in
          *\&*)
          echo “Rejected”
          ;;
          *\(*)
          echo “Rejected”
          ;;
          *\{*)
          echo “Rejected”
          ;;
          *\;*)
          echo “Rejected”
          ;;
          *\<*)
          echo “Rejected”
          ;;
          *\`*)
          echo “Rejected”
          ;;
          rsync\ –server*)
          $SSH_ORIGINAL_COMMAND
          ;;
          *)
          echo “Rejected”
          ;;
          esac

11. On [D], create the following files:

    1. rsync.exclude (chmod 600)
       1. Put in your exclusion rules in this file. Here’s my personal list of file types I exclude from my synching (the minus signs are on purpose, and should be included in the file):

          - .svn/
          - no_sync.*
          - *.pdf
          - *.log
          - *.bak
          - *.tmp
          - *.sh
          - *.zip
          - *.gz
          - *.tar
          - *.txt
          - *.cs
          - pdf/
          - *.ini
          - *~
          - .*

    2. sync.sh (chmod 700)
       1. Paste

          rsync –exclude-from=”exclude.sync” –progress –stats –compress –rsh=”/usr/bin/ssh -l [USER] -i /home/[USER]/.ssh/rsynckey” –archive –delete-after –delay-updates syncuser@[S-IP]::sometitle /var/www/destination

Explanation

Without going into too much details, the configurations I list here are the final results after comparing everything I could find on this page that talks about SSH authentication, this page on rsync daemon, this page on rsyncd.conf, and this page on the rsync client.

When you run the sync.sh script (by typing in ./sync.sh on [D]), it will:

1. compress the data before pulling it.
2. look in the exclude.sync folder for any exclusion rules.
3. login to [S-IP] using the username [USER].
4. not prompt you for the password of [USER].
5. use the private key we created called rsynckey, found in /home/[USER]/.ssh/ on [D].
6. know where to copy from based on the “sometitle” in the URL of the rsync command.
7. authenticate within rsync using the username syncuser.
8. prompt you for a password that you specified in rsyncd.secrets (I put it down as “mypassword”).
9. copy all files on [S]:/var/www/mycode to [D]:/var/www/destination.
   1. The source is specified in the rsyncd.conf file.
   2. The destination is specified in that last block of code above.
10. keep your [S] server relatively safe since the key can only be used to run rsync commands. Plus, no passwords are stored in the clear.
11. Block the syncing of files that start with a period, PDFs, log files, tmp files, etc. Edit this as your please.

Now, slowly troubleshoot and fix this up. When trying to test, keep an eye on [D-IP]:/home/[USER]/rsync.log. This file will show you error messages when rsync is acting weird. If you keep getting errors and that file isn’t filling up, it means SSH authentication is failing — make sure you follow these instructions correctly. If you really wanted to run the sync process from [S] to [D], you have to switch all of the public key steps around and make sure all of the files appear in opposite locations. For example, the conf file on [D] would dictate which files get overwritten, rather than read.

REMEMBER: Word Press converts my quotes into its own slanted versions. Make sure you change them back to their original non-slanty quotation marks!

While it’s true that the vast majority of stories on Digg come from an “elite” few, the truth is that they know how to submit a story in an eye-catching, informative, and interesting manner. I am going to teach you this skill. As an added bonus, this happens to be an important skill in crafting good emails, blog posts, and anything else that has a title or subject.

1) Submit titles that are clear, concise, informative, and yet, leave more to be desired. In short, spend 80% of your submission time thinking about the title because 95% of the time, it will be the only thing a person will use to judge if the rest of your submission is worth reading. This means:

1. Don’t ask questions as a title. Make statements in question form (How to mow lawns, Why I rock, etc)
2. Don’t be long winded with multiple sub-clauses. One clause.
3. Don’t use passive voice. Use passive voice and present tense.
4. Don’t leave out the critical information. The keywords you use are extremely crucial.
5. Don’t use URLs as a submission title. Ever.
6. Don’t just copy the title because sometimes the article title sucks.

Here’s the story title that made the front page on the DRM/EMI topic:

Press Release: EMI Music launches DRM-free downloads.

Here’s a list of stories that did not make the front page on the same topic:

• Deal or no deal: EMI and Apple’s DRM-free premium music? (question format)
• EMI Music Launches DRM-Free Superior Sound Quality Downloads (wordy)
• Times Onlines take on EMI Dropping DRM (huh? Onlines? What?)
• EMI unlocks their music! (didn’t mention “DRM”)

Focus your attention here. A bad title is pretty much a torpedo to your submission, no matter how great the content is.

2) The description usually doesn’t matter, but if you do include one, a bad one can actually hurt. The best and most recent example I know if this post (thanks for submitting though!!). Descriptions:

• Should be short and sweet. But this is only the case when the title is very informative.
• Don’t paste the first paragraph of the story. Paste the most interesting and juiciest piece.
• Piece together the best parts if no one sentence stands out as exceptional.

In short, if you write a bad description, people may actually become deterred from reading the rest. If the article has nothing juicy, keep your description to a minimum to avoid making potential readers bored before they even click. Then again, if the article is that dry, it is probably not worth submitting at all. An example for this article would be to piece together the first two or three bolded points. Alternatives would be to paste just one item in the list that you found to be counter intuitive or exceptionally interesting. As I previously implied, most of the time, people won’t even look at the description before deciding if the article is good or not.

3) Use power words. Words that make your audience feel special or privileged are the most effective. Examples:

• The secret to the stock market
• How to avoid Internet scams (”avoid” and “how to”)
• The easiest cheat to winning at everything
• Why rainbows are circular
• Five reasons you should know about Kung Fu
• Top reasons to stay in school

4) Use lists. Your audience is often working people, and everybody loves to save time by reading bullet points and lists. Thus, titles that imply that type of content also fair well (as well as content with lots of lists). Any type of list with a quantifiable amount is already ahead of the game:

1. Top 10 most expensive Google Ad keywords
2. The 5 most popular brands of cookies
3. 7 things to know about traveling to Vietnam

Note very similar lists without the numbers, and see how much less “presence” they have:

1. The most lucrative Google keywords
2. The most popular brands of soda
3. Things to know about traveling to China

It’s a subtle but powerful difference.

4) Submitting without friends is suicide. If you really want to get stuff to the front page, you need at least one or two Digg friends, minimum. Well, I’d say the true minimum is 10 or 15, but the point is that without some other people to get the ball rolling, many times your submission will die with one or two Diggs. This is how the big fish get their stories on the front page over and over. If you have friends that read Digg, get them to register, and get them to befriend you. If you Digg solo, you have no chance. By the way, my user name is DiggMichi. I haven’t gotten anything Dugg on that account though.

5) The time of day matters. Don’t submit it at 2AM. Submit it during regular hours to ensure regular readers see the article. But at the same time, don’t submit during the peak morning hours (9AM PST) because you will have intense competition. From my observation, the best time to submit is around noon, PST.

6) Submit technical stuff to Reddit at the same time. Reddit has a much more technical crowd. If you find yourself submitting articles from this site that are technical in nature, submit it to Reddit first. This is because the site is far less competitive and has many dual users with Digg. A user from Reddit that likes your story might vote it up on Reddit, thus getting more users to come, as well as Digg the story. But keep in mind that Reddit is a much more technical crowd, and submitting anything else might not go as far.

How I would Digg This

As a closing example, I thought I would present a few options in how I would name this story, if I were submitting it to Digg:

1. 6 easy secrets to getting Your stories Dugg
2. 6 must-know tips to avoid getting buried
3. 6 Do’s and Don’ts of submitting to Digg
4. The 6 obvious reasons why 99% of Digg submissions are junk

I only spent a minute thinking about this: you can do better. I try to keep my blog titles from being too cliche, which is why I don’t follow my own advice all the time (but look back, and you’ll see I do it on some articles). But this is another reason why I emphasize you not simply copy-and-paste titles of articles. Authors have different audiences in mind when writing their content. For example, I cater to technically versed readers, whereas Digg is a “general” news site. In summary, think about your audience when crafting your submissions.

Happy submitting!

P.S. Thanks to anybody that’s ever submitted my articles anywhere, even the guy that pasted the URL as the title.

My regular readers probably didn’t realize, but I was recently on the front page of Digg, Slashdot, and Reddit. Digg and Slashdot are notorious for killing servers that get linked in their stories. If you want to see the stats for the days in question, you should see my post about it. This post is a little geeky, so beware.

Here’s the quick and dirty list for keeping your server up during a Digg Crisis:

1. Download and enable wp-cache.
2. Use the .htaccess rule I explain below. It keeps the rest of your site operational and even lets people post comments directly from the mirror!
3. Turn off miscellaneous plugins. The biggest suspect was my related posts plugin (see below).

CPU Load

When I got on the front page of Reddit, my traffic spiked immediately. My server load went up to about 2.00. A “2.00″ is high, but is manageable and won’t cause any real problems. A “1.00″ approximately equates to “100% of CPU used.” So you can guess what a “2.00″ means. Realizing Digg was coming next, I knew I had to start making preparations, since this graph shows just how much bigger Digg is than Reddit.

I knew my hits would increase 100x, so I had to shut down as much unnecessary stuff as possible. First, I wanted to turn off every plugin I had on Word Press and then activate wp-cache, which would significantly reduces database overhead. Unfortunately, before I was finished configuring the cache settings, I hit the front page of Digg in record time. The server became non-responsive and I couldn’t even hit the “enable” button. My server load sky rocketed past 500.00. At 500, things stop working.

Integrating Dugg Mirror

Dugg Mirror is a service that creates copies of articles that hit the main page of Digg. Their goal is to serve as a backup in case the main source goes down (as it often does). As soon as my server was Dugg, my objective was to forward all traffic to the mirror.

When my server died, I was racing against time to redirect the traffic. Until I redirected the traffic, I couldn’t do anything to mitigate the problem (such as disabling plugins). It took me about five minutes (due to incredible lag) to connect to the server, go to the correct directory, and edit the .htaccess file to add this line at the very top:

RewriteCond %{HTTP_REFERER} (digg.com) [NC]
RewriteRule maybe-google-wanted-to-be-sued-youtube-and-plan-b http://duggmirror.com/tech_news/Why_Google_wanted_YouTube_to_be_Sued/ [R,L,NC]

The first line says, “If this visitor is from digg.com”. The second line (wrapped over two lines) says, “then redirect hits to my google article to the Dugg Mirror.” This redirected everybody who came to my site from Digg that wanted to see my article to the Mirror. Note that the rest of my site worked completely fine, and anybody trying to post a comment directly from the mirror was able to do so.

Upon applying this fix, my server load dropped to 4.00.

To make this work for your dugg articles, use the following:

RewriteCond %{HTTP_REFERER} (digg.com) [NC]
RewriteRule [article URL without beginning/trailing slash or domain] [dugg mirror URL] [R,L,NC]

The Finishing Touches

I made sure my most CPU/MySQL intensive plugins were off when a digg user came (since they were the ones causing problems). I put a snippet of code around my related post plugin that looked like this:

<?php if(FALSE === stristr($_SERVER['HTTP_REFERER'], ‘digg.com’)) { 
/* Intensive plugin-code */
} ?>

This snippet basically says, “If this visitor is from Digg, don’t do the burdensome plugins.” The thinking is that if a visitor is coming from Digg, I am likely Dugg, so if I wasn’t lucky enough to see it coming, at least I will mitigate some of the problems.

I also disabled my anti-comment-spam plugins and only kept Akismet running (since I hear Techcrunch uses it). I also initially disabled Spam Karma 2, but I eventually turned back on (I prefer it over Akismet).

As a good example horror story of how a plugin can kill you, I once had the Bad Behavior plugin installed (late 2005). It completely locked up my database when several search bots hit the site because it attempted to log each and everything the bots did. It took me days to figure out my blog was taking down my entire server because of this plugin! (There is a new version out now, but I am too scared to try it now.)

Anyway, my point is for you to be careful with plugins that use the database and only use ones you absolutely need, especially when getting Dugg.

I then finished enabling my blog cache, and my server load fell to 1.00. My server was down for about 5 minutes total. Minutes later, I pointed the traffic back onto my site with Word Press caching enabled and the load sat around 4.00. I won’t know for sure, but I think my server would have survived had I enabled the cache plugin from the beginning (which I tried to do!! ).

The next day, I was on Slashdot and my server never went down. This is why I conclude that not enabling the cache plugin had more to do with going down than any other factor.

Non-Word Press Administrators

Note that the web server was fine. MySQL failed. MySQL is much less robust than the web server when it comes to this sort of stuff, and requires significantly more baby-sitting. This is especially true for applications that weren’t designed to scale, such as Word Press. This is why the cache plugin is so powerful.

If you have a web application that is MySQL intensive and NOT Word Press, the steps you need to take to keep your site up are different:

• Minimize the SQL running on the landing page that is getting Dugg. This may involve turning off things like session logging. For example, I turned off a user tracker that inserted a record into the database every time a visitor came to the site. Disabling this sped things up quite a bit.
• Create as much static content as possible, at least for the first two hours. After the initial surge, traffic will drop to manageable levels (see hourly graphs near bottom). Your best bet is to “fake” part of your application with static content and a disclaimer to come back later.
• Increase the memory usage limit for MySQL.
• Increase the maximum allowed connections to MySQL.
• Make sure you are using indexes in your queries. The quick and very dirty way of explaining this is if you have a SQL statement “… WHERE blah = ’some value’”, make sure there is an index on the column blah if that table is more than 500 records and that column has many unique values (i.e., ignore columns like status, gender, or active/inactive). No, that’s not the ideal answer (this is why DBAs make the big bucks), but it’s the quick and dirty explanation why many Word Press plugins tend to contribute to a server dying when getting Dugg. Perhaps I will cover this in more depth another day.

I hope this helps!

I was a subscriber of Comcast broadband for the past few years. My IP hardly ever changed, which was great for remote desktop, and it was down maybe a handful of times over a year. The speeds were always near advertised rates, and over all, I really couldn’t complain.

Then Time Warner bought out Comcast and all hell broke loose. My connection definitely does not drop out every day (that would be exaggerating), but it has not been the same.

Over the last 6 months since the switch, I have had my connection drop at least a dozen times, packet loss (that’s like a minor connection hiccup) at least once a week, and my download speeds drop to dial up more times than I can count. I don’t understand how the Internet quality could drop so suddenly when only the owners changed.

I’m not trying to say Time Warner is a horrible Internet provider. I am asking why my experience is possible!

Perhaps I misunderstand how buy-outs work, but it’s not like Comcast’s Internet servers packed up and took a vacation. It’s not like they went in and downgraded their pipes. All of the infrastructure is the same, or so it should be. And with the additional resources of Time Warner, you would think quality would actually improve.

I don’t know if this is something I should blame on the monopoly status that Time Warner now has. Maybe they decided that they wanted to literally fix what wasn’t broken (upgrade all the servers). Maybe a new marketing blitz doubled their subscriber base. I have no idea. But it angers me that an otherwise great service got dragged into mediocrity by being bought out by a more successful company.

I’m glad net neutrality didn’t get smashed because the Internet is already jacked up enough from the consumer’s side.

Sometimes, I am too much of a hacker for my own good. Note the distinction: I am not a good hacker, apparently.

This morning, I had this wacky idea of making the default RSS link for WordPress (my blog) point to my FeedBurner link. But I didn’t want to change the code in WordPress because I didn’t want to have to do it again later when WordPress got an update. So I opted to use the .htaccess file. For those of you curious, the code was (this works):

RewriteRule ^feed http://feeds.feedburner.com/MichiKnows [L,NC]

In non-nerd terms, it meant all default blog software accessing my website would be automatically forwarded to FeedBurner. This is good because it centralizes my reader base so I can track how many I have. Unfortunately, I was dumb and forgot that FeedBurner scans my RSS feed to generate its version of it.

That meant after about six hours, my feed completely broke when I made FeedBurner go in an infinite loop.

I’ll take a second shot at this problem later, but for now, I’ll concede and put things back to the way there were. Thanks J.M. for pointing out this issue!

Update: My friend pointed me to this plugin. It is very easy to install, so I’ll leave my hacker ambitions alone for this task and use it instead.

I’ve always wondered what all of the folder names meant on Linux, and Saturday’s post on Slashdot about the /etc folder answered it.

From the responses:

All the system directories were kept to three letters, and all of the names are abbreviations — none are acronyms.

/adm = administrative (now found in /var/adm)
/bin - binaries
/sbin - system binaries
/dev - devices
/home - user home directories
/lib - libraries
/log = logs (now found in /var/log)
/mnt - temporary mount point
/root - root’s home directory in case /home is on another file system
/var - variable data, such as databases, news, and mail
/tmp - temporary files
/usr - mostly there because it wouldn’t fit on /
/etc - stuff that doesn’t fit any of the above

Neat, huh?

Another neat factoid regarding “/etc”:

Lot of hardcore UNIX guys pronounce it “et-see” because you sound retarded saying, “It’s in “et-cetera-slash-init-period-d” rather than “et-see init-dee”. Same reason people transliterate Ess-Que-Ell (SQL) into “Sequel” …It’s quicker, and it sounds better.

So only recently have I been paying attention to the RSS capabilities of my blog. I recently installed FeedBurner, which helps in tracking my readership. Well, another thing I did is install a module that lets me see how many people are currently viewing my blog. Usually, at any given time, I’ll see one search bot and one guest user (you). After my recent request for my readers to subscribe to my blog, I’ve noticed something.

Whenever I make a post, I see a spike of maybe a dozen visitors within one minute of making the post. Neat!

As a side note, my blog continues to suffer from periodic lag caused by FeedBurner’s tracking script. I’ve never seen this be an issue with Google Analytics. Grr. I continue to use it because the stats FeedBurner tracks are more granular than Analytics.

Are you one of my RSS readers? If not, click on that orange RSS icon!